agora inbox for [email protected]help / color / mirror / Atom feed
Re: [bug fix] Produce a crash dump before main() on Windows 46+ messages / 6 participants [nested] [flat]
* Re: [bug fix] Produce a crash dump before main() on Windows @ 2019-06-29 00:10 Amit Kapila <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Amit Kapila @ 2019-06-29 00:10 UTC (permalink / raw) To: Haribabu Kommi <[email protected]>; +Cc: Tsunakawa, Takayuki <[email protected]>; Michael Paquier <[email protected]>; Robert Haas <[email protected]>; Craig Ringer <[email protected]>; Magnus Hagander <[email protected]>; pgsql-hackers On Tue, Nov 6, 2018 at 10:24 AM Haribabu Kommi <[email protected]> wrote: > > On Thu, Jul 26, 2018 at 3:52 PM Tsunakawa, Takayuki <[email protected]> wrote: > > > Thanks for confirmation of that PostgreSQL runs as service. > > Based on the following details, we can decide whether this fix is required or not. > 1. Starting of Postgres server using pg_ctl without service is of production use or not? > 2. Without this fix, how difficult is the problem to find out that something is preventing the > server to start? In case if it is easy to find out, may be better to provide some troubleshoot > guide for windows users can help. > > I am in favor of doc fix if it easy to find the problem instead of assuming the user usage. > Tsunakawa/Haribabu - By reading this thread briefly, it seems we need some more inputs from other developers on whether to fix this or not, so ideally the status of this patch should be 'Needs Review'. Why it is in 'Waiting on Author' state? If something is required from Author, then do we expect to see the updated patch in the next few days? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v20 4/5] restructure archive modules docs in preparation for restore modules @ 2023-02-15 22:48 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Nathan Bossart @ 2023-02-15 22:48 UTC (permalink / raw) --- doc/src/sgml/archive-and-restore-modules.sgml | 226 ++++++++++-------- 1 file changed, 120 insertions(+), 106 deletions(-) diff --git a/doc/src/sgml/archive-and-restore-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml index cf7438a759..a52d15082d 100644 --- a/doc/src/sgml/archive-and-restore-modules.sgml +++ b/doc/src/sgml/archive-and-restore-modules.sgml @@ -1,9 +1,9 @@ -<!-- doc/src/sgml/archive-modules.sgml --> +<!-- doc/src/sgml/archive-and-restore-modules.sgml --> -<chapter id="archive-modules"> - <title>Archive Modules</title> - <indexterm zone="archive-modules"> - <primary>Archive Modules</primary> +<chapter id="archive-and-restore-modules"> + <title>Archive and Restore Modules</title> + <indexterm zone="archive-and-restore-modules"> + <primary>Archive and Restore Modules</primary> </indexterm> <para> @@ -14,45 +14,52 @@ performant. </para> - <para> - When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL - will submit completed WAL files to the module, and the server will avoid - recycling or removing these WAL files until the module indicates that the files - were successfully archived. It is ultimately up to the module to decide what - to do with each WAL file, but many recommendations are listed at - <xref linkend="backup-archiving-wal"/>. - </para> - - <para> - Archiving modules must at least consist of an initialization function (see - <xref linkend="archive-module-init"/>) and the required callbacks (see - <xref linkend="archive-module-callbacks"/>). However, archive modules are - also permitted to do much more (e.g., declare GUCs and register background - workers). - </para> - <para> The <filename>contrib/basic_archive</filename> module contains a working example, which demonstrates some useful techniques. </para> - <sect1 id="archive-module-init"> - <title>Initialization Functions</title> - <indexterm zone="archive-module-init"> - <primary>_PG_archive_module_init</primary> + <sect1 id="archive-modules"> + <title>Archive Modules</title> + <indexterm zone="archive-modules"> + <primary>Archive Modules</primary> </indexterm> + + <para> + When a custom <xref linkend="guc-archive-library"/> is configured, + PostgreSQL will submit completed WAL files to the module, and the server + will avoid recycling or removing these WAL files until the module indicates + that the files were successfully archived. It is ultimately up to the + module to decide what to do with each WAL file, but many recommendations are + listed at <xref linkend="backup-archiving-wal"/>. + </para> + <para> - An archive library is loaded by dynamically loading a shared library with the - <xref linkend="guc-archive-library"/>'s name as the library base name. The - normal library search path is used to locate the library. To provide the - required archive module callbacks and to indicate that the library is - actually an archive module, it needs to provide a function named - <function>_PG_archive_module_init</function>. The result of the function - must be a pointer to a struct of type - <structname>ArchiveModuleCallbacks</structname>, which contains everything - that the core code needs to know to make use of the archive module. The - return value needs to be of server lifetime, which is typically achieved by - defining it as a <literal>static const</literal> variable in global scope. + Archiving modules must at least consist of an initialization function (see + <xref linkend="archive-module-init"/>) and the required callbacks (see + <xref linkend="archive-module-callbacks"/>). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + </para> + + <sect2 id="archive-module-init"> + <title>Initialization Functions</title> + <indexterm zone="archive-module-init"> + <primary>_PG_archive_module_init</primary> + </indexterm> + + <para> + An archive library is loaded by dynamically loading a shared library with + the <xref linkend="guc-archive-library"/>'s name as the library base name. + The normal library search path is used to locate the library. To provide + the required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + <function>_PG_archive_module_init</function>. The result of the function + must be a pointer to a struct of type + <structname>ArchiveModuleCallbacks</structname>, which contains everything + that the core code needs to know to make use of the archive module. The + return value needs to be of server lifetime, which is typically achieved by + defining it as a <literal>static const</literal> variable in global scope. <programlisting> typedef struct ArchiveModuleCallbacks @@ -65,103 +72,110 @@ typedef struct ArchiveModuleCallbacks typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void); </programlisting> - Only the <function>archive_file_cb</function> callback is required. The - others are optional. - </para> - </sect1> + Only the <function>archive_file_cb</function> callback is required. The + others are optional. + </para> + </sect2> - <sect1 id="archive-module-callbacks"> - <title>Archive Module Callbacks</title> - <para> - The archive callbacks define the actual archiving behavior of the module. - The server will call them as required to process each individual WAL file. - </para> + <sect2 id="archive-module-callbacks"> + <title>Archive Module Callbacks</title> - <sect2 id="archive-module-startup"> - <title>Startup Callback</title> <para> - The <function>startup_cb</function> callback is called shortly after the - module is loaded. This callback can be used to perform any additional - initialization required. If the archive module has any state, it can use - <structfield>state->private_data</structfield> to store it. + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + </para> + + <sect3 id="archive-module-startup"> + <title>Startup Callback</title> + + <para> + The <function>startup_cb</function> callback is called shortly after the + module is loaded. This callback can be used to perform any additional + initialization required. If the archive module has any state, it can use + <structfield>state->private_data</structfield> to store it. <programlisting> typedef void (*ArchiveStartupCB) (ArchiveModuleState *state); </programlisting> - </para> - </sect2> + </para> + </sect3> - <sect2 id="archive-module-check"> - <title>Check Callback</title> - <para> - The <function>check_configured_cb</function> callback is called to determine - whether the module is fully configured and ready to accept WAL files (e.g., - its configuration parameters are set to valid values). If no - <function>check_configured_cb</function> is defined, the server always - assumes the module is configured. + <sect3 id="archive-module-check"> + <title>Check Callback</title> + + <para> + The <function>check_configured_cb</function> callback is called to + determine whether the module is fully configured and ready to accept WAL + files (e.g., its configuration parameters are set to valid values). If no + <function>check_configured_cb</function> is defined, the server always + assumes the module is configured. <programlisting> typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); </programlisting> - If <literal>true</literal> is returned, the server will proceed with - archiving the file by calling the <function>archive_file_cb</function> - callback. If <literal>false</literal> is returned, archiving will not - proceed, and the archiver will emit the following message to the server log: + If <literal>true</literal> is returned, the server will proceed with + archiving the file by calling the <function>archive_file_cb</function> + callback. If <literal>false</literal> is returned, archiving will not + proceed, and the archiver will emit the following message to the server + log: <screen> WARNING: archive_mode enabled, yet archiving is not configured </screen> - In the latter case, the server will periodically call this function, and - archiving will proceed only when it returns <literal>true</literal>. - </para> - - <note> - <para> - When returning <literal>false</literal>, it may be useful to append some - additional information to the generic warning message. To do that, provide - a message to the <function>arch_module_check_errdetail</function> macro - before returning <literal>false</literal>. Like - <function>errdetail()</function>, this macro accepts a format string - followed by an optional list of arguments. The resulting string will be - emitted as the <literal>DETAIL</literal> line of the warning message. + In the latter case, the server will periodically call this function, and + archiving will proceed only when it returns <literal>true</literal>. </para> - </note> - </sect2> - <sect2 id="archive-module-archive"> - <title>Archive Callback</title> - <para> - The <function>archive_file_cb</function> callback is called to archive a - single WAL file. + <note> + <para> + When returning <literal>false</literal>, it may be useful to append some + additional information to the generic warning message. To do that, + provide a message to the <function>arch_module_check_errdetail</function> + macro before returning <literal>false</literal>. Like + <function>errdetail()</function>, this macro accepts a format string + followed by an optional list of arguments. The resulting string will be + emitted as the <literal>DETAIL</literal> line of the warning message. + </para> + </note> + </sect3> + + <sect3 id="archive-module-archive"> + <title>Archive Callback</title> + + <para> + The <function>archive_file_cb</function> callback is called to archive a + single WAL file. <programlisting> typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path); </programlisting> - If <literal>true</literal> is returned, the server proceeds as if the file - was successfully archived, which may include recycling or removing the - original WAL file. If <literal>false</literal> is returned, the server will - keep the original WAL file and retry archiving later. - <replaceable>file</replaceable> will contain just the file name of the WAL - file to archive, while <replaceable>path</replaceable> contains the full - path of the WAL file (including the file name). - </para> - </sect2> + If <literal>true</literal> is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If <literal>false</literal> is returned, the server + will keep the original WAL file and retry archiving later. + <replaceable>file</replaceable> will contain just the file name of the WAL + file to archive, while <replaceable>path</replaceable> contains the full + path of the WAL file (including the file name). + </para> + </sect3> - <sect2 id="archive-module-shutdown"> - <title>Shutdown Callback</title> - <para> - The <function>shutdown_cb</function> callback is called when the archiver - process exits (e.g., after an error) or the value of - <xref linkend="guc-archive-library"/> changes. If no - <function>shutdown_cb</function> is defined, no special action is taken in - these situations. If the archive module has any state, this callback should - free it to avoid leaks. + <sect3 id="archive-module-shutdown"> + <title>Shutdown Callback</title> + + <para> + The <function>shutdown_cb</function> callback is called when the archiver + process exits (e.g., after an error) or the value of + <xref linkend="guc-archive-library"/> changes. If no + <function>shutdown_cb</function> is defined, no special action is taken in + these situations. If the archive module has any state, this callback + should free it to avoid leaks. <programlisting> typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state); </programlisting> - </para> + </para> + </sect3> </sect2> </sect1> </chapter> -- 2.25.1 --3V7upXqbjpZ4EhLz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v20-0005-introduce-restore_library.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v19 4/5] restructure archive modules docs in preparation for restore modules @ 2023-02-15 22:48 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Nathan Bossart @ 2023-02-15 22:48 UTC (permalink / raw) --- doc/src/sgml/archive-and-restore-modules.sgml | 226 ++++++++++-------- 1 file changed, 120 insertions(+), 106 deletions(-) diff --git a/doc/src/sgml/archive-and-restore-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml index cf7438a759..a52d15082d 100644 --- a/doc/src/sgml/archive-and-restore-modules.sgml +++ b/doc/src/sgml/archive-and-restore-modules.sgml @@ -1,9 +1,9 @@ -<!-- doc/src/sgml/archive-modules.sgml --> +<!-- doc/src/sgml/archive-and-restore-modules.sgml --> -<chapter id="archive-modules"> - <title>Archive Modules</title> - <indexterm zone="archive-modules"> - <primary>Archive Modules</primary> +<chapter id="archive-and-restore-modules"> + <title>Archive and Restore Modules</title> + <indexterm zone="archive-and-restore-modules"> + <primary>Archive and Restore Modules</primary> </indexterm> <para> @@ -14,45 +14,52 @@ performant. </para> - <para> - When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL - will submit completed WAL files to the module, and the server will avoid - recycling or removing these WAL files until the module indicates that the files - were successfully archived. It is ultimately up to the module to decide what - to do with each WAL file, but many recommendations are listed at - <xref linkend="backup-archiving-wal"/>. - </para> - - <para> - Archiving modules must at least consist of an initialization function (see - <xref linkend="archive-module-init"/>) and the required callbacks (see - <xref linkend="archive-module-callbacks"/>). However, archive modules are - also permitted to do much more (e.g., declare GUCs and register background - workers). - </para> - <para> The <filename>contrib/basic_archive</filename> module contains a working example, which demonstrates some useful techniques. </para> - <sect1 id="archive-module-init"> - <title>Initialization Functions</title> - <indexterm zone="archive-module-init"> - <primary>_PG_archive_module_init</primary> + <sect1 id="archive-modules"> + <title>Archive Modules</title> + <indexterm zone="archive-modules"> + <primary>Archive Modules</primary> </indexterm> + + <para> + When a custom <xref linkend="guc-archive-library"/> is configured, + PostgreSQL will submit completed WAL files to the module, and the server + will avoid recycling or removing these WAL files until the module indicates + that the files were successfully archived. It is ultimately up to the + module to decide what to do with each WAL file, but many recommendations are + listed at <xref linkend="backup-archiving-wal"/>. + </para> + <para> - An archive library is loaded by dynamically loading a shared library with the - <xref linkend="guc-archive-library"/>'s name as the library base name. The - normal library search path is used to locate the library. To provide the - required archive module callbacks and to indicate that the library is - actually an archive module, it needs to provide a function named - <function>_PG_archive_module_init</function>. The result of the function - must be a pointer to a struct of type - <structname>ArchiveModuleCallbacks</structname>, which contains everything - that the core code needs to know to make use of the archive module. The - return value needs to be of server lifetime, which is typically achieved by - defining it as a <literal>static const</literal> variable in global scope. + Archiving modules must at least consist of an initialization function (see + <xref linkend="archive-module-init"/>) and the required callbacks (see + <xref linkend="archive-module-callbacks"/>). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + </para> + + <sect2 id="archive-module-init"> + <title>Initialization Functions</title> + <indexterm zone="archive-module-init"> + <primary>_PG_archive_module_init</primary> + </indexterm> + + <para> + An archive library is loaded by dynamically loading a shared library with + the <xref linkend="guc-archive-library"/>'s name as the library base name. + The normal library search path is used to locate the library. To provide + the required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + <function>_PG_archive_module_init</function>. The result of the function + must be a pointer to a struct of type + <structname>ArchiveModuleCallbacks</structname>, which contains everything + that the core code needs to know to make use of the archive module. The + return value needs to be of server lifetime, which is typically achieved by + defining it as a <literal>static const</literal> variable in global scope. <programlisting> typedef struct ArchiveModuleCallbacks @@ -65,103 +72,110 @@ typedef struct ArchiveModuleCallbacks typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void); </programlisting> - Only the <function>archive_file_cb</function> callback is required. The - others are optional. - </para> - </sect1> + Only the <function>archive_file_cb</function> callback is required. The + others are optional. + </para> + </sect2> - <sect1 id="archive-module-callbacks"> - <title>Archive Module Callbacks</title> - <para> - The archive callbacks define the actual archiving behavior of the module. - The server will call them as required to process each individual WAL file. - </para> + <sect2 id="archive-module-callbacks"> + <title>Archive Module Callbacks</title> - <sect2 id="archive-module-startup"> - <title>Startup Callback</title> <para> - The <function>startup_cb</function> callback is called shortly after the - module is loaded. This callback can be used to perform any additional - initialization required. If the archive module has any state, it can use - <structfield>state->private_data</structfield> to store it. + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + </para> + + <sect3 id="archive-module-startup"> + <title>Startup Callback</title> + + <para> + The <function>startup_cb</function> callback is called shortly after the + module is loaded. This callback can be used to perform any additional + initialization required. If the archive module has any state, it can use + <structfield>state->private_data</structfield> to store it. <programlisting> typedef void (*ArchiveStartupCB) (ArchiveModuleState *state); </programlisting> - </para> - </sect2> + </para> + </sect3> - <sect2 id="archive-module-check"> - <title>Check Callback</title> - <para> - The <function>check_configured_cb</function> callback is called to determine - whether the module is fully configured and ready to accept WAL files (e.g., - its configuration parameters are set to valid values). If no - <function>check_configured_cb</function> is defined, the server always - assumes the module is configured. + <sect3 id="archive-module-check"> + <title>Check Callback</title> + + <para> + The <function>check_configured_cb</function> callback is called to + determine whether the module is fully configured and ready to accept WAL + files (e.g., its configuration parameters are set to valid values). If no + <function>check_configured_cb</function> is defined, the server always + assumes the module is configured. <programlisting> typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); </programlisting> - If <literal>true</literal> is returned, the server will proceed with - archiving the file by calling the <function>archive_file_cb</function> - callback. If <literal>false</literal> is returned, archiving will not - proceed, and the archiver will emit the following message to the server log: + If <literal>true</literal> is returned, the server will proceed with + archiving the file by calling the <function>archive_file_cb</function> + callback. If <literal>false</literal> is returned, archiving will not + proceed, and the archiver will emit the following message to the server + log: <screen> WARNING: archive_mode enabled, yet archiving is not configured </screen> - In the latter case, the server will periodically call this function, and - archiving will proceed only when it returns <literal>true</literal>. - </para> - - <note> - <para> - When returning <literal>false</literal>, it may be useful to append some - additional information to the generic warning message. To do that, provide - a message to the <function>arch_module_check_errdetail</function> macro - before returning <literal>false</literal>. Like - <function>errdetail()</function>, this macro accepts a format string - followed by an optional list of arguments. The resulting string will be - emitted as the <literal>DETAIL</literal> line of the warning message. + In the latter case, the server will periodically call this function, and + archiving will proceed only when it returns <literal>true</literal>. </para> - </note> - </sect2> - <sect2 id="archive-module-archive"> - <title>Archive Callback</title> - <para> - The <function>archive_file_cb</function> callback is called to archive a - single WAL file. + <note> + <para> + When returning <literal>false</literal>, it may be useful to append some + additional information to the generic warning message. To do that, + provide a message to the <function>arch_module_check_errdetail</function> + macro before returning <literal>false</literal>. Like + <function>errdetail()</function>, this macro accepts a format string + followed by an optional list of arguments. The resulting string will be + emitted as the <literal>DETAIL</literal> line of the warning message. + </para> + </note> + </sect3> + + <sect3 id="archive-module-archive"> + <title>Archive Callback</title> + + <para> + The <function>archive_file_cb</function> callback is called to archive a + single WAL file. <programlisting> typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path); </programlisting> - If <literal>true</literal> is returned, the server proceeds as if the file - was successfully archived, which may include recycling or removing the - original WAL file. If <literal>false</literal> is returned, the server will - keep the original WAL file and retry archiving later. - <replaceable>file</replaceable> will contain just the file name of the WAL - file to archive, while <replaceable>path</replaceable> contains the full - path of the WAL file (including the file name). - </para> - </sect2> + If <literal>true</literal> is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If <literal>false</literal> is returned, the server + will keep the original WAL file and retry archiving later. + <replaceable>file</replaceable> will contain just the file name of the WAL + file to archive, while <replaceable>path</replaceable> contains the full + path of the WAL file (including the file name). + </para> + </sect3> - <sect2 id="archive-module-shutdown"> - <title>Shutdown Callback</title> - <para> - The <function>shutdown_cb</function> callback is called when the archiver - process exits (e.g., after an error) or the value of - <xref linkend="guc-archive-library"/> changes. If no - <function>shutdown_cb</function> is defined, no special action is taken in - these situations. If the archive module has any state, this callback should - free it to avoid leaks. + <sect3 id="archive-module-shutdown"> + <title>Shutdown Callback</title> + + <para> + The <function>shutdown_cb</function> callback is called when the archiver + process exits (e.g., after an error) or the value of + <xref linkend="guc-archive-library"/> changes. If no + <function>shutdown_cb</function> is defined, no special action is taken in + these situations. If the archive module has any state, this callback + should free it to avoid leaks. <programlisting> typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state); </programlisting> - </para> + </para> + </sect3> </sect2> </sect1> </chapter> -- 2.25.1 --oyUTqETQ0mS9luUI Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v19-0005-introduce-restore_library.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v18 4/5] restructure archive modules docs in preparation for restore modules @ 2023-02-15 22:48 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Nathan Bossart @ 2023-02-15 22:48 UTC (permalink / raw) --- doc/src/sgml/archive-and-restore-modules.sgml | 226 ++++++++++-------- 1 file changed, 120 insertions(+), 106 deletions(-) diff --git a/doc/src/sgml/archive-and-restore-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml index cf7438a759..a52d15082d 100644 --- a/doc/src/sgml/archive-and-restore-modules.sgml +++ b/doc/src/sgml/archive-and-restore-modules.sgml @@ -1,9 +1,9 @@ -<!-- doc/src/sgml/archive-modules.sgml --> +<!-- doc/src/sgml/archive-and-restore-modules.sgml --> -<chapter id="archive-modules"> - <title>Archive Modules</title> - <indexterm zone="archive-modules"> - <primary>Archive Modules</primary> +<chapter id="archive-and-restore-modules"> + <title>Archive and Restore Modules</title> + <indexterm zone="archive-and-restore-modules"> + <primary>Archive and Restore Modules</primary> </indexterm> <para> @@ -14,45 +14,52 @@ performant. </para> - <para> - When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL - will submit completed WAL files to the module, and the server will avoid - recycling or removing these WAL files until the module indicates that the files - were successfully archived. It is ultimately up to the module to decide what - to do with each WAL file, but many recommendations are listed at - <xref linkend="backup-archiving-wal"/>. - </para> - - <para> - Archiving modules must at least consist of an initialization function (see - <xref linkend="archive-module-init"/>) and the required callbacks (see - <xref linkend="archive-module-callbacks"/>). However, archive modules are - also permitted to do much more (e.g., declare GUCs and register background - workers). - </para> - <para> The <filename>contrib/basic_archive</filename> module contains a working example, which demonstrates some useful techniques. </para> - <sect1 id="archive-module-init"> - <title>Initialization Functions</title> - <indexterm zone="archive-module-init"> - <primary>_PG_archive_module_init</primary> + <sect1 id="archive-modules"> + <title>Archive Modules</title> + <indexterm zone="archive-modules"> + <primary>Archive Modules</primary> </indexterm> + + <para> + When a custom <xref linkend="guc-archive-library"/> is configured, + PostgreSQL will submit completed WAL files to the module, and the server + will avoid recycling or removing these WAL files until the module indicates + that the files were successfully archived. It is ultimately up to the + module to decide what to do with each WAL file, but many recommendations are + listed at <xref linkend="backup-archiving-wal"/>. + </para> + <para> - An archive library is loaded by dynamically loading a shared library with the - <xref linkend="guc-archive-library"/>'s name as the library base name. The - normal library search path is used to locate the library. To provide the - required archive module callbacks and to indicate that the library is - actually an archive module, it needs to provide a function named - <function>_PG_archive_module_init</function>. The result of the function - must be a pointer to a struct of type - <structname>ArchiveModuleCallbacks</structname>, which contains everything - that the core code needs to know to make use of the archive module. The - return value needs to be of server lifetime, which is typically achieved by - defining it as a <literal>static const</literal> variable in global scope. + Archiving modules must at least consist of an initialization function (see + <xref linkend="archive-module-init"/>) and the required callbacks (see + <xref linkend="archive-module-callbacks"/>). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + </para> + + <sect2 id="archive-module-init"> + <title>Initialization Functions</title> + <indexterm zone="archive-module-init"> + <primary>_PG_archive_module_init</primary> + </indexterm> + + <para> + An archive library is loaded by dynamically loading a shared library with + the <xref linkend="guc-archive-library"/>'s name as the library base name. + The normal library search path is used to locate the library. To provide + the required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + <function>_PG_archive_module_init</function>. The result of the function + must be a pointer to a struct of type + <structname>ArchiveModuleCallbacks</structname>, which contains everything + that the core code needs to know to make use of the archive module. The + return value needs to be of server lifetime, which is typically achieved by + defining it as a <literal>static const</literal> variable in global scope. <programlisting> typedef struct ArchiveModuleCallbacks @@ -65,103 +72,110 @@ typedef struct ArchiveModuleCallbacks typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void); </programlisting> - Only the <function>archive_file_cb</function> callback is required. The - others are optional. - </para> - </sect1> + Only the <function>archive_file_cb</function> callback is required. The + others are optional. + </para> + </sect2> - <sect1 id="archive-module-callbacks"> - <title>Archive Module Callbacks</title> - <para> - The archive callbacks define the actual archiving behavior of the module. - The server will call them as required to process each individual WAL file. - </para> + <sect2 id="archive-module-callbacks"> + <title>Archive Module Callbacks</title> - <sect2 id="archive-module-startup"> - <title>Startup Callback</title> <para> - The <function>startup_cb</function> callback is called shortly after the - module is loaded. This callback can be used to perform any additional - initialization required. If the archive module has any state, it can use - <structfield>state->private_data</structfield> to store it. + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + </para> + + <sect3 id="archive-module-startup"> + <title>Startup Callback</title> + + <para> + The <function>startup_cb</function> callback is called shortly after the + module is loaded. This callback can be used to perform any additional + initialization required. If the archive module has any state, it can use + <structfield>state->private_data</structfield> to store it. <programlisting> typedef void (*ArchiveStartupCB) (ArchiveModuleState *state); </programlisting> - </para> - </sect2> + </para> + </sect3> - <sect2 id="archive-module-check"> - <title>Check Callback</title> - <para> - The <function>check_configured_cb</function> callback is called to determine - whether the module is fully configured and ready to accept WAL files (e.g., - its configuration parameters are set to valid values). If no - <function>check_configured_cb</function> is defined, the server always - assumes the module is configured. + <sect3 id="archive-module-check"> + <title>Check Callback</title> + + <para> + The <function>check_configured_cb</function> callback is called to + determine whether the module is fully configured and ready to accept WAL + files (e.g., its configuration parameters are set to valid values). If no + <function>check_configured_cb</function> is defined, the server always + assumes the module is configured. <programlisting> typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); </programlisting> - If <literal>true</literal> is returned, the server will proceed with - archiving the file by calling the <function>archive_file_cb</function> - callback. If <literal>false</literal> is returned, archiving will not - proceed, and the archiver will emit the following message to the server log: + If <literal>true</literal> is returned, the server will proceed with + archiving the file by calling the <function>archive_file_cb</function> + callback. If <literal>false</literal> is returned, archiving will not + proceed, and the archiver will emit the following message to the server + log: <screen> WARNING: archive_mode enabled, yet archiving is not configured </screen> - In the latter case, the server will periodically call this function, and - archiving will proceed only when it returns <literal>true</literal>. - </para> - - <note> - <para> - When returning <literal>false</literal>, it may be useful to append some - additional information to the generic warning message. To do that, provide - a message to the <function>arch_module_check_errdetail</function> macro - before returning <literal>false</literal>. Like - <function>errdetail()</function>, this macro accepts a format string - followed by an optional list of arguments. The resulting string will be - emitted as the <literal>DETAIL</literal> line of the warning message. + In the latter case, the server will periodically call this function, and + archiving will proceed only when it returns <literal>true</literal>. </para> - </note> - </sect2> - <sect2 id="archive-module-archive"> - <title>Archive Callback</title> - <para> - The <function>archive_file_cb</function> callback is called to archive a - single WAL file. + <note> + <para> + When returning <literal>false</literal>, it may be useful to append some + additional information to the generic warning message. To do that, + provide a message to the <function>arch_module_check_errdetail</function> + macro before returning <literal>false</literal>. Like + <function>errdetail()</function>, this macro accepts a format string + followed by an optional list of arguments. The resulting string will be + emitted as the <literal>DETAIL</literal> line of the warning message. + </para> + </note> + </sect3> + + <sect3 id="archive-module-archive"> + <title>Archive Callback</title> + + <para> + The <function>archive_file_cb</function> callback is called to archive a + single WAL file. <programlisting> typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path); </programlisting> - If <literal>true</literal> is returned, the server proceeds as if the file - was successfully archived, which may include recycling or removing the - original WAL file. If <literal>false</literal> is returned, the server will - keep the original WAL file and retry archiving later. - <replaceable>file</replaceable> will contain just the file name of the WAL - file to archive, while <replaceable>path</replaceable> contains the full - path of the WAL file (including the file name). - </para> - </sect2> + If <literal>true</literal> is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If <literal>false</literal> is returned, the server + will keep the original WAL file and retry archiving later. + <replaceable>file</replaceable> will contain just the file name of the WAL + file to archive, while <replaceable>path</replaceable> contains the full + path of the WAL file (including the file name). + </para> + </sect3> - <sect2 id="archive-module-shutdown"> - <title>Shutdown Callback</title> - <para> - The <function>shutdown_cb</function> callback is called when the archiver - process exits (e.g., after an error) or the value of - <xref linkend="guc-archive-library"/> changes. If no - <function>shutdown_cb</function> is defined, no special action is taken in - these situations. If the archive module has any state, this callback should - free it to avoid leaks. + <sect3 id="archive-module-shutdown"> + <title>Shutdown Callback</title> + + <para> + The <function>shutdown_cb</function> callback is called when the archiver + process exits (e.g., after an error) or the value of + <xref linkend="guc-archive-library"/> changes. If no + <function>shutdown_cb</function> is defined, no special action is taken in + these situations. If the archive module has any state, this callback + should free it to avoid leaks. <programlisting> typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state); </programlisting> - </para> + </para> + </sect3> </sect2> </sect1> </chapter> -- 2.25.1 --IJpNTDwzlM2Ie8A6 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v18-0005-introduce-restore_library.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v19 4/5] restructure archive modules docs in preparation for restore modules @ 2023-02-15 22:48 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Nathan Bossart @ 2023-02-15 22:48 UTC (permalink / raw) --- doc/src/sgml/archive-and-restore-modules.sgml | 226 ++++++++++-------- 1 file changed, 120 insertions(+), 106 deletions(-) diff --git a/doc/src/sgml/archive-and-restore-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml index cf7438a759..a52d15082d 100644 --- a/doc/src/sgml/archive-and-restore-modules.sgml +++ b/doc/src/sgml/archive-and-restore-modules.sgml @@ -1,9 +1,9 @@ -<!-- doc/src/sgml/archive-modules.sgml --> +<!-- doc/src/sgml/archive-and-restore-modules.sgml --> -<chapter id="archive-modules"> - <title>Archive Modules</title> - <indexterm zone="archive-modules"> - <primary>Archive Modules</primary> +<chapter id="archive-and-restore-modules"> + <title>Archive and Restore Modules</title> + <indexterm zone="archive-and-restore-modules"> + <primary>Archive and Restore Modules</primary> </indexterm> <para> @@ -14,45 +14,52 @@ performant. </para> - <para> - When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL - will submit completed WAL files to the module, and the server will avoid - recycling or removing these WAL files until the module indicates that the files - were successfully archived. It is ultimately up to the module to decide what - to do with each WAL file, but many recommendations are listed at - <xref linkend="backup-archiving-wal"/>. - </para> - - <para> - Archiving modules must at least consist of an initialization function (see - <xref linkend="archive-module-init"/>) and the required callbacks (see - <xref linkend="archive-module-callbacks"/>). However, archive modules are - also permitted to do much more (e.g., declare GUCs and register background - workers). - </para> - <para> The <filename>contrib/basic_archive</filename> module contains a working example, which demonstrates some useful techniques. </para> - <sect1 id="archive-module-init"> - <title>Initialization Functions</title> - <indexterm zone="archive-module-init"> - <primary>_PG_archive_module_init</primary> + <sect1 id="archive-modules"> + <title>Archive Modules</title> + <indexterm zone="archive-modules"> + <primary>Archive Modules</primary> </indexterm> + + <para> + When a custom <xref linkend="guc-archive-library"/> is configured, + PostgreSQL will submit completed WAL files to the module, and the server + will avoid recycling or removing these WAL files until the module indicates + that the files were successfully archived. It is ultimately up to the + module to decide what to do with each WAL file, but many recommendations are + listed at <xref linkend="backup-archiving-wal"/>. + </para> + <para> - An archive library is loaded by dynamically loading a shared library with the - <xref linkend="guc-archive-library"/>'s name as the library base name. The - normal library search path is used to locate the library. To provide the - required archive module callbacks and to indicate that the library is - actually an archive module, it needs to provide a function named - <function>_PG_archive_module_init</function>. The result of the function - must be a pointer to a struct of type - <structname>ArchiveModuleCallbacks</structname>, which contains everything - that the core code needs to know to make use of the archive module. The - return value needs to be of server lifetime, which is typically achieved by - defining it as a <literal>static const</literal> variable in global scope. + Archiving modules must at least consist of an initialization function (see + <xref linkend="archive-module-init"/>) and the required callbacks (see + <xref linkend="archive-module-callbacks"/>). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + </para> + + <sect2 id="archive-module-init"> + <title>Initialization Functions</title> + <indexterm zone="archive-module-init"> + <primary>_PG_archive_module_init</primary> + </indexterm> + + <para> + An archive library is loaded by dynamically loading a shared library with + the <xref linkend="guc-archive-library"/>'s name as the library base name. + The normal library search path is used to locate the library. To provide + the required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + <function>_PG_archive_module_init</function>. The result of the function + must be a pointer to a struct of type + <structname>ArchiveModuleCallbacks</structname>, which contains everything + that the core code needs to know to make use of the archive module. The + return value needs to be of server lifetime, which is typically achieved by + defining it as a <literal>static const</literal> variable in global scope. <programlisting> typedef struct ArchiveModuleCallbacks @@ -65,103 +72,110 @@ typedef struct ArchiveModuleCallbacks typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void); </programlisting> - Only the <function>archive_file_cb</function> callback is required. The - others are optional. - </para> - </sect1> + Only the <function>archive_file_cb</function> callback is required. The + others are optional. + </para> + </sect2> - <sect1 id="archive-module-callbacks"> - <title>Archive Module Callbacks</title> - <para> - The archive callbacks define the actual archiving behavior of the module. - The server will call them as required to process each individual WAL file. - </para> + <sect2 id="archive-module-callbacks"> + <title>Archive Module Callbacks</title> - <sect2 id="archive-module-startup"> - <title>Startup Callback</title> <para> - The <function>startup_cb</function> callback is called shortly after the - module is loaded. This callback can be used to perform any additional - initialization required. If the archive module has any state, it can use - <structfield>state->private_data</structfield> to store it. + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + </para> + + <sect3 id="archive-module-startup"> + <title>Startup Callback</title> + + <para> + The <function>startup_cb</function> callback is called shortly after the + module is loaded. This callback can be used to perform any additional + initialization required. If the archive module has any state, it can use + <structfield>state->private_data</structfield> to store it. <programlisting> typedef void (*ArchiveStartupCB) (ArchiveModuleState *state); </programlisting> - </para> - </sect2> + </para> + </sect3> - <sect2 id="archive-module-check"> - <title>Check Callback</title> - <para> - The <function>check_configured_cb</function> callback is called to determine - whether the module is fully configured and ready to accept WAL files (e.g., - its configuration parameters are set to valid values). If no - <function>check_configured_cb</function> is defined, the server always - assumes the module is configured. + <sect3 id="archive-module-check"> + <title>Check Callback</title> + + <para> + The <function>check_configured_cb</function> callback is called to + determine whether the module is fully configured and ready to accept WAL + files (e.g., its configuration parameters are set to valid values). If no + <function>check_configured_cb</function> is defined, the server always + assumes the module is configured. <programlisting> typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); </programlisting> - If <literal>true</literal> is returned, the server will proceed with - archiving the file by calling the <function>archive_file_cb</function> - callback. If <literal>false</literal> is returned, archiving will not - proceed, and the archiver will emit the following message to the server log: + If <literal>true</literal> is returned, the server will proceed with + archiving the file by calling the <function>archive_file_cb</function> + callback. If <literal>false</literal> is returned, archiving will not + proceed, and the archiver will emit the following message to the server + log: <screen> WARNING: archive_mode enabled, yet archiving is not configured </screen> - In the latter case, the server will periodically call this function, and - archiving will proceed only when it returns <literal>true</literal>. - </para> - - <note> - <para> - When returning <literal>false</literal>, it may be useful to append some - additional information to the generic warning message. To do that, provide - a message to the <function>arch_module_check_errdetail</function> macro - before returning <literal>false</literal>. Like - <function>errdetail()</function>, this macro accepts a format string - followed by an optional list of arguments. The resulting string will be - emitted as the <literal>DETAIL</literal> line of the warning message. + In the latter case, the server will periodically call this function, and + archiving will proceed only when it returns <literal>true</literal>. </para> - </note> - </sect2> - <sect2 id="archive-module-archive"> - <title>Archive Callback</title> - <para> - The <function>archive_file_cb</function> callback is called to archive a - single WAL file. + <note> + <para> + When returning <literal>false</literal>, it may be useful to append some + additional information to the generic warning message. To do that, + provide a message to the <function>arch_module_check_errdetail</function> + macro before returning <literal>false</literal>. Like + <function>errdetail()</function>, this macro accepts a format string + followed by an optional list of arguments. The resulting string will be + emitted as the <literal>DETAIL</literal> line of the warning message. + </para> + </note> + </sect3> + + <sect3 id="archive-module-archive"> + <title>Archive Callback</title> + + <para> + The <function>archive_file_cb</function> callback is called to archive a + single WAL file. <programlisting> typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path); </programlisting> - If <literal>true</literal> is returned, the server proceeds as if the file - was successfully archived, which may include recycling or removing the - original WAL file. If <literal>false</literal> is returned, the server will - keep the original WAL file and retry archiving later. - <replaceable>file</replaceable> will contain just the file name of the WAL - file to archive, while <replaceable>path</replaceable> contains the full - path of the WAL file (including the file name). - </para> - </sect2> + If <literal>true</literal> is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If <literal>false</literal> is returned, the server + will keep the original WAL file and retry archiving later. + <replaceable>file</replaceable> will contain just the file name of the WAL + file to archive, while <replaceable>path</replaceable> contains the full + path of the WAL file (including the file name). + </para> + </sect3> - <sect2 id="archive-module-shutdown"> - <title>Shutdown Callback</title> - <para> - The <function>shutdown_cb</function> callback is called when the archiver - process exits (e.g., after an error) or the value of - <xref linkend="guc-archive-library"/> changes. If no - <function>shutdown_cb</function> is defined, no special action is taken in - these situations. If the archive module has any state, this callback should - free it to avoid leaks. + <sect3 id="archive-module-shutdown"> + <title>Shutdown Callback</title> + + <para> + The <function>shutdown_cb</function> callback is called when the archiver + process exits (e.g., after an error) or the value of + <xref linkend="guc-archive-library"/> changes. If no + <function>shutdown_cb</function> is defined, no special action is taken in + these situations. If the archive module has any state, this callback + should free it to avoid leaks. <programlisting> typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state); </programlisting> - </para> + </para> + </sect3> </sect2> </sect1> </chapter> -- 2.25.1 --oyUTqETQ0mS9luUI Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v19-0005-introduce-restore_library.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v18 4/5] restructure archive modules docs in preparation for restore modules @ 2023-02-15 22:48 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Nathan Bossart @ 2023-02-15 22:48 UTC (permalink / raw) --- doc/src/sgml/archive-and-restore-modules.sgml | 226 ++++++++++-------- 1 file changed, 120 insertions(+), 106 deletions(-) diff --git a/doc/src/sgml/archive-and-restore-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml index cf7438a759..a52d15082d 100644 --- a/doc/src/sgml/archive-and-restore-modules.sgml +++ b/doc/src/sgml/archive-and-restore-modules.sgml @@ -1,9 +1,9 @@ -<!-- doc/src/sgml/archive-modules.sgml --> +<!-- doc/src/sgml/archive-and-restore-modules.sgml --> -<chapter id="archive-modules"> - <title>Archive Modules</title> - <indexterm zone="archive-modules"> - <primary>Archive Modules</primary> +<chapter id="archive-and-restore-modules"> + <title>Archive and Restore Modules</title> + <indexterm zone="archive-and-restore-modules"> + <primary>Archive and Restore Modules</primary> </indexterm> <para> @@ -14,45 +14,52 @@ performant. </para> - <para> - When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL - will submit completed WAL files to the module, and the server will avoid - recycling or removing these WAL files until the module indicates that the files - were successfully archived. It is ultimately up to the module to decide what - to do with each WAL file, but many recommendations are listed at - <xref linkend="backup-archiving-wal"/>. - </para> - - <para> - Archiving modules must at least consist of an initialization function (see - <xref linkend="archive-module-init"/>) and the required callbacks (see - <xref linkend="archive-module-callbacks"/>). However, archive modules are - also permitted to do much more (e.g., declare GUCs and register background - workers). - </para> - <para> The <filename>contrib/basic_archive</filename> module contains a working example, which demonstrates some useful techniques. </para> - <sect1 id="archive-module-init"> - <title>Initialization Functions</title> - <indexterm zone="archive-module-init"> - <primary>_PG_archive_module_init</primary> + <sect1 id="archive-modules"> + <title>Archive Modules</title> + <indexterm zone="archive-modules"> + <primary>Archive Modules</primary> </indexterm> + + <para> + When a custom <xref linkend="guc-archive-library"/> is configured, + PostgreSQL will submit completed WAL files to the module, and the server + will avoid recycling or removing these WAL files until the module indicates + that the files were successfully archived. It is ultimately up to the + module to decide what to do with each WAL file, but many recommendations are + listed at <xref linkend="backup-archiving-wal"/>. + </para> + <para> - An archive library is loaded by dynamically loading a shared library with the - <xref linkend="guc-archive-library"/>'s name as the library base name. The - normal library search path is used to locate the library. To provide the - required archive module callbacks and to indicate that the library is - actually an archive module, it needs to provide a function named - <function>_PG_archive_module_init</function>. The result of the function - must be a pointer to a struct of type - <structname>ArchiveModuleCallbacks</structname>, which contains everything - that the core code needs to know to make use of the archive module. The - return value needs to be of server lifetime, which is typically achieved by - defining it as a <literal>static const</literal> variable in global scope. + Archiving modules must at least consist of an initialization function (see + <xref linkend="archive-module-init"/>) and the required callbacks (see + <xref linkend="archive-module-callbacks"/>). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + </para> + + <sect2 id="archive-module-init"> + <title>Initialization Functions</title> + <indexterm zone="archive-module-init"> + <primary>_PG_archive_module_init</primary> + </indexterm> + + <para> + An archive library is loaded by dynamically loading a shared library with + the <xref linkend="guc-archive-library"/>'s name as the library base name. + The normal library search path is used to locate the library. To provide + the required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + <function>_PG_archive_module_init</function>. The result of the function + must be a pointer to a struct of type + <structname>ArchiveModuleCallbacks</structname>, which contains everything + that the core code needs to know to make use of the archive module. The + return value needs to be of server lifetime, which is typically achieved by + defining it as a <literal>static const</literal> variable in global scope. <programlisting> typedef struct ArchiveModuleCallbacks @@ -65,103 +72,110 @@ typedef struct ArchiveModuleCallbacks typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void); </programlisting> - Only the <function>archive_file_cb</function> callback is required. The - others are optional. - </para> - </sect1> + Only the <function>archive_file_cb</function> callback is required. The + others are optional. + </para> + </sect2> - <sect1 id="archive-module-callbacks"> - <title>Archive Module Callbacks</title> - <para> - The archive callbacks define the actual archiving behavior of the module. - The server will call them as required to process each individual WAL file. - </para> + <sect2 id="archive-module-callbacks"> + <title>Archive Module Callbacks</title> - <sect2 id="archive-module-startup"> - <title>Startup Callback</title> <para> - The <function>startup_cb</function> callback is called shortly after the - module is loaded. This callback can be used to perform any additional - initialization required. If the archive module has any state, it can use - <structfield>state->private_data</structfield> to store it. + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + </para> + + <sect3 id="archive-module-startup"> + <title>Startup Callback</title> + + <para> + The <function>startup_cb</function> callback is called shortly after the + module is loaded. This callback can be used to perform any additional + initialization required. If the archive module has any state, it can use + <structfield>state->private_data</structfield> to store it. <programlisting> typedef void (*ArchiveStartupCB) (ArchiveModuleState *state); </programlisting> - </para> - </sect2> + </para> + </sect3> - <sect2 id="archive-module-check"> - <title>Check Callback</title> - <para> - The <function>check_configured_cb</function> callback is called to determine - whether the module is fully configured and ready to accept WAL files (e.g., - its configuration parameters are set to valid values). If no - <function>check_configured_cb</function> is defined, the server always - assumes the module is configured. + <sect3 id="archive-module-check"> + <title>Check Callback</title> + + <para> + The <function>check_configured_cb</function> callback is called to + determine whether the module is fully configured and ready to accept WAL + files (e.g., its configuration parameters are set to valid values). If no + <function>check_configured_cb</function> is defined, the server always + assumes the module is configured. <programlisting> typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); </programlisting> - If <literal>true</literal> is returned, the server will proceed with - archiving the file by calling the <function>archive_file_cb</function> - callback. If <literal>false</literal> is returned, archiving will not - proceed, and the archiver will emit the following message to the server log: + If <literal>true</literal> is returned, the server will proceed with + archiving the file by calling the <function>archive_file_cb</function> + callback. If <literal>false</literal> is returned, archiving will not + proceed, and the archiver will emit the following message to the server + log: <screen> WARNING: archive_mode enabled, yet archiving is not configured </screen> - In the latter case, the server will periodically call this function, and - archiving will proceed only when it returns <literal>true</literal>. - </para> - - <note> - <para> - When returning <literal>false</literal>, it may be useful to append some - additional information to the generic warning message. To do that, provide - a message to the <function>arch_module_check_errdetail</function> macro - before returning <literal>false</literal>. Like - <function>errdetail()</function>, this macro accepts a format string - followed by an optional list of arguments. The resulting string will be - emitted as the <literal>DETAIL</literal> line of the warning message. + In the latter case, the server will periodically call this function, and + archiving will proceed only when it returns <literal>true</literal>. </para> - </note> - </sect2> - <sect2 id="archive-module-archive"> - <title>Archive Callback</title> - <para> - The <function>archive_file_cb</function> callback is called to archive a - single WAL file. + <note> + <para> + When returning <literal>false</literal>, it may be useful to append some + additional information to the generic warning message. To do that, + provide a message to the <function>arch_module_check_errdetail</function> + macro before returning <literal>false</literal>. Like + <function>errdetail()</function>, this macro accepts a format string + followed by an optional list of arguments. The resulting string will be + emitted as the <literal>DETAIL</literal> line of the warning message. + </para> + </note> + </sect3> + + <sect3 id="archive-module-archive"> + <title>Archive Callback</title> + + <para> + The <function>archive_file_cb</function> callback is called to archive a + single WAL file. <programlisting> typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path); </programlisting> - If <literal>true</literal> is returned, the server proceeds as if the file - was successfully archived, which may include recycling or removing the - original WAL file. If <literal>false</literal> is returned, the server will - keep the original WAL file and retry archiving later. - <replaceable>file</replaceable> will contain just the file name of the WAL - file to archive, while <replaceable>path</replaceable> contains the full - path of the WAL file (including the file name). - </para> - </sect2> + If <literal>true</literal> is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If <literal>false</literal> is returned, the server + will keep the original WAL file and retry archiving later. + <replaceable>file</replaceable> will contain just the file name of the WAL + file to archive, while <replaceable>path</replaceable> contains the full + path of the WAL file (including the file name). + </para> + </sect3> - <sect2 id="archive-module-shutdown"> - <title>Shutdown Callback</title> - <para> - The <function>shutdown_cb</function> callback is called when the archiver - process exits (e.g., after an error) or the value of - <xref linkend="guc-archive-library"/> changes. If no - <function>shutdown_cb</function> is defined, no special action is taken in - these situations. If the archive module has any state, this callback should - free it to avoid leaks. + <sect3 id="archive-module-shutdown"> + <title>Shutdown Callback</title> + + <para> + The <function>shutdown_cb</function> callback is called when the archiver + process exits (e.g., after an error) or the value of + <xref linkend="guc-archive-library"/> changes. If no + <function>shutdown_cb</function> is defined, no special action is taken in + these situations. If the archive module has any state, this callback + should free it to avoid leaks. <programlisting> typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state); </programlisting> - </para> + </para> + </sect3> </sect2> </sect1> </chapter> -- 2.25.1 --IJpNTDwzlM2Ie8A6 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v18-0005-introduce-restore_library.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v14 6/7] restructure archive modules docs in preparation for restore modules @ 2023-02-15 22:48 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Nathan Bossart @ 2023-02-15 22:48 UTC (permalink / raw) --- doc/src/sgml/archive-and-restore-modules.sgml | 207 ++++++++++-------- 1 file changed, 111 insertions(+), 96 deletions(-) diff --git a/doc/src/sgml/archive-and-restore-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml index 7cf44e82e2..f30ee591e7 100644 --- a/doc/src/sgml/archive-and-restore-modules.sgml +++ b/doc/src/sgml/archive-and-restore-modules.sgml @@ -1,9 +1,9 @@ -<!-- doc/src/sgml/archive-modules.sgml --> +<!-- doc/src/sgml/archive-and-restore-modules.sgml --> -<chapter id="archive-modules"> - <title>Archive Modules</title> - <indexterm zone="archive-modules"> - <primary>Archive Modules</primary> +<chapter id="archive-and-restore-modules"> + <title>Archive and Restore Modules</title> + <indexterm zone="archive-and-restore-modules"> + <primary>Archive and Restore Modules</primary> </indexterm> <para> @@ -14,45 +14,53 @@ performant. </para> - <para> - When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL - will submit completed WAL files to the module, and the server will avoid - recycling or removing these WAL files until the module indicates that the files - were successfully archived. It is ultimately up to the module to decide what - to do with each WAL file, but many recommendations are listed at - <xref linkend="backup-archiving-wal"/>. - </para> - - <para> - Archiving modules must at least consist of an initialization function (see - <xref linkend="archive-module-init"/>) and the required callbacks (see - <xref linkend="archive-module-callbacks"/>). However, archive modules are - also permitted to do much more (e.g., declare GUCs and register background - workers). - </para> - <para> The <filename>contrib/basic_archive</filename> module contains a working example, which demonstrates some useful techniques. </para> - <sect1 id="archive-module-init"> - <title>Initialization Functions</title> - <indexterm zone="archive-module-init"> - <primary>_PG_archive_module_init</primary> + <sect1 id="archive-modules"> + <title>Archive Modules</title> + <indexterm zone="archive-modules"> + <primary>Archive Modules</primary> </indexterm> + + <para> + When a custom <xref linkend="guc-archive-library"/> is configured, + PostgreSQL will submit completed WAL files to the module, and the server + will avoid recycling or removing these WAL files until the module indicates + that the files were successfully archived. It is ultimately up to the + module to decide what to do with each WAL file, but many recommendations are + listed at <xref linkend="backup-archiving-wal"/>. + </para> + <para> - An archive library is loaded by dynamically loading a shared library with the - <xref linkend="guc-archive-library"/>'s name as the library base name. The - normal library search path is used to locate the library. To provide the - required archive module callbacks and to indicate that the library is - actually an archive module, it needs to provide a function named - <function>_PG_archive_module_init</function>. The result of the function - must be a pointer to a struct of type - <structname>ArchiveModuleCallbacks</structname>, which contains everything - that the core code needs to know how to make use of the archive module. The - return value needs to be of server lifetime, which is typically achieved by - defining it as a <literal>static const</literal> variable in global scope. + Archiving modules must at least consist of an initialization function (see + <xref linkend="archive-module-init"/>) and the required callbacks (see + <xref linkend="archive-module-callbacks"/>). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + </para> + + <sect2 id="archive-module-init"> + <title>Initialization Functions</title> + <indexterm zone="archive-module-init"> + <primary>_PG_archive_module_init</primary> + </indexterm> + + <para> + An archive library is loaded by dynamically loading a shared library with + the <xref linkend="guc-archive-library"/>'s name as the library base name. + The normal library search path is used to locate the library. To provide + the required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + <function>_PG_archive_module_init</function>. The result of the function + must be a pointer to a struct of type + <structname>ArchiveModuleCallbacks</structname>, which contains everything + that the core code needs to know how to make use of the archive module. + The return value needs to be of server lifetime, which is typically + achieved by defining it as a <literal>static const</literal> variable in + global scope. <programlisting> typedef struct ArchiveModuleCallbacks @@ -65,91 +73,98 @@ typedef struct ArchiveModuleCallbacks typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void); </programlisting> - Only the <function>archive_file_cb</function> callback is required. The - others are optional. - </para> - </sect1> + Only the <function>archive_file_cb</function> callback is required. The + others are optional. + </para> + </sect2> - <sect1 id="archive-module-callbacks"> - <title>Archive Module Callbacks</title> - <para> - The archive callbacks define the actual archiving behavior of the module. - The server will call them as required to process each individual WAL file. - </para> + <sect2 id="archive-module-callbacks"> + <title>Archive Module Callbacks</title> - <sect2 id="archive-module-startup"> - <title>Startup Callback</title> <para> - The <function>startup_cb</function> callback is called shortly after the - module is loaded. This callback can be used to perform any additional - initialization required. If the archive module has a state, it can use - <structfield>state->private_data</structfield> to store it. + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + </para> + + <sect3 id="archive-module-startup"> + <title>Startup Callback</title> + + <para> + The <function>startup_cb</function> callback is called shortly after the + module is loaded. This callback can be used to perform any additional + initialization required. If the archive module has state, it can use + <structfield>state->private_data</structfield> to store it. <programlisting> typedef void (*ArchiveStartupCB) (ArchiveModuleState *state); </programlisting> - </para> - </sect2> + </para> + </sect3> - <sect2 id="archive-module-check"> - <title>Check Callback</title> - <para> - The <function>check_configured_cb</function> callback is called to determine - whether the module is fully configured and ready to accept WAL files (e.g., - its configuration parameters are set to valid values). If no - <function>check_configured_cb</function> is defined, the server always - assumes the module is configured. + <sect3 id="archive-module-check"> + <title>Check Callback</title> + + <para> + The <function>check_configured_cb</function> callback is called to + determine whether the module is fully configured and ready to accept WAL + files (e.g., its configuration parameters are set to valid values). If no + <function>check_configured_cb</function> is defined, the server always + assumes the module is configured. <programlisting> typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); </programlisting> - If <literal>true</literal> is returned, the server will proceed with - archiving the file by calling the <function>archive_file_cb</function> - callback. If <literal>false</literal> is returned, archiving will not - proceed, and the archiver will emit the following message to the server log: + If <literal>true</literal> is returned, the server will proceed with + archiving the file by calling the <function>archive_file_cb</function> + callback. If <literal>false</literal> is returned, archiving will not + proceed, and the archiver will emit the following message to the server + log: <screen> WARNING: archive_mode enabled, yet archiving is not configured </screen> - In the latter case, the server will periodically call this function, and - archiving will proceed only when it returns <literal>true</literal>. - </para> - </sect2> + In the latter case, the server will periodically call this function, and + archiving will proceed only when it returns <literal>true</literal>. + </para> + </sect3> - <sect2 id="archive-module-archive"> - <title>Archive Callback</title> - <para> - The <function>archive_file_cb</function> callback is called to archive a - single WAL file. + <sect3 id="archive-module-archive"> + <title>Archive Callback</title> + + <para> + The <function>archive_file_cb</function> callback is called to archive a + single WAL file. <programlisting> typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path); </programlisting> - If <literal>true</literal> is returned, the server proceeds as if the file - was successfully archived, which may include recycling or removing the - original WAL file. If <literal>false</literal> is returned, the server will - keep the original WAL file and retry archiving later. - <replaceable>file</replaceable> will contain just the file name of the WAL - file to archive, while <replaceable>path</replaceable> contains the full - path of the WAL file (including the file name). - </para> - </sect2> - - <sect2 id="archive-module-shutdown"> - <title>Shutdown Callback</title> - <para> - The <function>shutdown_cb</function> callback is called when the archiver - process exits (e.g., after an error) or the value of - <xref linkend="guc-archive-library"/> changes. If no - <function>shutdown_cb</function> is defined, no special action is taken in - these situations. If the archive module has a state, this callback should - free it to avoid leaks. + If <literal>true</literal> is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If <literal>false</literal> is returned, the server + will keep the original WAL file and retry archiving later. + <replaceable>file</replaceable> will contain just the file name of the WAL + file to archive, while <replaceable>path</replaceable> contains the full + path of the WAL file (including the file name). + </para> + </sect3> + + <sect3 id="archive-module-shutdown"> + <title>Shutdown Callback</title> + + <para> + The <function>shutdown_cb</function> callback is called when the archiver + process exits (e.g., after an error) or the value of + <xref linkend="guc-archive-library"/> changes. If no + <function>shutdown_cb</function> is defined, no special action is taken in + these situations. If the archive module has state, this callback should + free it to avoid leaks. <programlisting> typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state); </programlisting> - </para> + </para> + </sect3> </sect2> </sect1> </chapter> -- 2.25.1 --IS0zKkzwUGydFO0o Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v14-0007-introduce-restore_library.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v13 6/7] restructure archive modules docs in preparation for restore modules @ 2023-02-15 22:48 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Nathan Bossart @ 2023-02-15 22:48 UTC (permalink / raw) --- doc/src/sgml/archive-and-restore-modules.sgml | 207 ++++++++++-------- 1 file changed, 111 insertions(+), 96 deletions(-) diff --git a/doc/src/sgml/archive-and-restore-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml index 7cf44e82e2..f30ee591e7 100644 --- a/doc/src/sgml/archive-and-restore-modules.sgml +++ b/doc/src/sgml/archive-and-restore-modules.sgml @@ -1,9 +1,9 @@ -<!-- doc/src/sgml/archive-modules.sgml --> +<!-- doc/src/sgml/archive-and-restore-modules.sgml --> -<chapter id="archive-modules"> - <title>Archive Modules</title> - <indexterm zone="archive-modules"> - <primary>Archive Modules</primary> +<chapter id="archive-and-restore-modules"> + <title>Archive and Restore Modules</title> + <indexterm zone="archive-and-restore-modules"> + <primary>Archive and Restore Modules</primary> </indexterm> <para> @@ -14,45 +14,53 @@ performant. </para> - <para> - When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL - will submit completed WAL files to the module, and the server will avoid - recycling or removing these WAL files until the module indicates that the files - were successfully archived. It is ultimately up to the module to decide what - to do with each WAL file, but many recommendations are listed at - <xref linkend="backup-archiving-wal"/>. - </para> - - <para> - Archiving modules must at least consist of an initialization function (see - <xref linkend="archive-module-init"/>) and the required callbacks (see - <xref linkend="archive-module-callbacks"/>). However, archive modules are - also permitted to do much more (e.g., declare GUCs and register background - workers). - </para> - <para> The <filename>contrib/basic_archive</filename> module contains a working example, which demonstrates some useful techniques. </para> - <sect1 id="archive-module-init"> - <title>Initialization Functions</title> - <indexterm zone="archive-module-init"> - <primary>_PG_archive_module_init</primary> + <sect1 id="archive-modules"> + <title>Archive Modules</title> + <indexterm zone="archive-modules"> + <primary>Archive Modules</primary> </indexterm> + + <para> + When a custom <xref linkend="guc-archive-library"/> is configured, + PostgreSQL will submit completed WAL files to the module, and the server + will avoid recycling or removing these WAL files until the module indicates + that the files were successfully archived. It is ultimately up to the + module to decide what to do with each WAL file, but many recommendations are + listed at <xref linkend="backup-archiving-wal"/>. + </para> + <para> - An archive library is loaded by dynamically loading a shared library with the - <xref linkend="guc-archive-library"/>'s name as the library base name. The - normal library search path is used to locate the library. To provide the - required archive module callbacks and to indicate that the library is - actually an archive module, it needs to provide a function named - <function>_PG_archive_module_init</function>. The result of the function - must be a pointer to a struct of type - <structname>ArchiveModuleCallbacks</structname>, which contains everything - that the core code needs to know how to make use of the archive module. The - return value needs to be of server lifetime, which is typically achieved by - defining it as a <literal>static const</literal> variable in global scope. + Archiving modules must at least consist of an initialization function (see + <xref linkend="archive-module-init"/>) and the required callbacks (see + <xref linkend="archive-module-callbacks"/>). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + </para> + + <sect2 id="archive-module-init"> + <title>Initialization Functions</title> + <indexterm zone="archive-module-init"> + <primary>_PG_archive_module_init</primary> + </indexterm> + + <para> + An archive library is loaded by dynamically loading a shared library with + the <xref linkend="guc-archive-library"/>'s name as the library base name. + The normal library search path is used to locate the library. To provide + the required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + <function>_PG_archive_module_init</function>. The result of the function + must be a pointer to a struct of type + <structname>ArchiveModuleCallbacks</structname>, which contains everything + that the core code needs to know how to make use of the archive module. + The return value needs to be of server lifetime, which is typically + achieved by defining it as a <literal>static const</literal> variable in + global scope. <programlisting> typedef struct ArchiveModuleCallbacks @@ -65,91 +73,98 @@ typedef struct ArchiveModuleCallbacks typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void); </programlisting> - Only the <function>archive_file_cb</function> callback is required. The - others are optional. - </para> - </sect1> + Only the <function>archive_file_cb</function> callback is required. The + others are optional. + </para> + </sect2> - <sect1 id="archive-module-callbacks"> - <title>Archive Module Callbacks</title> - <para> - The archive callbacks define the actual archiving behavior of the module. - The server will call them as required to process each individual WAL file. - </para> + <sect2 id="archive-module-callbacks"> + <title>Archive Module Callbacks</title> - <sect2 id="archive-module-startup"> - <title>Startup Callback</title> <para> - The <function>startup_cb</function> callback is called shortly after the - module is loaded. This callback can be used to perform any additional - initialization required. If the archive module has a state, it can use - <structfield>state->private_data</structfield> to store it. + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + </para> + + <sect3 id="archive-module-startup"> + <title>Startup Callback</title> + + <para> + The <function>startup_cb</function> callback is called shortly after the + module is loaded. This callback can be used to perform any additional + initialization required. If the archive module has state, it can use + <structfield>state->private_data</structfield> to store it. <programlisting> typedef void (*ArchiveStartupCB) (ArchiveModuleState *state); </programlisting> - </para> - </sect2> + </para> + </sect3> - <sect2 id="archive-module-check"> - <title>Check Callback</title> - <para> - The <function>check_configured_cb</function> callback is called to determine - whether the module is fully configured and ready to accept WAL files (e.g., - its configuration parameters are set to valid values). If no - <function>check_configured_cb</function> is defined, the server always - assumes the module is configured. + <sect3 id="archive-module-check"> + <title>Check Callback</title> + + <para> + The <function>check_configured_cb</function> callback is called to + determine whether the module is fully configured and ready to accept WAL + files (e.g., its configuration parameters are set to valid values). If no + <function>check_configured_cb</function> is defined, the server always + assumes the module is configured. <programlisting> typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); </programlisting> - If <literal>true</literal> is returned, the server will proceed with - archiving the file by calling the <function>archive_file_cb</function> - callback. If <literal>false</literal> is returned, archiving will not - proceed, and the archiver will emit the following message to the server log: + If <literal>true</literal> is returned, the server will proceed with + archiving the file by calling the <function>archive_file_cb</function> + callback. If <literal>false</literal> is returned, archiving will not + proceed, and the archiver will emit the following message to the server + log: <screen> WARNING: archive_mode enabled, yet archiving is not configured </screen> - In the latter case, the server will periodically call this function, and - archiving will proceed only when it returns <literal>true</literal>. - </para> - </sect2> + In the latter case, the server will periodically call this function, and + archiving will proceed only when it returns <literal>true</literal>. + </para> + </sect3> - <sect2 id="archive-module-archive"> - <title>Archive Callback</title> - <para> - The <function>archive_file_cb</function> callback is called to archive a - single WAL file. + <sect3 id="archive-module-archive"> + <title>Archive Callback</title> + + <para> + The <function>archive_file_cb</function> callback is called to archive a + single WAL file. <programlisting> typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path); </programlisting> - If <literal>true</literal> is returned, the server proceeds as if the file - was successfully archived, which may include recycling or removing the - original WAL file. If <literal>false</literal> is returned, the server will - keep the original WAL file and retry archiving later. - <replaceable>file</replaceable> will contain just the file name of the WAL - file to archive, while <replaceable>path</replaceable> contains the full - path of the WAL file (including the file name). - </para> - </sect2> - - <sect2 id="archive-module-shutdown"> - <title>Shutdown Callback</title> - <para> - The <function>shutdown_cb</function> callback is called when the archiver - process exits (e.g., after an error) or the value of - <xref linkend="guc-archive-library"/> changes. If no - <function>shutdown_cb</function> is defined, no special action is taken in - these situations. If the archive module has a state, this callback should - free it to avoid leaks. + If <literal>true</literal> is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If <literal>false</literal> is returned, the server + will keep the original WAL file and retry archiving later. + <replaceable>file</replaceable> will contain just the file name of the WAL + file to archive, while <replaceable>path</replaceable> contains the full + path of the WAL file (including the file name). + </para> + </sect3> + + <sect3 id="archive-module-shutdown"> + <title>Shutdown Callback</title> + + <para> + The <function>shutdown_cb</function> callback is called when the archiver + process exits (e.g., after an error) or the value of + <xref linkend="guc-archive-library"/> changes. If no + <function>shutdown_cb</function> is defined, no special action is taken in + these situations. If the archive module has state, this callback should + free it to avoid leaks. <programlisting> typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state); </programlisting> - </para> + </para> + </sect3> </sect2> </sect1> </chapter> -- 2.25.1 --5vNYLRcllDrimb99 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13-0007-introduce-restore_library.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v12 5/6] restructure archive modules docs in preparation for restore modules @ 2023-02-15 22:48 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Nathan Bossart @ 2023-02-15 22:48 UTC (permalink / raw) --- doc/src/sgml/archive-and-restore-modules.sgml | 207 ++++++++++-------- 1 file changed, 111 insertions(+), 96 deletions(-) diff --git a/doc/src/sgml/archive-and-restore-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml index 7cf44e82e2..f30ee591e7 100644 --- a/doc/src/sgml/archive-and-restore-modules.sgml +++ b/doc/src/sgml/archive-and-restore-modules.sgml @@ -1,9 +1,9 @@ -<!-- doc/src/sgml/archive-modules.sgml --> +<!-- doc/src/sgml/archive-and-restore-modules.sgml --> -<chapter id="archive-modules"> - <title>Archive Modules</title> - <indexterm zone="archive-modules"> - <primary>Archive Modules</primary> +<chapter id="archive-and-restore-modules"> + <title>Archive and Restore Modules</title> + <indexterm zone="archive-and-restore-modules"> + <primary>Archive and Restore Modules</primary> </indexterm> <para> @@ -14,45 +14,53 @@ performant. </para> - <para> - When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL - will submit completed WAL files to the module, and the server will avoid - recycling or removing these WAL files until the module indicates that the files - were successfully archived. It is ultimately up to the module to decide what - to do with each WAL file, but many recommendations are listed at - <xref linkend="backup-archiving-wal"/>. - </para> - - <para> - Archiving modules must at least consist of an initialization function (see - <xref linkend="archive-module-init"/>) and the required callbacks (see - <xref linkend="archive-module-callbacks"/>). However, archive modules are - also permitted to do much more (e.g., declare GUCs and register background - workers). - </para> - <para> The <filename>contrib/basic_archive</filename> module contains a working example, which demonstrates some useful techniques. </para> - <sect1 id="archive-module-init"> - <title>Initialization Functions</title> - <indexterm zone="archive-module-init"> - <primary>_PG_archive_module_init</primary> + <sect1 id="archive-modules"> + <title>Archive Modules</title> + <indexterm zone="archive-modules"> + <primary>Archive Modules</primary> </indexterm> + + <para> + When a custom <xref linkend="guc-archive-library"/> is configured, + PostgreSQL will submit completed WAL files to the module, and the server + will avoid recycling or removing these WAL files until the module indicates + that the files were successfully archived. It is ultimately up to the + module to decide what to do with each WAL file, but many recommendations are + listed at <xref linkend="backup-archiving-wal"/>. + </para> + <para> - An archive library is loaded by dynamically loading a shared library with the - <xref linkend="guc-archive-library"/>'s name as the library base name. The - normal library search path is used to locate the library. To provide the - required archive module callbacks and to indicate that the library is - actually an archive module, it needs to provide a function named - <function>_PG_archive_module_init</function>. The result of the function - must be a pointer to a struct of type - <structname>ArchiveModuleCallbacks</structname>, which contains everything - that the core code needs to know how to make use of the archive module. The - return value needs to be of server lifetime, which is typically achieved by - defining it as a <literal>static const</literal> variable in global scope. + Archiving modules must at least consist of an initialization function (see + <xref linkend="archive-module-init"/>) and the required callbacks (see + <xref linkend="archive-module-callbacks"/>). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + </para> + + <sect2 id="archive-module-init"> + <title>Initialization Functions</title> + <indexterm zone="archive-module-init"> + <primary>_PG_archive_module_init</primary> + </indexterm> + + <para> + An archive library is loaded by dynamically loading a shared library with + the <xref linkend="guc-archive-library"/>'s name as the library base name. + The normal library search path is used to locate the library. To provide + the required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + <function>_PG_archive_module_init</function>. The result of the function + must be a pointer to a struct of type + <structname>ArchiveModuleCallbacks</structname>, which contains everything + that the core code needs to know how to make use of the archive module. + The return value needs to be of server lifetime, which is typically + achieved by defining it as a <literal>static const</literal> variable in + global scope. <programlisting> typedef struct ArchiveModuleCallbacks @@ -65,91 +73,98 @@ typedef struct ArchiveModuleCallbacks typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void); </programlisting> - Only the <function>archive_file_cb</function> callback is required. The - others are optional. - </para> - </sect1> + Only the <function>archive_file_cb</function> callback is required. The + others are optional. + </para> + </sect2> - <sect1 id="archive-module-callbacks"> - <title>Archive Module Callbacks</title> - <para> - The archive callbacks define the actual archiving behavior of the module. - The server will call them as required to process each individual WAL file. - </para> + <sect2 id="archive-module-callbacks"> + <title>Archive Module Callbacks</title> - <sect2 id="archive-module-startup"> - <title>Startup Callback</title> <para> - The <function>startup_cb</function> callback is called shortly after the - module is loaded. This callback can be used to perform any additional - initialization required. If the archive module has a state, it can use - <structfield>state->private_data</structfield> to store it. + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + </para> + + <sect3 id="archive-module-startup"> + <title>Startup Callback</title> + + <para> + The <function>startup_cb</function> callback is called shortly after the + module is loaded. This callback can be used to perform any additional + initialization required. If the archive module has state, it can use + <structfield>state->private_data</structfield> to store it. <programlisting> typedef void (*ArchiveStartupCB) (ArchiveModuleState *state); </programlisting> - </para> - </sect2> + </para> + </sect3> - <sect2 id="archive-module-check"> - <title>Check Callback</title> - <para> - The <function>check_configured_cb</function> callback is called to determine - whether the module is fully configured and ready to accept WAL files (e.g., - its configuration parameters are set to valid values). If no - <function>check_configured_cb</function> is defined, the server always - assumes the module is configured. + <sect3 id="archive-module-check"> + <title>Check Callback</title> + + <para> + The <function>check_configured_cb</function> callback is called to + determine whether the module is fully configured and ready to accept WAL + files (e.g., its configuration parameters are set to valid values). If no + <function>check_configured_cb</function> is defined, the server always + assumes the module is configured. <programlisting> typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); </programlisting> - If <literal>true</literal> is returned, the server will proceed with - archiving the file by calling the <function>archive_file_cb</function> - callback. If <literal>false</literal> is returned, archiving will not - proceed, and the archiver will emit the following message to the server log: + If <literal>true</literal> is returned, the server will proceed with + archiving the file by calling the <function>archive_file_cb</function> + callback. If <literal>false</literal> is returned, archiving will not + proceed, and the archiver will emit the following message to the server + log: <screen> WARNING: archive_mode enabled, yet archiving is not configured </screen> - In the latter case, the server will periodically call this function, and - archiving will proceed only when it returns <literal>true</literal>. - </para> - </sect2> + In the latter case, the server will periodically call this function, and + archiving will proceed only when it returns <literal>true</literal>. + </para> + </sect3> - <sect2 id="archive-module-archive"> - <title>Archive Callback</title> - <para> - The <function>archive_file_cb</function> callback is called to archive a - single WAL file. + <sect3 id="archive-module-archive"> + <title>Archive Callback</title> + + <para> + The <function>archive_file_cb</function> callback is called to archive a + single WAL file. <programlisting> typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path); </programlisting> - If <literal>true</literal> is returned, the server proceeds as if the file - was successfully archived, which may include recycling or removing the - original WAL file. If <literal>false</literal> is returned, the server will - keep the original WAL file and retry archiving later. - <replaceable>file</replaceable> will contain just the file name of the WAL - file to archive, while <replaceable>path</replaceable> contains the full - path of the WAL file (including the file name). - </para> - </sect2> - - <sect2 id="archive-module-shutdown"> - <title>Shutdown Callback</title> - <para> - The <function>shutdown_cb</function> callback is called when the archiver - process exits (e.g., after an error) or the value of - <xref linkend="guc-archive-library"/> changes. If no - <function>shutdown_cb</function> is defined, no special action is taken in - these situations. If the archive module has a state, this callback should - free it to avoid leaks. + If <literal>true</literal> is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If <literal>false</literal> is returned, the server + will keep the original WAL file and retry archiving later. + <replaceable>file</replaceable> will contain just the file name of the WAL + file to archive, while <replaceable>path</replaceable> contains the full + path of the WAL file (including the file name). + </para> + </sect3> + + <sect3 id="archive-module-shutdown"> + <title>Shutdown Callback</title> + + <para> + The <function>shutdown_cb</function> callback is called when the archiver + process exits (e.g., after an error) or the value of + <xref linkend="guc-archive-library"/> changes. If no + <function>shutdown_cb</function> is defined, no special action is taken in + these situations. If the archive module has state, this callback should + free it to avoid leaks. <programlisting> typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state); </programlisting> - </para> + </para> + </sect3> </sect2> </sect1> </chapter> -- 2.25.1 --YiEDa0DAkWCtVeE4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v12-0006-introduce-restore_library.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v15 6/7] restructure archive modules docs in preparation for restore modules @ 2023-02-15 22:48 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Nathan Bossart @ 2023-02-15 22:48 UTC (permalink / raw) --- doc/src/sgml/archive-and-restore-modules.sgml | 206 ++++++++++-------- 1 file changed, 110 insertions(+), 96 deletions(-) diff --git a/doc/src/sgml/archive-and-restore-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml index 7064307d9e..7ed50a3b52 100644 --- a/doc/src/sgml/archive-and-restore-modules.sgml +++ b/doc/src/sgml/archive-and-restore-modules.sgml @@ -1,9 +1,9 @@ -<!-- doc/src/sgml/archive-modules.sgml --> +<!-- doc/src/sgml/archive-and-restore-modules.sgml --> -<chapter id="archive-modules"> - <title>Archive Modules</title> - <indexterm zone="archive-modules"> - <primary>Archive Modules</primary> +<chapter id="archive-and-restore-modules"> + <title>Archive and Restore Modules</title> + <indexterm zone="archive-and-restore-modules"> + <primary>Archive and Restore Modules</primary> </indexterm> <para> @@ -14,45 +14,52 @@ performant. </para> - <para> - When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL - will submit completed WAL files to the module, and the server will avoid - recycling or removing these WAL files until the module indicates that the files - were successfully archived. It is ultimately up to the module to decide what - to do with each WAL file, but many recommendations are listed at - <xref linkend="backup-archiving-wal"/>. - </para> - - <para> - Archiving modules must at least consist of an initialization function (see - <xref linkend="archive-module-init"/>) and the required callbacks (see - <xref linkend="archive-module-callbacks"/>). However, archive modules are - also permitted to do much more (e.g., declare GUCs and register background - workers). - </para> - <para> The <filename>contrib/basic_archive</filename> module contains a working example, which demonstrates some useful techniques. </para> - <sect1 id="archive-module-init"> - <title>Initialization Functions</title> - <indexterm zone="archive-module-init"> - <primary>_PG_archive_module_init</primary> + <sect1 id="archive-modules"> + <title>Archive Modules</title> + <indexterm zone="archive-modules"> + <primary>Archive Modules</primary> </indexterm> + + <para> + When a custom <xref linkend="guc-archive-library"/> is configured, + PostgreSQL will submit completed WAL files to the module, and the server + will avoid recycling or removing these WAL files until the module indicates + that the files were successfully archived. It is ultimately up to the + module to decide what to do with each WAL file, but many recommendations are + listed at <xref linkend="backup-archiving-wal"/>. + </para> + <para> - An archive library is loaded by dynamically loading a shared library with the - <xref linkend="guc-archive-library"/>'s name as the library base name. The - normal library search path is used to locate the library. To provide the - required archive module callbacks and to indicate that the library is - actually an archive module, it needs to provide a function named - <function>_PG_archive_module_init</function>. The result of the function - must be a pointer to a struct of type - <structname>ArchiveModuleCallbacks</structname>, which contains everything - that the core code needs to know to make use of the archive module. The - return value needs to be of server lifetime, which is typically achieved by - defining it as a <literal>static const</literal> variable in global scope. + Archiving modules must at least consist of an initialization function (see + <xref linkend="archive-module-init"/>) and the required callbacks (see + <xref linkend="archive-module-callbacks"/>). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + </para> + + <sect2 id="archive-module-init"> + <title>Initialization Functions</title> + <indexterm zone="archive-module-init"> + <primary>_PG_archive_module_init</primary> + </indexterm> + + <para> + An archive library is loaded by dynamically loading a shared library with + the <xref linkend="guc-archive-library"/>'s name as the library base name. + The normal library search path is used to locate the library. To provide + the required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + <function>_PG_archive_module_init</function>. The result of the function + must be a pointer to a struct of type + <structname>ArchiveModuleCallbacks</structname>, which contains everything + that the core code needs to know to make use of the archive module. The + return value needs to be of server lifetime, which is typically achieved by + defining it as a <literal>static const</literal> variable in global scope. <programlisting> typedef struct ArchiveModuleCallbacks @@ -65,91 +72,98 @@ typedef struct ArchiveModuleCallbacks typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void); </programlisting> - Only the <function>archive_file_cb</function> callback is required. The - others are optional. - </para> - </sect1> + Only the <function>archive_file_cb</function> callback is required. The + others are optional. + </para> + </sect2> - <sect1 id="archive-module-callbacks"> - <title>Archive Module Callbacks</title> - <para> - The archive callbacks define the actual archiving behavior of the module. - The server will call them as required to process each individual WAL file. - </para> + <sect2 id="archive-module-callbacks"> + <title>Archive Module Callbacks</title> - <sect2 id="archive-module-startup"> - <title>Startup Callback</title> <para> - The <function>startup_cb</function> callback is called shortly after the - module is loaded. This callback can be used to perform any additional - initialization required. If the archive module has any state, it can use - <structfield>state->private_data</structfield> to store it. + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + </para> + + <sect3 id="archive-module-startup"> + <title>Startup Callback</title> + + <para> + The <function>startup_cb</function> callback is called shortly after the + module is loaded. This callback can be used to perform any additional + initialization required. If the archive module has any state, it can use + <structfield>state->private_data</structfield> to store it. <programlisting> typedef void (*ArchiveStartupCB) (ArchiveModuleState *state); </programlisting> - </para> - </sect2> + </para> + </sect3> - <sect2 id="archive-module-check"> - <title>Check Callback</title> - <para> - The <function>check_configured_cb</function> callback is called to determine - whether the module is fully configured and ready to accept WAL files (e.g., - its configuration parameters are set to valid values). If no - <function>check_configured_cb</function> is defined, the server always - assumes the module is configured. + <sect3 id="archive-module-check"> + <title>Check Callback</title> + + <para> + The <function>check_configured_cb</function> callback is called to + determine whether the module is fully configured and ready to accept WAL + files (e.g., its configuration parameters are set to valid values). If no + <function>check_configured_cb</function> is defined, the server always + assumes the module is configured. <programlisting> typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); </programlisting> - If <literal>true</literal> is returned, the server will proceed with - archiving the file by calling the <function>archive_file_cb</function> - callback. If <literal>false</literal> is returned, archiving will not - proceed, and the archiver will emit the following message to the server log: + If <literal>true</literal> is returned, the server will proceed with + archiving the file by calling the <function>archive_file_cb</function> + callback. If <literal>false</literal> is returned, archiving will not + proceed, and the archiver will emit the following message to the server + log: <screen> WARNING: archive_mode enabled, yet archiving is not configured </screen> - In the latter case, the server will periodically call this function, and - archiving will proceed only when it returns <literal>true</literal>. - </para> - </sect2> + In the latter case, the server will periodically call this function, and + archiving will proceed only when it returns <literal>true</literal>. + </para> + </sect3> - <sect2 id="archive-module-archive"> - <title>Archive Callback</title> - <para> - The <function>archive_file_cb</function> callback is called to archive a - single WAL file. + <sect3 id="archive-module-archive"> + <title>Archive Callback</title> + + <para> + The <function>archive_file_cb</function> callback is called to archive a + single WAL file. <programlisting> typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path); </programlisting> - If <literal>true</literal> is returned, the server proceeds as if the file - was successfully archived, which may include recycling or removing the - original WAL file. If <literal>false</literal> is returned, the server will - keep the original WAL file and retry archiving later. - <replaceable>file</replaceable> will contain just the file name of the WAL - file to archive, while <replaceable>path</replaceable> contains the full - path of the WAL file (including the file name). - </para> - </sect2> - - <sect2 id="archive-module-shutdown"> - <title>Shutdown Callback</title> - <para> - The <function>shutdown_cb</function> callback is called when the archiver - process exits (e.g., after an error) or the value of - <xref linkend="guc-archive-library"/> changes. If no - <function>shutdown_cb</function> is defined, no special action is taken in - these situations. If the archive module has any state, this callback should - free it to avoid leaks. + If <literal>true</literal> is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If <literal>false</literal> is returned, the server + will keep the original WAL file and retry archiving later. + <replaceable>file</replaceable> will contain just the file name of the WAL + file to archive, while <replaceable>path</replaceable> contains the full + path of the WAL file (including the file name). + </para> + </sect3> + + <sect3 id="archive-module-shutdown"> + <title>Shutdown Callback</title> + + <para> + The <function>shutdown_cb</function> callback is called when the archiver + process exits (e.g., after an error) or the value of + <xref linkend="guc-archive-library"/> changes. If no + <function>shutdown_cb</function> is defined, no special action is taken in + these situations. If the archive module has any state, this callback + should free it to avoid leaks. <programlisting> typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state); </programlisting> - </para> + </para> + </sect3> </sect2> </sect1> </chapter> -- 2.25.1 --k+w/mQv8wyuph6w0 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v15-0007-introduce-restore_library.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v20 4/5] restructure archive modules docs in preparation for restore modules @ 2023-02-15 22:48 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Nathan Bossart @ 2023-02-15 22:48 UTC (permalink / raw) --- doc/src/sgml/archive-and-restore-modules.sgml | 226 ++++++++++-------- 1 file changed, 120 insertions(+), 106 deletions(-) diff --git a/doc/src/sgml/archive-and-restore-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml index cf7438a759..a52d15082d 100644 --- a/doc/src/sgml/archive-and-restore-modules.sgml +++ b/doc/src/sgml/archive-and-restore-modules.sgml @@ -1,9 +1,9 @@ -<!-- doc/src/sgml/archive-modules.sgml --> +<!-- doc/src/sgml/archive-and-restore-modules.sgml --> -<chapter id="archive-modules"> - <title>Archive Modules</title> - <indexterm zone="archive-modules"> - <primary>Archive Modules</primary> +<chapter id="archive-and-restore-modules"> + <title>Archive and Restore Modules</title> + <indexterm zone="archive-and-restore-modules"> + <primary>Archive and Restore Modules</primary> </indexterm> <para> @@ -14,45 +14,52 @@ performant. </para> - <para> - When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL - will submit completed WAL files to the module, and the server will avoid - recycling or removing these WAL files until the module indicates that the files - were successfully archived. It is ultimately up to the module to decide what - to do with each WAL file, but many recommendations are listed at - <xref linkend="backup-archiving-wal"/>. - </para> - - <para> - Archiving modules must at least consist of an initialization function (see - <xref linkend="archive-module-init"/>) and the required callbacks (see - <xref linkend="archive-module-callbacks"/>). However, archive modules are - also permitted to do much more (e.g., declare GUCs and register background - workers). - </para> - <para> The <filename>contrib/basic_archive</filename> module contains a working example, which demonstrates some useful techniques. </para> - <sect1 id="archive-module-init"> - <title>Initialization Functions</title> - <indexterm zone="archive-module-init"> - <primary>_PG_archive_module_init</primary> + <sect1 id="archive-modules"> + <title>Archive Modules</title> + <indexterm zone="archive-modules"> + <primary>Archive Modules</primary> </indexterm> + + <para> + When a custom <xref linkend="guc-archive-library"/> is configured, + PostgreSQL will submit completed WAL files to the module, and the server + will avoid recycling or removing these WAL files until the module indicates + that the files were successfully archived. It is ultimately up to the + module to decide what to do with each WAL file, but many recommendations are + listed at <xref linkend="backup-archiving-wal"/>. + </para> + <para> - An archive library is loaded by dynamically loading a shared library with the - <xref linkend="guc-archive-library"/>'s name as the library base name. The - normal library search path is used to locate the library. To provide the - required archive module callbacks and to indicate that the library is - actually an archive module, it needs to provide a function named - <function>_PG_archive_module_init</function>. The result of the function - must be a pointer to a struct of type - <structname>ArchiveModuleCallbacks</structname>, which contains everything - that the core code needs to know to make use of the archive module. The - return value needs to be of server lifetime, which is typically achieved by - defining it as a <literal>static const</literal> variable in global scope. + Archiving modules must at least consist of an initialization function (see + <xref linkend="archive-module-init"/>) and the required callbacks (see + <xref linkend="archive-module-callbacks"/>). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + </para> + + <sect2 id="archive-module-init"> + <title>Initialization Functions</title> + <indexterm zone="archive-module-init"> + <primary>_PG_archive_module_init</primary> + </indexterm> + + <para> + An archive library is loaded by dynamically loading a shared library with + the <xref linkend="guc-archive-library"/>'s name as the library base name. + The normal library search path is used to locate the library. To provide + the required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + <function>_PG_archive_module_init</function>. The result of the function + must be a pointer to a struct of type + <structname>ArchiveModuleCallbacks</structname>, which contains everything + that the core code needs to know to make use of the archive module. The + return value needs to be of server lifetime, which is typically achieved by + defining it as a <literal>static const</literal> variable in global scope. <programlisting> typedef struct ArchiveModuleCallbacks @@ -65,103 +72,110 @@ typedef struct ArchiveModuleCallbacks typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void); </programlisting> - Only the <function>archive_file_cb</function> callback is required. The - others are optional. - </para> - </sect1> + Only the <function>archive_file_cb</function> callback is required. The + others are optional. + </para> + </sect2> - <sect1 id="archive-module-callbacks"> - <title>Archive Module Callbacks</title> - <para> - The archive callbacks define the actual archiving behavior of the module. - The server will call them as required to process each individual WAL file. - </para> + <sect2 id="archive-module-callbacks"> + <title>Archive Module Callbacks</title> - <sect2 id="archive-module-startup"> - <title>Startup Callback</title> <para> - The <function>startup_cb</function> callback is called shortly after the - module is loaded. This callback can be used to perform any additional - initialization required. If the archive module has any state, it can use - <structfield>state->private_data</structfield> to store it. + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + </para> + + <sect3 id="archive-module-startup"> + <title>Startup Callback</title> + + <para> + The <function>startup_cb</function> callback is called shortly after the + module is loaded. This callback can be used to perform any additional + initialization required. If the archive module has any state, it can use + <structfield>state->private_data</structfield> to store it. <programlisting> typedef void (*ArchiveStartupCB) (ArchiveModuleState *state); </programlisting> - </para> - </sect2> + </para> + </sect3> - <sect2 id="archive-module-check"> - <title>Check Callback</title> - <para> - The <function>check_configured_cb</function> callback is called to determine - whether the module is fully configured and ready to accept WAL files (e.g., - its configuration parameters are set to valid values). If no - <function>check_configured_cb</function> is defined, the server always - assumes the module is configured. + <sect3 id="archive-module-check"> + <title>Check Callback</title> + + <para> + The <function>check_configured_cb</function> callback is called to + determine whether the module is fully configured and ready to accept WAL + files (e.g., its configuration parameters are set to valid values). If no + <function>check_configured_cb</function> is defined, the server always + assumes the module is configured. <programlisting> typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); </programlisting> - If <literal>true</literal> is returned, the server will proceed with - archiving the file by calling the <function>archive_file_cb</function> - callback. If <literal>false</literal> is returned, archiving will not - proceed, and the archiver will emit the following message to the server log: + If <literal>true</literal> is returned, the server will proceed with + archiving the file by calling the <function>archive_file_cb</function> + callback. If <literal>false</literal> is returned, archiving will not + proceed, and the archiver will emit the following message to the server + log: <screen> WARNING: archive_mode enabled, yet archiving is not configured </screen> - In the latter case, the server will periodically call this function, and - archiving will proceed only when it returns <literal>true</literal>. - </para> - - <note> - <para> - When returning <literal>false</literal>, it may be useful to append some - additional information to the generic warning message. To do that, provide - a message to the <function>arch_module_check_errdetail</function> macro - before returning <literal>false</literal>. Like - <function>errdetail()</function>, this macro accepts a format string - followed by an optional list of arguments. The resulting string will be - emitted as the <literal>DETAIL</literal> line of the warning message. + In the latter case, the server will periodically call this function, and + archiving will proceed only when it returns <literal>true</literal>. </para> - </note> - </sect2> - <sect2 id="archive-module-archive"> - <title>Archive Callback</title> - <para> - The <function>archive_file_cb</function> callback is called to archive a - single WAL file. + <note> + <para> + When returning <literal>false</literal>, it may be useful to append some + additional information to the generic warning message. To do that, + provide a message to the <function>arch_module_check_errdetail</function> + macro before returning <literal>false</literal>. Like + <function>errdetail()</function>, this macro accepts a format string + followed by an optional list of arguments. The resulting string will be + emitted as the <literal>DETAIL</literal> line of the warning message. + </para> + </note> + </sect3> + + <sect3 id="archive-module-archive"> + <title>Archive Callback</title> + + <para> + The <function>archive_file_cb</function> callback is called to archive a + single WAL file. <programlisting> typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path); </programlisting> - If <literal>true</literal> is returned, the server proceeds as if the file - was successfully archived, which may include recycling or removing the - original WAL file. If <literal>false</literal> is returned, the server will - keep the original WAL file and retry archiving later. - <replaceable>file</replaceable> will contain just the file name of the WAL - file to archive, while <replaceable>path</replaceable> contains the full - path of the WAL file (including the file name). - </para> - </sect2> + If <literal>true</literal> is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If <literal>false</literal> is returned, the server + will keep the original WAL file and retry archiving later. + <replaceable>file</replaceable> will contain just the file name of the WAL + file to archive, while <replaceable>path</replaceable> contains the full + path of the WAL file (including the file name). + </para> + </sect3> - <sect2 id="archive-module-shutdown"> - <title>Shutdown Callback</title> - <para> - The <function>shutdown_cb</function> callback is called when the archiver - process exits (e.g., after an error) or the value of - <xref linkend="guc-archive-library"/> changes. If no - <function>shutdown_cb</function> is defined, no special action is taken in - these situations. If the archive module has any state, this callback should - free it to avoid leaks. + <sect3 id="archive-module-shutdown"> + <title>Shutdown Callback</title> + + <para> + The <function>shutdown_cb</function> callback is called when the archiver + process exits (e.g., after an error) or the value of + <xref linkend="guc-archive-library"/> changes. If no + <function>shutdown_cb</function> is defined, no special action is taken in + these situations. If the archive module has any state, this callback + should free it to avoid leaks. <programlisting> typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state); </programlisting> - </para> + </para> + </sect3> </sect2> </sect1> </chapter> -- 2.25.1 --3V7upXqbjpZ4EhLz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v20-0005-introduce-restore_library.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v19 4/5] restructure archive modules docs in preparation for restore modules @ 2023-02-15 22:48 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Nathan Bossart @ 2023-02-15 22:48 UTC (permalink / raw) --- doc/src/sgml/archive-and-restore-modules.sgml | 226 ++++++++++-------- 1 file changed, 120 insertions(+), 106 deletions(-) diff --git a/doc/src/sgml/archive-and-restore-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml index cf7438a759..a52d15082d 100644 --- a/doc/src/sgml/archive-and-restore-modules.sgml +++ b/doc/src/sgml/archive-and-restore-modules.sgml @@ -1,9 +1,9 @@ -<!-- doc/src/sgml/archive-modules.sgml --> +<!-- doc/src/sgml/archive-and-restore-modules.sgml --> -<chapter id="archive-modules"> - <title>Archive Modules</title> - <indexterm zone="archive-modules"> - <primary>Archive Modules</primary> +<chapter id="archive-and-restore-modules"> + <title>Archive and Restore Modules</title> + <indexterm zone="archive-and-restore-modules"> + <primary>Archive and Restore Modules</primary> </indexterm> <para> @@ -14,45 +14,52 @@ performant. </para> - <para> - When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL - will submit completed WAL files to the module, and the server will avoid - recycling or removing these WAL files until the module indicates that the files - were successfully archived. It is ultimately up to the module to decide what - to do with each WAL file, but many recommendations are listed at - <xref linkend="backup-archiving-wal"/>. - </para> - - <para> - Archiving modules must at least consist of an initialization function (see - <xref linkend="archive-module-init"/>) and the required callbacks (see - <xref linkend="archive-module-callbacks"/>). However, archive modules are - also permitted to do much more (e.g., declare GUCs and register background - workers). - </para> - <para> The <filename>contrib/basic_archive</filename> module contains a working example, which demonstrates some useful techniques. </para> - <sect1 id="archive-module-init"> - <title>Initialization Functions</title> - <indexterm zone="archive-module-init"> - <primary>_PG_archive_module_init</primary> + <sect1 id="archive-modules"> + <title>Archive Modules</title> + <indexterm zone="archive-modules"> + <primary>Archive Modules</primary> </indexterm> + + <para> + When a custom <xref linkend="guc-archive-library"/> is configured, + PostgreSQL will submit completed WAL files to the module, and the server + will avoid recycling or removing these WAL files until the module indicates + that the files were successfully archived. It is ultimately up to the + module to decide what to do with each WAL file, but many recommendations are + listed at <xref linkend="backup-archiving-wal"/>. + </para> + <para> - An archive library is loaded by dynamically loading a shared library with the - <xref linkend="guc-archive-library"/>'s name as the library base name. The - normal library search path is used to locate the library. To provide the - required archive module callbacks and to indicate that the library is - actually an archive module, it needs to provide a function named - <function>_PG_archive_module_init</function>. The result of the function - must be a pointer to a struct of type - <structname>ArchiveModuleCallbacks</structname>, which contains everything - that the core code needs to know to make use of the archive module. The - return value needs to be of server lifetime, which is typically achieved by - defining it as a <literal>static const</literal> variable in global scope. + Archiving modules must at least consist of an initialization function (see + <xref linkend="archive-module-init"/>) and the required callbacks (see + <xref linkend="archive-module-callbacks"/>). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + </para> + + <sect2 id="archive-module-init"> + <title>Initialization Functions</title> + <indexterm zone="archive-module-init"> + <primary>_PG_archive_module_init</primary> + </indexterm> + + <para> + An archive library is loaded by dynamically loading a shared library with + the <xref linkend="guc-archive-library"/>'s name as the library base name. + The normal library search path is used to locate the library. To provide + the required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + <function>_PG_archive_module_init</function>. The result of the function + must be a pointer to a struct of type + <structname>ArchiveModuleCallbacks</structname>, which contains everything + that the core code needs to know to make use of the archive module. The + return value needs to be of server lifetime, which is typically achieved by + defining it as a <literal>static const</literal> variable in global scope. <programlisting> typedef struct ArchiveModuleCallbacks @@ -65,103 +72,110 @@ typedef struct ArchiveModuleCallbacks typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void); </programlisting> - Only the <function>archive_file_cb</function> callback is required. The - others are optional. - </para> - </sect1> + Only the <function>archive_file_cb</function> callback is required. The + others are optional. + </para> + </sect2> - <sect1 id="archive-module-callbacks"> - <title>Archive Module Callbacks</title> - <para> - The archive callbacks define the actual archiving behavior of the module. - The server will call them as required to process each individual WAL file. - </para> + <sect2 id="archive-module-callbacks"> + <title>Archive Module Callbacks</title> - <sect2 id="archive-module-startup"> - <title>Startup Callback</title> <para> - The <function>startup_cb</function> callback is called shortly after the - module is loaded. This callback can be used to perform any additional - initialization required. If the archive module has any state, it can use - <structfield>state->private_data</structfield> to store it. + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + </para> + + <sect3 id="archive-module-startup"> + <title>Startup Callback</title> + + <para> + The <function>startup_cb</function> callback is called shortly after the + module is loaded. This callback can be used to perform any additional + initialization required. If the archive module has any state, it can use + <structfield>state->private_data</structfield> to store it. <programlisting> typedef void (*ArchiveStartupCB) (ArchiveModuleState *state); </programlisting> - </para> - </sect2> + </para> + </sect3> - <sect2 id="archive-module-check"> - <title>Check Callback</title> - <para> - The <function>check_configured_cb</function> callback is called to determine - whether the module is fully configured and ready to accept WAL files (e.g., - its configuration parameters are set to valid values). If no - <function>check_configured_cb</function> is defined, the server always - assumes the module is configured. + <sect3 id="archive-module-check"> + <title>Check Callback</title> + + <para> + The <function>check_configured_cb</function> callback is called to + determine whether the module is fully configured and ready to accept WAL + files (e.g., its configuration parameters are set to valid values). If no + <function>check_configured_cb</function> is defined, the server always + assumes the module is configured. <programlisting> typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); </programlisting> - If <literal>true</literal> is returned, the server will proceed with - archiving the file by calling the <function>archive_file_cb</function> - callback. If <literal>false</literal> is returned, archiving will not - proceed, and the archiver will emit the following message to the server log: + If <literal>true</literal> is returned, the server will proceed with + archiving the file by calling the <function>archive_file_cb</function> + callback. If <literal>false</literal> is returned, archiving will not + proceed, and the archiver will emit the following message to the server + log: <screen> WARNING: archive_mode enabled, yet archiving is not configured </screen> - In the latter case, the server will periodically call this function, and - archiving will proceed only when it returns <literal>true</literal>. - </para> - - <note> - <para> - When returning <literal>false</literal>, it may be useful to append some - additional information to the generic warning message. To do that, provide - a message to the <function>arch_module_check_errdetail</function> macro - before returning <literal>false</literal>. Like - <function>errdetail()</function>, this macro accepts a format string - followed by an optional list of arguments. The resulting string will be - emitted as the <literal>DETAIL</literal> line of the warning message. + In the latter case, the server will periodically call this function, and + archiving will proceed only when it returns <literal>true</literal>. </para> - </note> - </sect2> - <sect2 id="archive-module-archive"> - <title>Archive Callback</title> - <para> - The <function>archive_file_cb</function> callback is called to archive a - single WAL file. + <note> + <para> + When returning <literal>false</literal>, it may be useful to append some + additional information to the generic warning message. To do that, + provide a message to the <function>arch_module_check_errdetail</function> + macro before returning <literal>false</literal>. Like + <function>errdetail()</function>, this macro accepts a format string + followed by an optional list of arguments. The resulting string will be + emitted as the <literal>DETAIL</literal> line of the warning message. + </para> + </note> + </sect3> + + <sect3 id="archive-module-archive"> + <title>Archive Callback</title> + + <para> + The <function>archive_file_cb</function> callback is called to archive a + single WAL file. <programlisting> typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path); </programlisting> - If <literal>true</literal> is returned, the server proceeds as if the file - was successfully archived, which may include recycling or removing the - original WAL file. If <literal>false</literal> is returned, the server will - keep the original WAL file and retry archiving later. - <replaceable>file</replaceable> will contain just the file name of the WAL - file to archive, while <replaceable>path</replaceable> contains the full - path of the WAL file (including the file name). - </para> - </sect2> + If <literal>true</literal> is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If <literal>false</literal> is returned, the server + will keep the original WAL file and retry archiving later. + <replaceable>file</replaceable> will contain just the file name of the WAL + file to archive, while <replaceable>path</replaceable> contains the full + path of the WAL file (including the file name). + </para> + </sect3> - <sect2 id="archive-module-shutdown"> - <title>Shutdown Callback</title> - <para> - The <function>shutdown_cb</function> callback is called when the archiver - process exits (e.g., after an error) or the value of - <xref linkend="guc-archive-library"/> changes. If no - <function>shutdown_cb</function> is defined, no special action is taken in - these situations. If the archive module has any state, this callback should - free it to avoid leaks. + <sect3 id="archive-module-shutdown"> + <title>Shutdown Callback</title> + + <para> + The <function>shutdown_cb</function> callback is called when the archiver + process exits (e.g., after an error) or the value of + <xref linkend="guc-archive-library"/> changes. If no + <function>shutdown_cb</function> is defined, no special action is taken in + these situations. If the archive module has any state, this callback + should free it to avoid leaks. <programlisting> typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state); </programlisting> - </para> + </para> + </sect3> </sect2> </sect1> </chapter> -- 2.25.1 --oyUTqETQ0mS9luUI Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v19-0005-introduce-restore_library.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v18 4/5] restructure archive modules docs in preparation for restore modules @ 2023-02-15 22:48 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Nathan Bossart @ 2023-02-15 22:48 UTC (permalink / raw) --- doc/src/sgml/archive-and-restore-modules.sgml | 226 ++++++++++-------- 1 file changed, 120 insertions(+), 106 deletions(-) diff --git a/doc/src/sgml/archive-and-restore-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml index cf7438a759..a52d15082d 100644 --- a/doc/src/sgml/archive-and-restore-modules.sgml +++ b/doc/src/sgml/archive-and-restore-modules.sgml @@ -1,9 +1,9 @@ -<!-- doc/src/sgml/archive-modules.sgml --> +<!-- doc/src/sgml/archive-and-restore-modules.sgml --> -<chapter id="archive-modules"> - <title>Archive Modules</title> - <indexterm zone="archive-modules"> - <primary>Archive Modules</primary> +<chapter id="archive-and-restore-modules"> + <title>Archive and Restore Modules</title> + <indexterm zone="archive-and-restore-modules"> + <primary>Archive and Restore Modules</primary> </indexterm> <para> @@ -14,45 +14,52 @@ performant. </para> - <para> - When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL - will submit completed WAL files to the module, and the server will avoid - recycling or removing these WAL files until the module indicates that the files - were successfully archived. It is ultimately up to the module to decide what - to do with each WAL file, but many recommendations are listed at - <xref linkend="backup-archiving-wal"/>. - </para> - - <para> - Archiving modules must at least consist of an initialization function (see - <xref linkend="archive-module-init"/>) and the required callbacks (see - <xref linkend="archive-module-callbacks"/>). However, archive modules are - also permitted to do much more (e.g., declare GUCs and register background - workers). - </para> - <para> The <filename>contrib/basic_archive</filename> module contains a working example, which demonstrates some useful techniques. </para> - <sect1 id="archive-module-init"> - <title>Initialization Functions</title> - <indexterm zone="archive-module-init"> - <primary>_PG_archive_module_init</primary> + <sect1 id="archive-modules"> + <title>Archive Modules</title> + <indexterm zone="archive-modules"> + <primary>Archive Modules</primary> </indexterm> + + <para> + When a custom <xref linkend="guc-archive-library"/> is configured, + PostgreSQL will submit completed WAL files to the module, and the server + will avoid recycling or removing these WAL files until the module indicates + that the files were successfully archived. It is ultimately up to the + module to decide what to do with each WAL file, but many recommendations are + listed at <xref linkend="backup-archiving-wal"/>. + </para> + <para> - An archive library is loaded by dynamically loading a shared library with the - <xref linkend="guc-archive-library"/>'s name as the library base name. The - normal library search path is used to locate the library. To provide the - required archive module callbacks and to indicate that the library is - actually an archive module, it needs to provide a function named - <function>_PG_archive_module_init</function>. The result of the function - must be a pointer to a struct of type - <structname>ArchiveModuleCallbacks</structname>, which contains everything - that the core code needs to know to make use of the archive module. The - return value needs to be of server lifetime, which is typically achieved by - defining it as a <literal>static const</literal> variable in global scope. + Archiving modules must at least consist of an initialization function (see + <xref linkend="archive-module-init"/>) and the required callbacks (see + <xref linkend="archive-module-callbacks"/>). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + </para> + + <sect2 id="archive-module-init"> + <title>Initialization Functions</title> + <indexterm zone="archive-module-init"> + <primary>_PG_archive_module_init</primary> + </indexterm> + + <para> + An archive library is loaded by dynamically loading a shared library with + the <xref linkend="guc-archive-library"/>'s name as the library base name. + The normal library search path is used to locate the library. To provide + the required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + <function>_PG_archive_module_init</function>. The result of the function + must be a pointer to a struct of type + <structname>ArchiveModuleCallbacks</structname>, which contains everything + that the core code needs to know to make use of the archive module. The + return value needs to be of server lifetime, which is typically achieved by + defining it as a <literal>static const</literal> variable in global scope. <programlisting> typedef struct ArchiveModuleCallbacks @@ -65,103 +72,110 @@ typedef struct ArchiveModuleCallbacks typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void); </programlisting> - Only the <function>archive_file_cb</function> callback is required. The - others are optional. - </para> - </sect1> + Only the <function>archive_file_cb</function> callback is required. The + others are optional. + </para> + </sect2> - <sect1 id="archive-module-callbacks"> - <title>Archive Module Callbacks</title> - <para> - The archive callbacks define the actual archiving behavior of the module. - The server will call them as required to process each individual WAL file. - </para> + <sect2 id="archive-module-callbacks"> + <title>Archive Module Callbacks</title> - <sect2 id="archive-module-startup"> - <title>Startup Callback</title> <para> - The <function>startup_cb</function> callback is called shortly after the - module is loaded. This callback can be used to perform any additional - initialization required. If the archive module has any state, it can use - <structfield>state->private_data</structfield> to store it. + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + </para> + + <sect3 id="archive-module-startup"> + <title>Startup Callback</title> + + <para> + The <function>startup_cb</function> callback is called shortly after the + module is loaded. This callback can be used to perform any additional + initialization required. If the archive module has any state, it can use + <structfield>state->private_data</structfield> to store it. <programlisting> typedef void (*ArchiveStartupCB) (ArchiveModuleState *state); </programlisting> - </para> - </sect2> + </para> + </sect3> - <sect2 id="archive-module-check"> - <title>Check Callback</title> - <para> - The <function>check_configured_cb</function> callback is called to determine - whether the module is fully configured and ready to accept WAL files (e.g., - its configuration parameters are set to valid values). If no - <function>check_configured_cb</function> is defined, the server always - assumes the module is configured. + <sect3 id="archive-module-check"> + <title>Check Callback</title> + + <para> + The <function>check_configured_cb</function> callback is called to + determine whether the module is fully configured and ready to accept WAL + files (e.g., its configuration parameters are set to valid values). If no + <function>check_configured_cb</function> is defined, the server always + assumes the module is configured. <programlisting> typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); </programlisting> - If <literal>true</literal> is returned, the server will proceed with - archiving the file by calling the <function>archive_file_cb</function> - callback. If <literal>false</literal> is returned, archiving will not - proceed, and the archiver will emit the following message to the server log: + If <literal>true</literal> is returned, the server will proceed with + archiving the file by calling the <function>archive_file_cb</function> + callback. If <literal>false</literal> is returned, archiving will not + proceed, and the archiver will emit the following message to the server + log: <screen> WARNING: archive_mode enabled, yet archiving is not configured </screen> - In the latter case, the server will periodically call this function, and - archiving will proceed only when it returns <literal>true</literal>. - </para> - - <note> - <para> - When returning <literal>false</literal>, it may be useful to append some - additional information to the generic warning message. To do that, provide - a message to the <function>arch_module_check_errdetail</function> macro - before returning <literal>false</literal>. Like - <function>errdetail()</function>, this macro accepts a format string - followed by an optional list of arguments. The resulting string will be - emitted as the <literal>DETAIL</literal> line of the warning message. + In the latter case, the server will periodically call this function, and + archiving will proceed only when it returns <literal>true</literal>. </para> - </note> - </sect2> - <sect2 id="archive-module-archive"> - <title>Archive Callback</title> - <para> - The <function>archive_file_cb</function> callback is called to archive a - single WAL file. + <note> + <para> + When returning <literal>false</literal>, it may be useful to append some + additional information to the generic warning message. To do that, + provide a message to the <function>arch_module_check_errdetail</function> + macro before returning <literal>false</literal>. Like + <function>errdetail()</function>, this macro accepts a format string + followed by an optional list of arguments. The resulting string will be + emitted as the <literal>DETAIL</literal> line of the warning message. + </para> + </note> + </sect3> + + <sect3 id="archive-module-archive"> + <title>Archive Callback</title> + + <para> + The <function>archive_file_cb</function> callback is called to archive a + single WAL file. <programlisting> typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path); </programlisting> - If <literal>true</literal> is returned, the server proceeds as if the file - was successfully archived, which may include recycling or removing the - original WAL file. If <literal>false</literal> is returned, the server will - keep the original WAL file and retry archiving later. - <replaceable>file</replaceable> will contain just the file name of the WAL - file to archive, while <replaceable>path</replaceable> contains the full - path of the WAL file (including the file name). - </para> - </sect2> + If <literal>true</literal> is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If <literal>false</literal> is returned, the server + will keep the original WAL file and retry archiving later. + <replaceable>file</replaceable> will contain just the file name of the WAL + file to archive, while <replaceable>path</replaceable> contains the full + path of the WAL file (including the file name). + </para> + </sect3> - <sect2 id="archive-module-shutdown"> - <title>Shutdown Callback</title> - <para> - The <function>shutdown_cb</function> callback is called when the archiver - process exits (e.g., after an error) or the value of - <xref linkend="guc-archive-library"/> changes. If no - <function>shutdown_cb</function> is defined, no special action is taken in - these situations. If the archive module has any state, this callback should - free it to avoid leaks. + <sect3 id="archive-module-shutdown"> + <title>Shutdown Callback</title> + + <para> + The <function>shutdown_cb</function> callback is called when the archiver + process exits (e.g., after an error) or the value of + <xref linkend="guc-archive-library"/> changes. If no + <function>shutdown_cb</function> is defined, no special action is taken in + these situations. If the archive module has any state, this callback + should free it to avoid leaks. <programlisting> typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state); </programlisting> - </para> + </para> + </sect3> </sect2> </sect1> </chapter> -- 2.25.1 --IJpNTDwzlM2Ie8A6 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v18-0005-introduce-restore_library.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v16 4/5] restructure archive modules docs in preparation for restore modules @ 2023-02-15 22:48 Nathan Bossart <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Nathan Bossart @ 2023-02-15 22:48 UTC (permalink / raw) --- doc/src/sgml/archive-and-restore-modules.sgml | 206 ++++++++++-------- 1 file changed, 110 insertions(+), 96 deletions(-) diff --git a/doc/src/sgml/archive-and-restore-modules.sgml b/doc/src/sgml/archive-and-restore-modules.sgml index 7064307d9e..7ed50a3b52 100644 --- a/doc/src/sgml/archive-and-restore-modules.sgml +++ b/doc/src/sgml/archive-and-restore-modules.sgml @@ -1,9 +1,9 @@ -<!-- doc/src/sgml/archive-modules.sgml --> +<!-- doc/src/sgml/archive-and-restore-modules.sgml --> -<chapter id="archive-modules"> - <title>Archive Modules</title> - <indexterm zone="archive-modules"> - <primary>Archive Modules</primary> +<chapter id="archive-and-restore-modules"> + <title>Archive and Restore Modules</title> + <indexterm zone="archive-and-restore-modules"> + <primary>Archive and Restore Modules</primary> </indexterm> <para> @@ -14,45 +14,52 @@ performant. </para> - <para> - When a custom <xref linkend="guc-archive-library"/> is configured, PostgreSQL - will submit completed WAL files to the module, and the server will avoid - recycling or removing these WAL files until the module indicates that the files - were successfully archived. It is ultimately up to the module to decide what - to do with each WAL file, but many recommendations are listed at - <xref linkend="backup-archiving-wal"/>. - </para> - - <para> - Archiving modules must at least consist of an initialization function (see - <xref linkend="archive-module-init"/>) and the required callbacks (see - <xref linkend="archive-module-callbacks"/>). However, archive modules are - also permitted to do much more (e.g., declare GUCs and register background - workers). - </para> - <para> The <filename>contrib/basic_archive</filename> module contains a working example, which demonstrates some useful techniques. </para> - <sect1 id="archive-module-init"> - <title>Initialization Functions</title> - <indexterm zone="archive-module-init"> - <primary>_PG_archive_module_init</primary> + <sect1 id="archive-modules"> + <title>Archive Modules</title> + <indexterm zone="archive-modules"> + <primary>Archive Modules</primary> </indexterm> + + <para> + When a custom <xref linkend="guc-archive-library"/> is configured, + PostgreSQL will submit completed WAL files to the module, and the server + will avoid recycling or removing these WAL files until the module indicates + that the files were successfully archived. It is ultimately up to the + module to decide what to do with each WAL file, but many recommendations are + listed at <xref linkend="backup-archiving-wal"/>. + </para> + <para> - An archive library is loaded by dynamically loading a shared library with the - <xref linkend="guc-archive-library"/>'s name as the library base name. The - normal library search path is used to locate the library. To provide the - required archive module callbacks and to indicate that the library is - actually an archive module, it needs to provide a function named - <function>_PG_archive_module_init</function>. The result of the function - must be a pointer to a struct of type - <structname>ArchiveModuleCallbacks</structname>, which contains everything - that the core code needs to know to make use of the archive module. The - return value needs to be of server lifetime, which is typically achieved by - defining it as a <literal>static const</literal> variable in global scope. + Archiving modules must at least consist of an initialization function (see + <xref linkend="archive-module-init"/>) and the required callbacks (see + <xref linkend="archive-module-callbacks"/>). However, archive modules are + also permitted to do much more (e.g., declare GUCs and register background + workers). + </para> + + <sect2 id="archive-module-init"> + <title>Initialization Functions</title> + <indexterm zone="archive-module-init"> + <primary>_PG_archive_module_init</primary> + </indexterm> + + <para> + An archive library is loaded by dynamically loading a shared library with + the <xref linkend="guc-archive-library"/>'s name as the library base name. + The normal library search path is used to locate the library. To provide + the required archive module callbacks and to indicate that the library is + actually an archive module, it needs to provide a function named + <function>_PG_archive_module_init</function>. The result of the function + must be a pointer to a struct of type + <structname>ArchiveModuleCallbacks</structname>, which contains everything + that the core code needs to know to make use of the archive module. The + return value needs to be of server lifetime, which is typically achieved by + defining it as a <literal>static const</literal> variable in global scope. <programlisting> typedef struct ArchiveModuleCallbacks @@ -65,91 +72,98 @@ typedef struct ArchiveModuleCallbacks typedef const ArchiveModuleCallbacks *(*ArchiveModuleInit) (void); </programlisting> - Only the <function>archive_file_cb</function> callback is required. The - others are optional. - </para> - </sect1> + Only the <function>archive_file_cb</function> callback is required. The + others are optional. + </para> + </sect2> - <sect1 id="archive-module-callbacks"> - <title>Archive Module Callbacks</title> - <para> - The archive callbacks define the actual archiving behavior of the module. - The server will call them as required to process each individual WAL file. - </para> + <sect2 id="archive-module-callbacks"> + <title>Archive Module Callbacks</title> - <sect2 id="archive-module-startup"> - <title>Startup Callback</title> <para> - The <function>startup_cb</function> callback is called shortly after the - module is loaded. This callback can be used to perform any additional - initialization required. If the archive module has any state, it can use - <structfield>state->private_data</structfield> to store it. + The archive callbacks define the actual archiving behavior of the module. + The server will call them as required to process each individual WAL file. + </para> + + <sect3 id="archive-module-startup"> + <title>Startup Callback</title> + + <para> + The <function>startup_cb</function> callback is called shortly after the + module is loaded. This callback can be used to perform any additional + initialization required. If the archive module has any state, it can use + <structfield>state->private_data</structfield> to store it. <programlisting> typedef void (*ArchiveStartupCB) (ArchiveModuleState *state); </programlisting> - </para> - </sect2> + </para> + </sect3> - <sect2 id="archive-module-check"> - <title>Check Callback</title> - <para> - The <function>check_configured_cb</function> callback is called to determine - whether the module is fully configured and ready to accept WAL files (e.g., - its configuration parameters are set to valid values). If no - <function>check_configured_cb</function> is defined, the server always - assumes the module is configured. + <sect3 id="archive-module-check"> + <title>Check Callback</title> + + <para> + The <function>check_configured_cb</function> callback is called to + determine whether the module is fully configured and ready to accept WAL + files (e.g., its configuration parameters are set to valid values). If no + <function>check_configured_cb</function> is defined, the server always + assumes the module is configured. <programlisting> typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); </programlisting> - If <literal>true</literal> is returned, the server will proceed with - archiving the file by calling the <function>archive_file_cb</function> - callback. If <literal>false</literal> is returned, archiving will not - proceed, and the archiver will emit the following message to the server log: + If <literal>true</literal> is returned, the server will proceed with + archiving the file by calling the <function>archive_file_cb</function> + callback. If <literal>false</literal> is returned, archiving will not + proceed, and the archiver will emit the following message to the server + log: <screen> WARNING: archive_mode enabled, yet archiving is not configured </screen> - In the latter case, the server will periodically call this function, and - archiving will proceed only when it returns <literal>true</literal>. - </para> - </sect2> + In the latter case, the server will periodically call this function, and + archiving will proceed only when it returns <literal>true</literal>. + </para> + </sect3> - <sect2 id="archive-module-archive"> - <title>Archive Callback</title> - <para> - The <function>archive_file_cb</function> callback is called to archive a - single WAL file. + <sect3 id="archive-module-archive"> + <title>Archive Callback</title> + + <para> + The <function>archive_file_cb</function> callback is called to archive a + single WAL file. <programlisting> typedef bool (*ArchiveFileCB) (ArchiveModuleState *state, const char *file, const char *path); </programlisting> - If <literal>true</literal> is returned, the server proceeds as if the file - was successfully archived, which may include recycling or removing the - original WAL file. If <literal>false</literal> is returned, the server will - keep the original WAL file and retry archiving later. - <replaceable>file</replaceable> will contain just the file name of the WAL - file to archive, while <replaceable>path</replaceable> contains the full - path of the WAL file (including the file name). - </para> - </sect2> - - <sect2 id="archive-module-shutdown"> - <title>Shutdown Callback</title> - <para> - The <function>shutdown_cb</function> callback is called when the archiver - process exits (e.g., after an error) or the value of - <xref linkend="guc-archive-library"/> changes. If no - <function>shutdown_cb</function> is defined, no special action is taken in - these situations. If the archive module has any state, this callback should - free it to avoid leaks. + If <literal>true</literal> is returned, the server proceeds as if the file + was successfully archived, which may include recycling or removing the + original WAL file. If <literal>false</literal> is returned, the server + will keep the original WAL file and retry archiving later. + <replaceable>file</replaceable> will contain just the file name of the WAL + file to archive, while <replaceable>path</replaceable> contains the full + path of the WAL file (including the file name). + </para> + </sect3> + + <sect3 id="archive-module-shutdown"> + <title>Shutdown Callback</title> + + <para> + The <function>shutdown_cb</function> callback is called when the archiver + process exits (e.g., after an error) or the value of + <xref linkend="guc-archive-library"/> changes. If no + <function>shutdown_cb</function> is defined, no special action is taken in + these situations. If the archive module has any state, this callback + should free it to avoid leaks. <programlisting> typedef void (*ArchiveShutdownCB) (ArchiveModuleState *state); </programlisting> - </para> + </para> + </sect3> </sect2> </sect1> </chapter> -- 2.25.1 --SLDf9lqlvOQaIe6s Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v16-0005-introduce-restore_library.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-10 12:00 Masahiko Sawada <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Masahiko Sawada @ 2024-02-10 12:00 UTC (permalink / raw) To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: shveta malik <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Fri, Feb 9, 2024 at 4:08 PM Zhijie Hou (Fujitsu) <[email protected]> wrote: > > On Friday, February 9, 2024 2:44 PM Masahiko Sawada <[email protected]> wrote: > > > > On Thu, Feb 8, 2024 at 8:01 PM shveta malik <[email protected]> wrote: > > > > > > On Thu, Feb 8, 2024 at 12:08 PM Peter Smith <[email protected]> > > wrote: > > > > > > > > Here are some review comments for patch v80_2-0001. > > > > > > Thanks for the feedback Peter. Addressed the comments in v81. > > > Attached patch001 for early feedback. Rest of the patches need > > > rebasing and thus will post those later. > > > > > > It also addresses comments by Amit in [1]. > > > > Thank you for updating the patch! Here are random comments: > > Thanks for the comments! > > > > > --- > > + > > + /* > > + * Register the callback function to clean up the shared memory of > > slot > > + * synchronization. > > + */ > > + SlotSyncInitialize(); > > > > I think it would have a wider impact than expected. IIUC this callback is needed > > only for processes who calls synchronize_slots(). Why do we want all processes > > to register this callback? > > I think the current style is similar to the ReplicationSlotInitialize() above it. For backend, > both of them can only be used when user calls slot SQL functions. So, I think it could be fine to > register it at the general place which can also avoid registering the same again for the later > slotsync worker patch. Yes, but it seems to be a legitimate case since replication slot code involves many functions that need the callback to clear the flag. On the other hand, in the slotsync code, only one function, SyncReplicationSlots(), needs the callback at least in 0001 patch. > Another alternative is to register the callback when calling slotsync functions > and unregister it after the function call. And register the callback in > slotsyncworkmain() for the slotsync worker patch, although this may adds a few > more codes. Another idea is that SyncReplicationSlots() calls synchronize_slots() in PG_ENSURE_ERROR_CLEANUP() block instead of PG_TRY(), to make sure to clear the flag in case of ERROR or FATAL. And the slotsync worker uses the before_shmem_callback to clear the flag. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-10 13:10 Amit Kapila <[email protected]> parent: Masahiko Sawada <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Amit Kapila @ 2024-02-10 13:10 UTC (permalink / raw) To: Masahiko Sawada <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Sat, Feb 10, 2024 at 5:31 PM Masahiko Sawada <[email protected]> wrote: > > On Fri, Feb 9, 2024 at 4:08 PM Zhijie Hou (Fujitsu) > <[email protected]> wrote: > > > Another alternative is to register the callback when calling slotsync functions > > and unregister it after the function call. And register the callback in > > slotsyncworkmain() for the slotsync worker patch, although this may adds a few > > more codes. > > Another idea is that SyncReplicationSlots() calls synchronize_slots() > in PG_ENSURE_ERROR_CLEANUP() block instead of PG_TRY(), to make sure > to clear the flag in case of ERROR or FATAL. And the slotsync worker > uses the before_shmem_callback to clear the flag. > +1. This sounds like a better way to clear the flag. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 46+ messages in thread
* RE: Synchronizing slots from primary to standby @ 2024-02-11 13:23 Zhijie Hou (Fujitsu) <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 2 replies; 46+ messages in thread From: Zhijie Hou (Fujitsu) @ 2024-02-11 13:23 UTC (permalink / raw) To: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; +Cc: shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Saturday, February 10, 2024 9:10 PM Amit Kapila <[email protected]> wrote: > > On Sat, Feb 10, 2024 at 5:31 PM Masahiko Sawada <[email protected]> > wrote: > > > > On Fri, Feb 9, 2024 at 4:08 PM Zhijie Hou (Fujitsu) > > <[email protected]> wrote: > > > > > Another alternative is to register the callback when calling > > > slotsync functions and unregister it after the function call. And > > > register the callback in > > > slotsyncworkmain() for the slotsync worker patch, although this may > > > adds a few more codes. > > > > Another idea is that SyncReplicationSlots() calls synchronize_slots() > > in PG_ENSURE_ERROR_CLEANUP() block instead of PG_TRY(), to make sure > > to clear the flag in case of ERROR or FATAL. And the slotsync worker > > uses the before_shmem_callback to clear the flag. > > > > +1. This sounds like a better way to clear the flag. Agreed. Here is the V84 patch which addressed this. Apart from above, I removed the txn start/end codes from 0001 as they are used in the slotsync worker patch. And I also ran pgindent and pgperltidy for the patch. Best Regards, Hou zj Attachments: [application/octet-stream] v84-0001-Add-a-slot-synchronization-function.patch (70.4K, ../../OS0PR01MB57162A56DA47A4A2263147CA94492@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v84-0001-Add-a-slot-synchronization-function.patch) download | inline diff: From f634c5c8d0b9b78b1b0231a6c1933578132337d2 Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Wed, 7 Feb 2024 10:37:31 +0530 Subject: [PATCH v84] Add a slot synchronization function. This commit introduces a new SQL function pg_sync_replication_slots() which is used to synchronize the logical replication slots from the primary server to the physical standby so that logical replication can be resumed after failover. A new 'synced' flag is introduced in pg_replication_slots view, indicating whether the slot has been synchronized from the primary server. On a standby, synced slots cannot be dropped or consumed, and any attempt to perform logical decoding on them will result in an error. The logical replication slots on the primary can be synchronized to the hot standby by enabling failover during slot creation (e.g. using the "failover" parameter of pg_create_logical_replication_slot(), or using the "failover" option of the CREATE SUBSCRIPTION command), and then calling pg_sync_replication_slots() function on the standby. For the synchronization to work, it is mandatory to have a physical replication slot between the primary and the standby, 'hot_standby_feedback' must be enabled on the standby and a valid dbname must be specified in 'primary_conninfo'. If a logical slot is invalidated on the primary, then that slot on the standby is also invalidated. If a logical slot on the primary is valid but is invalidated on the standby, then that slot is dropped but will be recreated on the standby in next pg_sync_replication_slots() call provided the slot still exists on the primary server. It is okay to recreate such slots as long as these are not consumable on the standby (which is the case currently). This situation may occur due to the following reasons: - The 'max_slot_wal_keep_size' on the standby is insufficient to retain WAL records from the restart_lsn of the slot. - 'primary_slot_name' is temporarily reset to null and the physical slot is removed. We may also see the slots invalidated and dropped on the standby if the primary changes 'wal_level' to a level lower than logical. Changing the primary 'wal_level' to a level lower than logical is only possible if the logical slots are removed on the primary server, so it's expected to see the slots being removed on the standby too (and re-created if they are re-created on the primary server). The slots synchronization status on the standby can be monitored using 'synced' column of pg_replication_slots view. --- .../test_decoding/expected/permissions.out | 3 + contrib/test_decoding/sql/permissions.sql | 1 + doc/src/sgml/config.sgml | 9 +- doc/src/sgml/func.sgml | 33 +- doc/src/sgml/logicaldecoding.sgml | 54 ++ doc/src/sgml/protocol.sgml | 6 +- doc/src/sgml/system-views.sgml | 22 +- src/backend/catalog/system_views.sql | 3 +- src/backend/replication/logical/Makefile | 1 + src/backend/replication/logical/logical.c | 12 + src/backend/replication/logical/meson.build | 1 + src/backend/replication/logical/slotsync.c | 886 ++++++++++++++++++ src/backend/replication/slot.c | 104 +- src/backend/replication/slotfuncs.c | 72 +- src/backend/replication/walsender.c | 19 +- src/backend/storage/ipc/ipci.c | 3 + src/include/catalog/pg_proc.dat | 10 +- src/include/replication/slot.h | 19 +- src/include/replication/slotsync.h | 23 + src/include/replication/walsender.h | 3 + .../t/040_standby_failover_slots_sync.pl | 114 +++ src/test/regress/expected/rules.out | 5 +- src/tools/pgindent/typedefs.list | 2 + 23 files changed, 1370 insertions(+), 35 deletions(-) create mode 100644 src/backend/replication/logical/slotsync.c create mode 100644 src/include/replication/slotsync.h diff --git a/contrib/test_decoding/expected/permissions.out b/contrib/test_decoding/expected/permissions.out index d6eaba8c55..8d100646ce 100644 --- a/contrib/test_decoding/expected/permissions.out +++ b/contrib/test_decoding/expected/permissions.out @@ -64,6 +64,9 @@ DETAIL: Only roles with the REPLICATION attribute may use replication slots. SELECT pg_drop_replication_slot('regression_slot'); ERROR: permission denied to use replication slots DETAIL: Only roles with the REPLICATION attribute may use replication slots. +SELECT pg_sync_replication_slots(); +ERROR: permission denied to use replication slots +DETAIL: Only roles with the REPLICATION attribute may use replication slots. RESET ROLE; -- replication users can drop superuser created slots SET ROLE regress_lr_superuser; diff --git a/contrib/test_decoding/sql/permissions.sql b/contrib/test_decoding/sql/permissions.sql index 312b514593..94db936aee 100644 --- a/contrib/test_decoding/sql/permissions.sql +++ b/contrib/test_decoding/sql/permissions.sql @@ -29,6 +29,7 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_d INSERT INTO lr_test VALUES('lr_superuser_init'); SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); SELECT pg_drop_replication_slot('regression_slot'); +SELECT pg_sync_replication_slots(); RESET ROLE; -- replication users can drop superuser created slots diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 61038472c5..037a3b8a64 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4612,8 +4612,13 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" <varname>primary_conninfo</varname> string, or in a separate <filename>~/.pgpass</filename> file on the standby server (use <literal>replication</literal> as the database name). - Do not specify a database name in the - <varname>primary_conninfo</varname> string. + </para> + <para> + For replication slot synchronization (see + <xref linkend="logicaldecoding-replication-slots-synchronization"/>), + it is also necessary to specify a valid <literal>dbname</literal> + in the <varname>primary_conninfo</varname> string. This will only be + used for slot synchronization. It is ignored for streaming. </para> <para> This parameter can only be set in the <filename>postgresql.conf</filename> diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 11d537b341..42bedf8651 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -28075,7 +28075,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset </row> <row> - <entry role="func_table_entry"><para role="func_signature"> + <entry id="pg-create-logical-replication-slot" role="func_table_entry"><para role="func_signature"> <indexterm> <primary>pg_create_logical_replication_slot</primary> </indexterm> @@ -28444,6 +28444,37 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset record is flushed along with its transaction. </para></entry> </row> + + <row> + <entry id="pg-sync-replication-slots" role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>pg_sync_replication_slots</primary> + </indexterm> + <function>pg_sync_replication_slots</function> () + <returnvalue>void</returnvalue> + </para> + <para> + Synchronize the logical failover slots from the primary server to the standby server. + This function can only be executed on the standby server. See + <xref linkend="logicaldecoding-replication-slots-synchronization"/> for details. + </para> + + <caution> + <para> + If, after executing the function, + <link linkend="guc-hot-standby-feedback"> + <varname>hot_standby_feedback</varname></link> is disabled on + the standby or the physical slot configured in + <link linkend="guc-primary-slot-name"> + <varname>primary_slot_name</varname></link> is + removed, then it is possible that the necessary rows of the + synchronized slot will be removed by the VACUUM process on the primary + server, resulting in the synchronized slot becoming invalidated. + </para> + </caution> + </entry> + </row> + </tbody> </tgroup> </table> diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index cd152d4ced..a37c5404c6 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -358,6 +358,60 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU So if a slot is no longer required it should be dropped. </para> </caution> + + </sect2> + + <sect2 id="logicaldecoding-replication-slots-synchronization"> + <title>Replication Slot Synchronization</title> + <para> + The logical replication slots on the primary can be synchronized to + the hot standby by enabling <literal>failover</literal> during slot + creation (e.g. using the <literal>failover</literal> parameter of + <link linkend="pg-create-logical-replication-slot"> + <function>pg_create_logical_replication_slot</function></link>, or + using the <link linkend="sql-createsubscription-params-with-failover"> + <literal>failover</literal></link> option of + <command>CREATE SUBSCRIPTION</command>), and then calling + <link linkend="pg-sync-replication-slots"> + <function>pg_sync_replication_slots</function></link> + on the standby. For the synchronization to work, it is mandatory to + have a physical replication slot between the primary and the standby, and + <link linkend="guc-hot-standby-feedback"><varname>hot_standby_feedback</varname></link> + must be enabled on the standby. It is also necessary to specify a valid + <literal>dbname</literal> in the + <link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>. + </para> + + <para> + The ability to resume logical replication after failover depends upon the + <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield> + value for the synchronized slots on the standby at the time of failover. + Only persistent slots that have attained synced state as true on the standby + before failover can be used for logical replication after failover. + Temporary slots will be dropped, therefore logical replication for those + slots cannot be resumed. For example, if the synchronized slot could not + become persistent on the standby due to a disabled subscription, then the + subscription cannot be resumed after failover even when it is enabled. + </para> + + <para> + To resume logical replication after failover from the synced logical + slots, the subscription's 'conninfo' must be altered to point to the + new primary server. This is done using + <link linkend="sql-altersubscription-params-connection"><command>ALTER SUBSCRIPTION ... CONNECTION</command></link>. + It is recommended that subscriptions are first disabled before promoting + the standby and are re-enabled after altering the connection string. + </para> + <caution> + <para> + There is a chance that the old primary is up again during the promotion + and if subscriptions are not disabled, the logical subscribers may + continue to receive data from the old primary server even after promotion + until the connection string is altered. This might result in data + inconsistency issues, preventing the logical subscribers from being + able to continue replication from the new primary server. + </para> + </caution> </sect2> <sect2 id="logicaldecoding-explanation-output-plugins"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index ed1d62f5f8..7ffddfbd82 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -2062,7 +2062,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" <term><literal>FAILOVER [ <replaceable class="parameter">boolean</replaceable> ]</literal></term> <listitem> <para> - If true, the slot is enabled to be synced to the standbys. + If true, the slot is enabled to be synced to the standbys + so that logical replication can be resumed after failover. The default is false. </para> </listitem> @@ -2162,7 +2163,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" <term><literal>FAILOVER [ <replaceable class="parameter">boolean</replaceable> ]</literal></term> <listitem> <para> - If true, the slot is enabled to be synced to the standbys. + If true, the slot is enabled to be synced to the standbys + so that logical replication can be resumed after failover. </para> </listitem> </varlistentry> diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index dd468b31ea..4ea2177b34 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2561,10 +2561,28 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <structfield>failover</structfield> <type>bool</type> </para> <para> - True if this is a logical slot enabled to be synced to the standbys. - Always false for physical slots. + True if this is a logical slot enabled to be synced to the standbys + so that logical replication can be resumed from the new primary + after failover. Always false for physical slots. </para></entry> </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>synced</structfield> <type>bool</type> + </para> + <para> + True if this is a logical slot that was synced from a primary server. + </para> + <para> + On a hot standby, the slots with the synced column marked as true can + neither be used for logical decoding nor dropped by the user. The value + of this column has no meaning on the primary server; the column value on + the primary is default false for all slots but may (if leftover from a + promoted standby) also be true. + </para></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 6791bff9dd..04227a72d1 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1024,7 +1024,8 @@ CREATE VIEW pg_replication_slots AS L.safe_wal_size, L.two_phase, L.conflict_reason, - L.failover + L.failover, + L.synced FROM pg_get_replication_slots() AS L LEFT JOIN pg_database D ON (L.datoid = D.oid); diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index 2dc25e37bb..ba03eeff1c 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -25,6 +25,7 @@ OBJS = \ proto.o \ relation.o \ reorderbuffer.o \ + slotsync.o \ snapbuild.o \ tablesync.o \ worker.o diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index ca09c683f1..a53815f2ed 100644 --- a/src/backend/replication/logical/logical.c +++ b/src/backend/replication/logical/logical.c @@ -524,6 +524,18 @@ CreateDecodingContext(XLogRecPtr start_lsn, errmsg("replication slot \"%s\" was not created in this database", NameStr(slot->data.name)))); + /* + * Do not allow consumption of a "synchronized" slot until the standby + * gets promoted. + */ + if (RecoveryInProgress() && slot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot use replication slot \"%s\" for logical decoding", + NameStr(slot->data.name)), + errdetail("This slot is being synchronized from the primary server."), + errhint("Specify another replication slot.")); + /* * Check if slot has been invalidated due to max_slot_wal_keep_size. Avoid * "cannot get changes" wording in this errmsg because that'd be diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build index 1050eb2c09..3dec36a6de 100644 --- a/src/backend/replication/logical/meson.build +++ b/src/backend/replication/logical/meson.build @@ -11,6 +11,7 @@ backend_sources += files( 'proto.c', 'relation.c', 'reorderbuffer.c', + 'slotsync.c', 'snapbuild.c', 'tablesync.c', 'worker.c', diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c new file mode 100644 index 0000000000..b925c7fe7d --- /dev/null +++ b/src/backend/replication/logical/slotsync.c @@ -0,0 +1,886 @@ +/*------------------------------------------------------------------------- + * slotsync.c + * Functionality for synchronizing slots to a standby server from the + * primary server. + * + * Copyright (c) 2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/slotsync.c + * + * This file contains the code for slot synchronization on a physical standby + * to fetch logical failover slots information from the primary server, create + * the slots on the standby and synchronize them. This is done by a call to SQL + * function pg_sync_replication_slots. + * + * If on physical standby, the WAL corresponding to the remote's restart_lsn + * is not available or the remote's catalog_xmin precedes the oldest xid for which + * it is guaranteed that rows wouldn't have been removed then we cannot create + * the local standby slot because that would mean moving the local slot + * backward and decoding won't be possible via such a slot. In this case, the + * slot will be marked as RS_TEMPORARY. Once the primary server catches up, + * the slot will be marked as RS_PERSISTENT (which means sync-ready) after + * which we can call pg_sync_replication_slots() periodically to perform + * syncs. + * + * Any standby synchronized slots will be dropped if they no longer need + * to be synchronized. See comment atop drop_local_obsolete_slots() for more + * details. + *--------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/xlog_internal.h" +#include "access/xlogrecovery.h" +#include "catalog/pg_database.h" +#include "commands/dbcommands.h" +#include "replication/logical.h" +#include "replication/slotsync.h" +#include "storage/ipc.h" +#include "storage/lmgr.h" +#include "storage/procarray.h" +#include "utils/builtins.h" +#include "utils/pg_lsn.h" + +/* Struct for sharing information to control slot synchronization. */ +typedef struct SlotSyncCtxStruct +{ + /* prevents concurrent slot syncs to avoid slot overwrites */ + bool syncing; + slock_t mutex; +} SlotSyncCtxStruct; + +SlotSyncCtxStruct *SlotSyncCtx = NULL; + +/* + * Flag to tell if we are syncing replication slots. Unlike the 'syncing' flag + * in SlotSyncCtxStruct, this flag is true only if the current process is + * performing slot synchronization. + */ +static bool syncing_slots = false; + +/* + * Structure to hold information fetched from the primary server about a logical + * replication slot. + */ +typedef struct RemoteSlot +{ + char *name; + char *plugin; + char *database; + bool two_phase; + bool failover; + XLogRecPtr restart_lsn; + XLogRecPtr confirmed_lsn; + TransactionId catalog_xmin; + + /* RS_INVAL_NONE if valid, or the reason of invalidation */ + ReplicationSlotInvalidationCause invalidated; +} RemoteSlot; + +/* + * If necessary, update local synced slot metadata based on the data from the + * remote slot. + * + * If no update was needed (the data of the remote slot is the same as the + * local slot) return false, otherwise true. + */ +static bool +update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot = MyReplicationSlot; + bool xmin_changed; + bool restart_lsn_changed; + NameData plugin_name; + + Assert(slot->data.invalidated == RS_INVAL_NONE); + + xmin_changed = (remote_slot->catalog_xmin != slot->data.catalog_xmin); + restart_lsn_changed = (remote_slot->restart_lsn != slot->data.restart_lsn); + + if (!xmin_changed && !restart_lsn_changed && + remote_dbid == slot->data.database && + remote_slot->two_phase == slot->data.two_phase && + remote_slot->failover == slot->data.failover && + remote_slot->confirmed_lsn == slot->data.confirmed_flush && + strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) + return false; + + /* Avoid expensive operations while holding a spinlock. */ + namestrcpy(&plugin_name, remote_slot->plugin); + + SpinLockAcquire(&slot->mutex); + slot->data.plugin = plugin_name; + slot->data.database = remote_dbid; + slot->data.two_phase = remote_slot->two_phase; + slot->data.failover = remote_slot->failover; + slot->data.restart_lsn = remote_slot->restart_lsn; + slot->data.confirmed_flush = remote_slot->confirmed_lsn; + slot->data.catalog_xmin = remote_slot->catalog_xmin; + slot->effective_catalog_xmin = remote_slot->catalog_xmin; + SpinLockRelease(&slot->mutex); + + if (xmin_changed) + ReplicationSlotsComputeRequiredXmin(false); + + if (restart_lsn_changed) + ReplicationSlotsComputeRequiredLSN(); + + return true; +} + +/* + * Get the list of local logical slots which are synchronized from the + * primary server. + */ +static List * +get_local_synced_slots(void) +{ + List *local_slots = NIL; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + + for (int i = 0; i < max_replication_slots; i++) + { + ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i]; + + /* Check if it is a synchronized slot */ + if (s->in_use && s->data.synced) + { + Assert(SlotIsLogical(s)); + local_slots = lappend(local_slots, s); + } + } + + LWLockRelease(ReplicationSlotControlLock); + + return local_slots; +} + +/* + * Helper function to check if local_slot is required to be retained. + * + * Return false either if local_slot does not exist in the remote_slots list + * or is invalidated while the corresponding remote slot is still valid, + * otherwise true. + */ +static bool +local_sync_slot_required(ReplicationSlot *local_slot, List *remote_slots) +{ + bool remote_exists = false; + bool locally_invalidated = false; + + foreach_ptr(RemoteSlot, remote_slot, remote_slots) + { + if (strcmp(remote_slot->name, NameStr(local_slot->data.name)) == 0) + { + remote_exists = true; + + /* + * If remote slot is not invalidated but local slot is marked as + * invalidated, then set locally_invalidated flag. + */ + SpinLockAcquire(&local_slot->mutex); + locally_invalidated = + (remote_slot->invalidated == RS_INVAL_NONE) && + (local_slot->data.invalidated != RS_INVAL_NONE); + SpinLockRelease(&local_slot->mutex); + + break; + } + } + + return (remote_exists && !locally_invalidated); +} + +/* + * Drop local obsolete slots. + * + * Drop the local slots that no longer need to be synced i.e. these either do + * not exist on the primary or are no longer enabled for failover. + * + * Additionally, drop any slots that are valid on the primary but got + * invalidated on the standby. This situation may occur due to the following + * reasons: + * - The 'max_slot_wal_keep_size' on the standby is insufficient to retain WAL + * records from the restart_lsn of the slot. + * - 'primary_slot_name' is temporarily reset to null and the physical slot is + * removed. + * These dropped slots will get recreated in next sync-cycle and it is okay to + * drop and recreate such slots as long as these are not consumable on the + * standby (which is the case currently). + * + * Note: Change of 'wal_level' on the primary server to a level lower than + * logical may also result in slot invalidation and removal on the standby. + * This is because such 'wal_level' change is only possible if the logical + * slots are removed on the primary server, so it's expected to see the + * slots being invalidated and removed on the standby too (and re-created + * if they are re-created on the primary server). + */ +static void +drop_local_obsolete_slots(List *remote_slot_list) +{ + List *local_slots = get_local_synced_slots(); + + foreach_ptr(ReplicationSlot, local_slot, local_slots) + { + /* Drop the local slot if it is not required to be retained. */ + if (!local_sync_slot_required(local_slot, remote_slot_list)) + { + bool synced_slot; + + /* + * Use shared lock to prevent a conflict with + * ReplicationSlotsDropDBSlots(), trying to drop the same slot + * during a drop-database operation. + */ + LockSharedObject(DatabaseRelationId, local_slot->data.database, + 0, AccessShareLock); + + /* + * In the small window between getting the slot to drop and + * locking the database, there is a possibility of a parallel + * database drop by the startup process and the creation of a new + * slot by the user. This new user-created slot may end up using + * the same shared memory as that of 'local_slot'. Thus check if + * local_slot is still the synced one before performing actual + * drop. + */ + SpinLockAcquire(&local_slot->mutex); + synced_slot = local_slot->in_use && local_slot->data.synced; + SpinLockRelease(&local_slot->mutex); + + if (synced_slot) + { + ReplicationSlotAcquire(NameStr(local_slot->data.name), true); + ReplicationSlotDropAcquired(); + } + + UnlockSharedObject(DatabaseRelationId, local_slot->data.database, + 0, AccessShareLock); + + ereport(LOG, + errmsg("dropped replication slot \"%s\" of dbid %d", + NameStr(local_slot->data.name), + local_slot->data.database)); + } + } +} + +/* + * Reserve WAL for the currently active local slot using the specified WAL + * location (restart_lsn). + * + * If the given WAL location has been removed, reserve WAL using the oldest + * existing WAL segment. + */ +static void +reserve_wal_for_local_slot(XLogRecPtr restart_lsn) +{ + XLogSegNo oldest_segno; + XLogSegNo segno; + ReplicationSlot *slot = MyReplicationSlot; + + Assert(slot != NULL); + Assert(XLogRecPtrIsInvalid(slot->data.restart_lsn)); + + while (true) + { + SpinLockAcquire(&slot->mutex); + slot->data.restart_lsn = restart_lsn; + SpinLockRelease(&slot->mutex); + + /* Prevent WAL removal as fast as possible */ + ReplicationSlotsComputeRequiredLSN(); + + XLByteToSeg(slot->data.restart_lsn, segno, wal_segment_size); + + /* + * Find the oldest existing WAL segment file. + * + * Normally, we can determine it by using the last removed segment + * number. However, if no WAL segment files have been removed by a + * checkpoint since startup, we need to search for the oldest segment + * file currently existing in XLOGDIR. + */ + oldest_segno = XLogGetLastRemovedSegno() + 1; + + if (oldest_segno == 1) + { + TimeLineID cur_timeline; + + GetWalRcvFlushRecPtr(NULL, &cur_timeline); + oldest_segno = XLogGetOldestSegno(cur_timeline); + } + + /* + * If all required WAL is still there, great, otherwise retry. The + * slot should prevent further removal of WAL, unless there's a + * concurrent ReplicationSlotsComputeRequiredLSN() after we've written + * the new restart_lsn above, so normally we should never need to loop + * more than twice. + */ + if (segno >= oldest_segno) + break; + + /* Retry using the location of the oldest wal segment */ + XLogSegNoOffsetToRecPtr(oldest_segno, 0, wal_segment_size, restart_lsn); + } +} + +/* + * If the remote restart_lsn and catalog_xmin have caught up with the + * local ones, then update the LSNs and persist the local synced slot for + * future synchronization; otherwise, do nothing. + */ +static void +update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot = MyReplicationSlot; + + /* + * Check if the primary server has caught up. Refer to the comment atop + * the file for details on this check. + */ + if (remote_slot->restart_lsn < slot->data.restart_lsn || + TransactionIdPrecedes(remote_slot->catalog_xmin, + slot->data.catalog_xmin)) + { + /* + * The remote slot didn't catch up to locally reserved position. + * + * We do not drop the slot because the restart_lsn can be ahead of the + * current location when recreating the slot in the next cycle. It may + * take more time to create such a slot. Therefore, we keep this slot + * and attempt the wait and synchronization in the next cycle. + */ + return; + } + + /* First time slot update, the function must return true */ + if (!update_local_synced_slot(remote_slot, remote_dbid)) + elog(ERROR, "failed to update slot"); + + ReplicationSlotPersist(); + + ereport(LOG, + errmsg("newly created slot \"%s\" is sync-ready now", + remote_slot->name)); +} + +/* + * Synchronize a single slot to the given position. + * + * This creates a new slot if there is no existing one and updates the + * metadata of the slot as per the data received from the primary server. + * + * The slot is created as a temporary slot and stays in the same state until the + * the remote_slot catches up with locally reserved position and local slot is + * updated. The slot is then persisted and is considered as sync-ready for + * periodic syncs. + */ +static void +synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot; + XLogRecPtr latestFlushPtr; + + /* + * Make sure that concerned WAL is received and flushed before syncing + * slot to target lsn received from the primary server. + */ + latestFlushPtr = GetStandbyFlushRecPtr(NULL); + if (remote_slot->confirmed_lsn > latestFlushPtr) + elog(ERROR, + "skipping slot synchronization as the received slot sync" + " LSN %X/%X for slot \"%s\" is ahead of the standby position %X/%X", + LSN_FORMAT_ARGS(remote_slot->confirmed_lsn), + remote_slot->name, + LSN_FORMAT_ARGS(latestFlushPtr)); + + /* Search for the named slot */ + if ((slot = SearchNamedReplicationSlot(remote_slot->name, true))) + { + bool synced; + + SpinLockAcquire(&slot->mutex); + synced = slot->data.synced; + SpinLockRelease(&slot->mutex); + + /* User-created slot with the same name exists, raise ERROR. */ + if (!synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("exiting from slot synchronization because same" + " name slot \"%s\" already exists on the standby", + remote_slot->name)); + + /* + * The slot has been synchronized before. + * + * It is important to acquire the slot here before checking + * invalidation. If we don't acquire the slot first, there could be a + * race condition that the local slot could be invalidated just after + * checking the 'invalidated' flag here and we could end up + * overwriting 'invalidated' flag to remote_slot's value. See + * InvalidatePossiblyObsoleteSlot() where it invalidates slot directly + * if the slot is not acquired by other processes. + */ + ReplicationSlotAcquire(remote_slot->name, true); + + Assert(slot == MyReplicationSlot); + + /* + * Copy the invalidation cause from remote only if local slot is not + * invalidated locally, we don't want to overwrite existing one. + */ + if (slot->data.invalidated == RS_INVAL_NONE && + remote_slot->invalidated != RS_INVAL_NONE) + { + SpinLockAcquire(&slot->mutex); + slot->data.invalidated = remote_slot->invalidated; + SpinLockRelease(&slot->mutex); + + /* Make sure the invalidated state persists across server restart */ + ReplicationSlotMarkDirty(); + ReplicationSlotSave(); + } + + /* Skip the sync of an invalidated slot */ + if (slot->data.invalidated != RS_INVAL_NONE) + { + ReplicationSlotRelease(); + return; + } + + /* Slot not ready yet, let's attempt to make it sync-ready now. */ + if (slot->data.persistency == RS_TEMPORARY) + { + update_and_persist_local_synced_slot(remote_slot, remote_dbid); + } + + /* Slot ready for sync, so sync it. */ + else + { + /* + * Sanity check: As long as the invalidations are handled + * appropriately as above, this should never happen. + */ + if (remote_slot->restart_lsn < slot->data.restart_lsn) + elog(ERROR, + "cannot synchronize local slot \"%s\" LSN(%X/%X)" + " to remote slot's LSN(%X/%X) as synchronization" + " would move it backwards", remote_slot->name, + LSN_FORMAT_ARGS(slot->data.restart_lsn), + LSN_FORMAT_ARGS(remote_slot->restart_lsn)); + + /* Make sure the slot changes persist across server restart */ + if (update_local_synced_slot(remote_slot, remote_dbid)) + { + ReplicationSlotMarkDirty(); + ReplicationSlotSave(); + } + } + } + /* Otherwise create the slot first. */ + else + { + NameData plugin_name; + TransactionId xmin_horizon = InvalidTransactionId; + + /* Skip creating the local slot if remote_slot is invalidated already */ + if (remote_slot->invalidated != RS_INVAL_NONE) + return; + + ReplicationSlotCreate(remote_slot->name, true, RS_TEMPORARY, + remote_slot->two_phase, + remote_slot->failover, + true); + + /* For shorter lines. */ + slot = MyReplicationSlot; + + /* Avoid expensive operations while holding a spinlock. */ + namestrcpy(&plugin_name, remote_slot->plugin); + + SpinLockAcquire(&slot->mutex); + slot->data.database = remote_dbid; + slot->data.plugin = plugin_name; + SpinLockRelease(&slot->mutex); + + reserve_wal_for_local_slot(remote_slot->restart_lsn); + + LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE); + xmin_horizon = GetOldestSafeDecodingTransactionId(true); + SpinLockAcquire(&slot->mutex); + slot->effective_catalog_xmin = xmin_horizon; + slot->data.catalog_xmin = xmin_horizon; + SpinLockRelease(&slot->mutex); + ReplicationSlotsComputeRequiredXmin(true); + LWLockRelease(ProcArrayLock); + + update_and_persist_local_synced_slot(remote_slot, remote_dbid); + } + + ReplicationSlotRelease(); +} + +/* + * Synchronize slots. + * + * Gets the failover logical slots info from the primary server and updates + * the slots locally. Creates the slots if not present on the standby. + */ +static void +synchronize_slots(WalReceiverConn *wrconn) +{ +#define SLOTSYNC_COLUMN_COUNT 9 + Oid slotRow[SLOTSYNC_COLUMN_COUNT] = {TEXTOID, TEXTOID, LSNOID, + LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID}; + + WalRcvExecResult *res; + TupleTableSlot *tupslot; + StringInfoData s; + List *remote_slot_list = NIL; + + SpinLockAcquire(&SlotSyncCtx->mutex); + if (SlotSyncCtx->syncing) + { + SpinLockRelease(&SlotSyncCtx->mutex); + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot synchronize replication slots concurrently")); + } + + SlotSyncCtx->syncing = true; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = true; + + initStringInfo(&s); + + /* Construct query to fetch slots with failover enabled. */ + appendStringInfo(&s, + "SELECT slot_name, plugin, confirmed_flush_lsn," + " restart_lsn, catalog_xmin, two_phase, failover," + " database, conflict_reason" + " FROM pg_catalog.pg_replication_slots" + " WHERE failover and NOT temporary"); + + /* Execute the query */ + res = walrcv_exec(wrconn, s.data, SLOTSYNC_COLUMN_COUNT, slotRow); + pfree(s.data); + + if (res->status != WALRCV_OK_TUPLES) + ereport(ERROR, + errmsg("could not fetch failover logical slots info from the primary server: %s", + res->err)); + + /* Construct the remote_slot tuple and synchronize each slot locally */ + tupslot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple); + while (tuplestore_gettupleslot(res->tuplestore, true, false, tupslot)) + { + bool isnull; + RemoteSlot *remote_slot = palloc0(sizeof(RemoteSlot)); + Datum d; + int col = 0; + + remote_slot->name = TextDatumGetCString(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->plugin = TextDatumGetCString(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + /* + * It is possible to get null values for LSN and Xmin if slot is + * invalidated on the primary server, so handle accordingly. + */ + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->confirmed_lsn = isnull ? InvalidXLogRecPtr : + DatumGetLSN(d); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->restart_lsn = isnull ? InvalidXLogRecPtr : DatumGetLSN(d); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->catalog_xmin = isnull ? InvalidTransactionId : + DatumGetTransactionId(d); + + remote_slot->two_phase = DatumGetBool(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->failover = DatumGetBool(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->database = TextDatumGetCString(slot_getattr(tupslot, + ++col, &isnull)); + Assert(!isnull); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->invalidated = isnull ? RS_INVAL_NONE : + GetSlotInvalidationCause(TextDatumGetCString(d)); + + /* Sanity check */ + Assert(col == SLOTSYNC_COLUMN_COUNT); + + /* + * If restart_lsn, confirmed_lsn or catalog_xmin is invalid but the + * slot is valid, that means we have fetched the remote_slot in its + * RS_EPHEMERAL state. In such a case, don't sync it; we can always + * sync it in the next sync cycle when the remote_slot is persisted + * and has valid lsn(s) and xmin values. + * + * XXX: In future, if we plan to expose 'slot->data.persistency' in + * pg_replication_slots view, then we can avoid fetching RS_EPHEMERAL + * slots in the first place. + */ + if ((XLogRecPtrIsInvalid(remote_slot->restart_lsn) || + XLogRecPtrIsInvalid(remote_slot->confirmed_lsn) || + !TransactionIdIsValid(remote_slot->catalog_xmin)) && + remote_slot->invalidated == RS_INVAL_NONE) + pfree(remote_slot); + else + /* Create list of remote slots */ + remote_slot_list = lappend(remote_slot_list, remote_slot); + + ExecClearTuple(tupslot); + } + + /* Drop local slots that no longer need to be synced. */ + drop_local_obsolete_slots(remote_slot_list); + + /* Now sync the slots locally */ + foreach_ptr(RemoteSlot, remote_slot, remote_slot_list) + { + Oid remote_dbid = get_database_oid(remote_slot->database, false); + + /* + * Use shared lock to prevent a conflict with + * ReplicationSlotsDropDBSlots(), trying to drop the same slot during + * a drop-database operation. + */ + LockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); + + synchronize_one_slot(remote_slot, remote_dbid); + + UnlockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); + } + + /* We are done, free remote_slot_list elements */ + list_free_deep(remote_slot_list); + + walrcv_clear_result(res); + + SpinLockAcquire(&SlotSyncCtx->mutex); + SlotSyncCtx->syncing = false; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = false; +} + +/* + * Validate the 'primary_slot_name' using the specified primary server + * connection. + */ +static void +validate_primary_slot_name(WalReceiverConn *wrconn) +{ +#define PRIMARY_INFO_OUTPUT_COL_COUNT 1 + WalRcvExecResult *res; + Oid slotRow[PRIMARY_INFO_OUTPUT_COL_COUNT] = {BOOLOID}; + StringInfoData cmd; + bool isnull; + TupleTableSlot *tupslot; + bool valid; + + initStringInfo(&cmd); + appendStringInfo(&cmd, + "SELECT count(*) = 1" + " FROM pg_catalog.pg_replication_slots" + " WHERE slot_type='physical' AND slot_name=%s", + quote_literal_cstr(PrimarySlotName)); + + res = walrcv_exec(wrconn, cmd.data, PRIMARY_INFO_OUTPUT_COL_COUNT, slotRow); + pfree(cmd.data); + + if (res->status != WALRCV_OK_TUPLES) + ereport(ERROR, + errmsg("could not fetch primary_slot_name \"%s\" info from the primary server: %s", + PrimarySlotName, res->err), + errhint("Check if \"primary_slot_name\" is configured correctly.")); + + tupslot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple); + if (!tuplestore_gettupleslot(res->tuplestore, true, false, tupslot)) + elog(ERROR, + "failed to fetch tuple for the primary server slot specified by \"primary_slot_name\""); + + valid = DatumGetBool(slot_getattr(tupslot, 1, &isnull)); + Assert(!isnull); + + if (!valid) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + /* translator: second %s is a GUC variable name */ + errdetail("The replication slot \"%s\" specified by \"%s\" does not exist on the primary server.", + PrimarySlotName, "primary_slot_name")); + + ExecClearTuple(tupslot); + walrcv_clear_result(res); +} + +/* + * Check all necessary GUCs for slot synchronization are set + * appropriately, otherwise raise ERROR. + */ +void +ValidateSlotSyncParams(void) +{ + char *dbname; + + /* + * A physical replication slot(primary_slot_name) is required on the + * primary to ensure that the rows needed by the standby are not removed + * after restarting, so that the synchronized slot on the standby will not + * be invalidated. + */ + if (PrimarySlotName == NULL || *PrimarySlotName == '\0') + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be defined.", "primary_slot_name")); + + /* + * hot_standby_feedback must be enabled to cooperate with the physical + * replication slot, which allows informing the primary about the xmin and + * catalog_xmin values on the standby. + */ + if (!hot_standby_feedback) + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be enabled.", "hot_standby_feedback")); + + /* + * Logical decoding requires wal_level >= logical and we currently only + * synchronize logical slots. + */ + if (wal_level < WAL_LEVEL_LOGICAL) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"wal_level\" must be >= logical.")); + + /* + * The primary_conninfo is required to make connection to primary for + * getting slots information. + */ + if (PrimaryConnInfo == NULL || *PrimaryConnInfo == '\0') + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be defined.", "primary_conninfo")); + + /* + * The slot synchronization needs a database connection for walrcv_exec to + * work. + */ + dbname = walrcv_get_dbname_from_conninfo(PrimaryConnInfo); + if (dbname == NULL) + ereport(ERROR, + + /* + * translator: 'dbname' is a specific option; %s is a GUC variable + * name + */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("'dbname' must be specified in \"%s\".", "primary_conninfo")); +} + +/* + * Is current process syncing replication slots ? + */ +bool +IsSyncingReplicationSlots(void) +{ + return syncing_slots; +} + +/* + * Amount of shared memory required for slot synchronization. + */ +Size +SlotSyncShmemSize(void) +{ + return sizeof(SlotSyncCtxStruct); +} + +/* + * Allocate and initialize the shared memory of slot synchronization. + */ +void +SlotSyncShmemInit(void) +{ + bool found; + + SlotSyncCtx = (SlotSyncCtxStruct *) + ShmemInitStruct("Slot Sync Data", SlotSyncShmemSize(), &found); + + if (!found) + { + SlotSyncCtx->syncing = false; + SpinLockInit(&SlotSyncCtx->mutex); + } +} + +/* + * Error cleanup callback for slot synchronization. + */ +static void +slotsync_failure_callback(int code, Datum arg) +{ + WalReceiverConn *wrconn = (WalReceiverConn *) DatumGetPointer(arg); + + if (syncing_slots) + { + /* + * If syncing_slots is true, it indicates that the process errored out + * without resetting the flag. So, we need to clean up shared memory + * and reset the flag here. + */ + SpinLockAcquire(&SlotSyncCtx->mutex); + SlotSyncCtx->syncing = false; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = false; + } + + walrcv_disconnect(wrconn); +} + +/* + * Synchronize the failover enabled replication slots using the specified + * primary server connection. + */ +void +SyncReplicationSlots(WalReceiverConn *wrconn) +{ + PG_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); + { + validate_primary_slot_name(wrconn); + + synchronize_slots(wrconn); + } + PG_END_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); + + walrcv_disconnect(wrconn); +} diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index fd4e96c9d6..04738ab764 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -46,6 +46,7 @@ #include "common/string.h" #include "miscadmin.h" #include "pgstat.h" +#include "replication/slotsync.h" #include "replication/slot.h" #include "storage/fd.h" #include "storage/ipc.h" @@ -103,7 +104,6 @@ int max_replication_slots = 10; /* the maximum number of replication * slots */ static void ReplicationSlotShmemExit(int code, Datum arg); -static void ReplicationSlotDropAcquired(void); static void ReplicationSlotDropPtr(ReplicationSlot *slot); /* internal persistency functions */ @@ -250,11 +250,12 @@ ReplicationSlotValidateName(const char *name, int elevel) * user will only get commit prepared. * failover: If enabled, allows the slot to be synced to standbys so * that logical replication can be resumed after failover. + * synced: True if the slot is synchronized from the primary server. */ void ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotPersistency persistency, - bool two_phase, bool failover) + bool two_phase, bool failover, bool synced) { ReplicationSlot *slot = NULL; int i; @@ -263,6 +264,34 @@ ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotValidateName(name, ERROR); + if (failover) + { + /* + * Do not allow users to create failover enabled temporary slots, + * because temporary slots will not be synced to the standby. + * + * However, failover enabled temporary slots can be created during + * slot synchronization. See the comments atop slotsync.c for details. + */ + if (persistency == RS_TEMPORARY && !IsSyncingReplicationSlots()) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a temporary replication slot")); + + /* + * Do not allow users to create the failover enabled slots on the + * standby as we do not support sync to the cascading standby. + * + * However, failover enabled slots can be created during slot + * synchronization because we need to retain the same values as the + * remote slot. + */ + if (RecoveryInProgress() && !IsSyncingReplicationSlots()) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a replication slot created on the standby")); + } + /* * If some other backend ran this code concurrently with us, we'd likely * both allocate the same slot, and that would be bad. We'd also be at @@ -315,6 +344,7 @@ ReplicationSlotCreate(const char *name, bool db_specific, slot->data.two_phase = two_phase; slot->data.two_phase_at = InvalidXLogRecPtr; slot->data.failover = failover; + slot->data.synced = synced; /* and then data only present in shared memory */ slot->just_dirtied = false; @@ -677,6 +707,16 @@ ReplicationSlotDrop(const char *name, bool nowait) ReplicationSlotAcquire(name, nowait); + /* + * Do not allow users to drop the slots which are currently being synced + * from the primary to the standby. + */ + if (RecoveryInProgress() && MyReplicationSlot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot drop replication slot \"%s\"", name), + errdetail("This slot is being synced from the primary server.")); + ReplicationSlotDropAcquired(); } @@ -696,6 +736,38 @@ ReplicationSlotAlter(const char *name, bool failover) errmsg("cannot use %s with a physical replication slot", "ALTER_REPLICATION_SLOT")); + /* + * Do not allow users to enable failover for temporary slots as we do not + * support sync temporary slots to the standby. + */ + if (failover && MyReplicationSlot->data.persistency == RS_TEMPORARY) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a temporary replication slot")); + + if (RecoveryInProgress()) + { + /* + * Do not allow users to alter the slots which are currently being + * synced from the primary to the standby. + */ + if (MyReplicationSlot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot alter replication slot \"%s\"", name), + errdetail("This slot is being synced from the primary server.")); + + /* + * Do not allow users to enable failover on the standby as we do not + * support sync to the cascading standby. + */ + if (failover) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a replication slot" + " on the standby")); + } + if (MyReplicationSlot->data.failover != failover) { SpinLockAcquire(&MyReplicationSlot->mutex); @@ -712,7 +784,7 @@ ReplicationSlotAlter(const char *name, bool failover) /* * Permanently drop the currently acquired replication slot. */ -static void +void ReplicationSlotDropAcquired(void) { ReplicationSlot *slot = MyReplicationSlot; @@ -868,8 +940,8 @@ ReplicationSlotMarkDirty(void) } /* - * Convert a slot that's marked as RS_EPHEMERAL to a RS_PERSISTENT slot, - * guaranteeing it will be there after an eventual crash. + * Convert a slot that's marked as RS_EPHEMERAL or RS_TEMPORARY to a + * RS_PERSISTENT slot, guaranteeing it will be there after an eventual crash. */ void ReplicationSlotPersist(void) @@ -2189,3 +2261,25 @@ RestoreSlotFromDisk(const char *name) (errmsg("too many replication slots active before shutdown"), errhint("Increase max_replication_slots and try again."))); } + +/* + * Maps the pg_replication_slots.conflict_reason text value to + * ReplicationSlotInvalidationCause enum value + */ +ReplicationSlotInvalidationCause +GetSlotInvalidationCause(char *conflict_reason) +{ + Assert(conflict_reason); + + if (strcmp(conflict_reason, SLOT_INVAL_WAL_REMOVED_TEXT) == 0) + return RS_INVAL_WAL_REMOVED; + else if (strcmp(conflict_reason, SLOT_INVAL_HORIZON_TEXT) == 0) + return RS_INVAL_HORIZON; + else if (strcmp(conflict_reason, SLOT_INVAL_WAL_LEVEL_TEXT) == 0) + return RS_INVAL_WAL_LEVEL; + else + Assert(0); + + /* Keep compiler quiet */ + return RS_INVAL_NONE; +} diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index eb685089b3..527e0c8ed5 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -21,7 +21,9 @@ #include "replication/decode.h" #include "replication/logical.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "utils/builtins.h" +#include "utils/guc.h" #include "utils/inval.h" #include "utils/pg_lsn.h" #include "utils/resowner.h" @@ -43,7 +45,7 @@ create_physical_replication_slot(char *name, bool immediately_reserve, /* acquire replication slot, this will check for conflicting names */ ReplicationSlotCreate(name, false, temporary ? RS_TEMPORARY : RS_PERSISTENT, false, - false); + false, false); if (immediately_reserve) { @@ -136,7 +138,7 @@ create_logical_replication_slot(char *name, char *plugin, */ ReplicationSlotCreate(name, true, temporary ? RS_TEMPORARY : RS_EPHEMERAL, two_phase, - failover); + failover, false); /* * Create logical decoding context to find start point or, if we don't @@ -237,7 +239,7 @@ pg_drop_replication_slot(PG_FUNCTION_ARGS) Datum pg_get_replication_slots(PG_FUNCTION_ARGS) { -#define PG_GET_REPLICATION_SLOTS_COLS 16 +#define PG_GET_REPLICATION_SLOTS_COLS 17 ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; XLogRecPtr currlsn; int slotno; @@ -418,21 +420,23 @@ pg_get_replication_slots(PG_FUNCTION_ARGS) break; case RS_INVAL_WAL_REMOVED: - values[i++] = CStringGetTextDatum("wal_removed"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_WAL_REMOVED_TEXT); break; case RS_INVAL_HORIZON: - values[i++] = CStringGetTextDatum("rows_removed"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_HORIZON_TEXT); break; case RS_INVAL_WAL_LEVEL: - values[i++] = CStringGetTextDatum("wal_level_insufficient"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_WAL_LEVEL_TEXT); break; } } values[i++] = BoolGetDatum(slot_contents.data.failover); + values[i++] = BoolGetDatum(slot_contents.data.synced); + Assert(i == PG_GET_REPLICATION_SLOTS_COLS); tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, @@ -700,7 +704,6 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) XLogRecPtr src_restart_lsn; bool src_islogical; bool temporary; - bool failover; char *plugin; Datum values[2]; bool nulls[2]; @@ -756,7 +759,6 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) src_islogical = SlotIsLogical(&first_slot_contents); src_restart_lsn = first_slot_contents.data.restart_lsn; temporary = (first_slot_contents.data.persistency == RS_TEMPORARY); - failover = first_slot_contents.data.failover; plugin = logical_slot ? NameStr(first_slot_contents.data.plugin) : NULL; /* Check type of replication slot */ @@ -791,12 +793,20 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) * We must not try to read WAL, since we haven't reserved it yet -- * hence pass find_startpoint false. confirmed_flush will be set * below, by copying from the source slot. + * + * To avoid potential issues with the slot synchronization where the + * restart_lsn of a replication slot can go backward, we set the + * failover option to false here. This situation occurs when a slot + * on the primary server is dropped and immediately replaced with a + * new slot of the same name, created by copying from another existing + * slot. However, the slot synchronization will only observe the + * restart_lsn of the same slot going backward. */ create_logical_replication_slot(NameStr(*dst_name), plugin, temporary, false, - failover, + false, src_restart_lsn, false); } @@ -943,3 +953,47 @@ pg_copy_physical_replication_slot_b(PG_FUNCTION_ARGS) { return copy_replication_slot(fcinfo, false); } + +/* + * Synchronize failover enabled replication slots to a standby server + * from the primary server. + */ +Datum +pg_sync_replication_slots(PG_FUNCTION_ARGS) +{ + WalReceiverConn *wrconn; + char *err; + StringInfoData app_name; + + CheckSlotPermissions(); + + if (!RecoveryInProgress()) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("replication slots can only be synchronized to a standby server")); + + /* Load the libpq-specific functions */ + load_file("libpqwalreceiver", false); + + ValidateSlotSyncParams(); + + initStringInfo(&app_name); + if (cluster_name[0]) + appendStringInfo(&app_name, "%s_slotsync", cluster_name); + else + appendStringInfoString(&app_name, "slotsync"); + + /* Connect to the primary server. */ + wrconn = walrcv_connect(PrimaryConnInfo, false, false, false, + app_name.data, &err); + pfree(app_name.data); + + if (!wrconn) + ereport(ERROR, + errcode(ERRCODE_CONNECTION_FAILURE), + errmsg("could not connect to the primary server: %s", err)); + + SyncReplicationSlots(wrconn); + + PG_RETURN_VOID(); +} diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 77c8baa32a..2d94379f1a 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -72,6 +72,7 @@ #include "postmaster/interrupt.h" #include "replication/decode.h" #include "replication/logical.h" +#include "replication/slotsync.h" #include "replication/slot.h" #include "replication/snapbuild.h" #include "replication/syncrep.h" @@ -243,7 +244,6 @@ static void WalSndShutdown(void) pg_attribute_noreturn(); static void XLogSendPhysical(void); static void XLogSendLogical(void); static void WalSndDone(WalSndSendDataCallback send_data); -static XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); static void IdentifySystem(void); static void UploadManifest(void); static bool HandleUploadManifestPacket(StringInfo buf, off_t *offset, @@ -1224,7 +1224,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) { ReplicationSlotCreate(cmd->slotname, false, cmd->temporary ? RS_TEMPORARY : RS_PERSISTENT, - false, false); + false, false, false); if (reserve_wal) { @@ -1255,7 +1255,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) */ ReplicationSlotCreate(cmd->slotname, true, cmd->temporary ? RS_TEMPORARY : RS_EPHEMERAL, - two_phase, failover); + two_phase, failover, false); /* * Do options check early so that we can bail before calling the @@ -3375,14 +3375,17 @@ WalSndDone(WalSndSendDataCallback send_data) } /* - * Returns the latest point in WAL that has been safely flushed to disk, and - * can be sent to the standby. This should only be called when in recovery, - * ie. we're streaming to a cascaded standby. + * Returns the latest point in WAL that has been safely flushed to disk. + * This should only be called when in recovery. + * + * This is called either by cascading walsender to find WAL postion to be sent + * to a cascaded standby or by slot synchronization function to validate remote + * slot's lsn before syncing it locally. * * As a side-effect, *tli is updated to the TLI of the last * replayed WAL record. */ -static XLogRecPtr +XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli) { XLogRecPtr replayPtr; @@ -3391,6 +3394,8 @@ GetStandbyFlushRecPtr(TimeLineID *tli) TimeLineID receiveTLI; XLogRecPtr result; + Assert(am_cascading_walsender || IsSyncingReplicationSlots()); + /* * We can safely send what's already been replayed. Also, if walreceiver * is streaming WAL from the same timeline, we can send anything that it diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index 7084e18861..7e7941d625 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -36,6 +36,7 @@ #include "replication/logicallauncher.h" #include "replication/origin.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/walreceiver.h" #include "replication/walsender.h" #include "storage/bufmgr.h" @@ -153,6 +154,7 @@ CalculateShmemSize(int *num_semaphores) size = add_size(size, StatsShmemSize()); size = add_size(size, WaitEventExtensionShmemSize()); size = add_size(size, InjectionPointShmemSize()); + size = add_size(size, SlotSyncShmemSize()); #ifdef EXEC_BACKEND size = add_size(size, ShmemBackendArraySize()); #endif @@ -347,6 +349,7 @@ CreateOrAttachShmemStructs(void) WalSummarizerShmemInit(); PgArchShmemInit(); ApplyLauncherShmemInit(); + SlotSyncShmemInit(); /* * Set up other modules that need some shared memory space diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 29af4ce65d..9c120fc2b7 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11127,9 +11127,9 @@ proname => 'pg_get_replication_slots', prorows => '10', proisstrict => 'f', proretset => 't', provolatile => 's', prorettype => 'record', proargtypes => '', - proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,text,bool}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,conflict_reason,failover}', + proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,text,bool,bool}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,conflict_reason,failover,synced}', prosrc => 'pg_get_replication_slots' }, { oid => '3786', descr => 'set up a logical replication slot', proname => 'pg_create_logical_replication_slot', provolatile => 'v', @@ -11212,6 +11212,10 @@ proname => 'pg_logical_emit_message', provolatile => 'v', proparallel => 'u', prorettype => 'pg_lsn', proargtypes => 'bool text bytea bool', prosrc => 'pg_logical_emit_message_bytea' }, +{ oid => '9929', descr => 'sync replication slots from the primary to the standby', + proname => 'pg_sync_replication_slots', provolatile => 'v', proparallel => 'u', + prorettype => 'void', proargtypes => '', + prosrc => 'pg_sync_replication_slots' }, # event triggers { oid => '3566', descr => 'list objects dropped by the current command', diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index da4c776492..e706ca834c 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -52,6 +52,14 @@ typedef enum ReplicationSlotInvalidationCause RS_INVAL_WAL_LEVEL, } ReplicationSlotInvalidationCause; +/* + * The possible values for 'conflict_reason' returned in + * pg_get_replication_slots. + */ +#define SLOT_INVAL_WAL_REMOVED_TEXT "wal_removed" +#define SLOT_INVAL_HORIZON_TEXT "rows_removed" +#define SLOT_INVAL_WAL_LEVEL_TEXT "wal_level_insufficient" + /* * On-Disk data of a replication slot, preserved across restarts. */ @@ -112,6 +120,11 @@ typedef struct ReplicationSlotPersistentData /* plugin name */ NameData plugin; + /* + * Was this slot synchronized from the primary server? + */ + char synced; + /* * Is this a failover slot (sync candidate for standbys)? Only relevant * for logical slots on the primary server. @@ -224,9 +237,11 @@ extern void ReplicationSlotsShmemInit(void); /* management of individual slots */ extern void ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotPersistency persistency, - bool two_phase, bool failover); + bool two_phase, bool failover, + bool synced); extern void ReplicationSlotPersist(void); extern void ReplicationSlotDrop(const char *name, bool nowait); +extern void ReplicationSlotDropAcquired(void); extern void ReplicationSlotAlter(const char *name, bool failover); extern void ReplicationSlotAcquire(const char *name, bool nowait); @@ -259,5 +274,7 @@ extern void CheckPointReplicationSlots(bool is_shutdown); extern void CheckSlotRequirements(void); extern void CheckSlotPermissions(void); +extern ReplicationSlotInvalidationCause + GetSlotInvalidationCause(char *conflict_reason); #endif /* SLOT_H */ diff --git a/src/include/replication/slotsync.h b/src/include/replication/slotsync.h new file mode 100644 index 0000000000..e86d8a47b8 --- /dev/null +++ b/src/include/replication/slotsync.h @@ -0,0 +1,23 @@ +/*------------------------------------------------------------------------- + * + * slotsync.h + * Exports for slot synchronization. + * + * Portions Copyright (c) 2016-2024, PostgreSQL Global Development Group + * + * src/include/replication/slotsync.h + * + *------------------------------------------------------------------------- + */ +#ifndef SLOTSYNC_H +#define SLOTSYNC_H + +#include "replication/walreceiver.h" + +extern void ValidateSlotSyncParams(void); +extern bool IsSyncingReplicationSlots(void); +extern Size SlotSyncShmemSize(void); +extern void SlotSyncShmemInit(void); +extern void SyncReplicationSlots(WalReceiverConn *wrconn); + +#endif /* SLOTSYNC_H */ diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h index 1b58d50b3b..a3b78bffcf 100644 --- a/src/include/replication/walsender.h +++ b/src/include/replication/walsender.h @@ -12,6 +12,8 @@ #ifndef _WALSENDER_H #define _WALSENDER_H +#include "access/xlogdefs.h" + /* * What to do with a snapshot in create replication slot command. */ @@ -44,6 +46,7 @@ extern void WalSndWakeup(bool physical, bool logical); extern void WalSndInitStopping(void); extern void WalSndWaitStopping(void); extern void HandleWalSndInitStopping(void); +extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); extern void WalSndRqstFileReload(void); /* diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index bc58ff4cab..505ccb622b 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -97,4 +97,118 @@ my ($result, $stdout, $stderr) = $subscriber1->psql('postgres', ok( $stderr =~ /ERROR: cannot set failover for enabled subscription/, "altering failover is not allowed for enabled subscription"); +################################################## +# Test logical failover slots on the standby +# Configure standby1 to replicate and synchronize logical slots configured +# for failover on the primary +# +# failover slot lsub1_slot ->| ----> subscriber1 (connected via logical replication) +# failover slot lsub2_slot | inactive +# primary ---> | +# physical slot sb1_slot --->| ----> standby1 (connected via streaming replication) +# | lsub1_slot, lsub2_slot (synced_slot) +################################################## + +my $primary = $publisher; +my $backup_name = 'backup'; +$primary->backup($backup_name); + +# Create a standby +my $standby1 = PostgreSQL::Test::Cluster->new('standby1'); +$standby1->init_from_backup( + $primary, $backup_name, + has_streaming => 1, + has_restoring => 1); + +my $connstr_1 = $primary->connstr; +$standby1->append_conf( + 'postgresql.conf', qq( +hot_standby_feedback = on +primary_slot_name = 'sb1_slot' +primary_conninfo = '$connstr_1 dbname=postgres' +)); + +$primary->psql('postgres', + q{SELECT pg_create_logical_replication_slot('lsub2_slot', 'test_decoding', false, false, true);} +); + +$primary->psql('postgres', + q{SELECT pg_create_physical_replication_slot('sb1_slot');}); + +my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; +my $offset = -s $standby1->logfile; + +# Start the standby so that slot syncing can begin +$standby1->start; + +$primary->wait_for_catchup('regress_mysub1'); + +# Do not allow any further advancement of the restart_lsn for the lsub1_slot. +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); + +# Wait for the replication slot to become inactive on the publisher +$primary->poll_query_until( + 'postgres', + "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", + 1); + +# Wait for the standby to catch up so that the standby is not lagging behind +# the subscriber. +$primary->wait_for_replay_catchup($standby1); + +# Synchronize the primary server slots to the standby. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Confirm that the logical failover slots are created on the standby and are +# flagged as 'synced' +is( $standby1->safe_psql( + 'postgres', + q{SELECT count(*) = 2 FROM pg_replication_slots WHERE slot_name IN ('lsub1_slot', 'lsub2_slot') AND synced;} + ), + "t", + 'logical slots have synced as true on standby'); + +################################################## +# Test that the synchronized slot will be dropped if the corresponding remote +# slot on the primary server has been dropped. +################################################## + +$primary->psql('postgres', "SELECT pg_drop_replication_slot('lsub2_slot');"); + +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +is( $standby1->safe_psql( + 'postgres', + q{SELECT count(*) = 0 FROM pg_replication_slots WHERE slot_name = 'lsub2_slot';} + ), + "t", + 'synchronized slot has been dropped'); + +################################################## +# Test that a synchronized slot can not be decoded, altered or dropped by the +# user +################################################## + +# Attempting to perform logical decoding on a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql('postgres', + "select * from pg_logical_slot_get_changes('lsub1_slot', NULL, NULL);"); +ok( $stderr =~ + /ERROR: cannot use replication slot "lsub1_slot" for logical decoding/, + "logical decoding is not allowed on synced slot"); + +# Attempting to alter a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql( + 'postgres', + qq[ALTER_REPLICATION_SLOT lsub1_slot (failover);], + replication => 'database'); +ok($stderr =~ /ERROR: cannot alter replication slot "lsub1_slot"/, + "synced slot on standby cannot be altered"); + +# Attempting to drop a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql('postgres', + "SELECT pg_drop_replication_slot('lsub1_slot');"); +ok($stderr =~ /ERROR: cannot drop replication slot "lsub1_slot"/, + "synced slot on standby cannot be dropped"); + done_testing(); diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index abc944e8b8..b7488d760e 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1474,8 +1474,9 @@ pg_replication_slots| SELECT l.slot_name, l.safe_wal_size, l.two_phase, l.conflict_reason, - l.failover - FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, conflict_reason, failover) + l.failover, + l.synced + FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, conflict_reason, failover, synced) LEFT JOIN pg_database d ON ((l.datoid = d.oid))); pg_roles| SELECT pg_authid.rolname, pg_authid.rolsuper, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 91433d439b..d808aad8b0 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2325,6 +2325,7 @@ RelocationBufferInfo RelptrFreePageBtree RelptrFreePageManager RelptrFreePageSpanLeader +RemoteSlot RenameStmt ReopenPtrType ReorderBuffer @@ -2584,6 +2585,7 @@ SlabBlock SlabContext SlabSlot SlotNumber +SlotSyncCtxStruct SlruCtl SlruCtlData SlruErrorCause -- 2.43.0 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-12 09:40 Amit Kapila <[email protected]> parent: Zhijie Hou (Fujitsu) <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: Amit Kapila @ 2024-02-12 09:40 UTC (permalink / raw) To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Sun, Feb 11, 2024 at 6:53 PM Zhijie Hou (Fujitsu) <[email protected]> wrote: > > Agreed. Here is the V84 patch which addressed this. > Few comments: ============= 1. Isn't the new function (pg_sync_replication_slots()) allowed to sync the slots from physical standby to another cascading standby? Won't it be better to simply disallow syncing slots on cascading standby to keep it consistent with slotsync worker behavior? 2. Previously, I commented to keep the declaration and definition of functions in the same order but I see that it still doesn't match in the below case: @@ -44,6 +46,7 @@ extern void WalSndWakeup(bool physical, bool logical); extern void WalSndInitStopping(void); extern void WalSndWaitStopping(void); extern void HandleWalSndInitStopping(void); +extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); extern void WalSndRqstFileReload(void); I think we can keep the new declaration just before WalSndSignals(). That would be more consistent. 3. + <para> + True if this is a logical slot that was synced from a primary server. + </para> + <para> + On a hot standby, the slots with the synced column marked as true can + neither be used for logical decoding nor dropped by the user. The value I don't think we need a separate para here. Apart from this, I have made several cosmetic changes in the attached. Please include these in the next version unless you see any problems. -- With Regards, Amit Kapila. diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index b925c7fe7d..06103d6837 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -80,8 +80,8 @@ typedef struct RemoteSlot } RemoteSlot; /* - * If necessary, update local synced slot metadata based on the data from the - * remote slot. + * If necessary, update the local synced slot's metadata based on the data + * from the remote slot. * * If no update was needed (the data of the remote slot is the same as the * local slot) return false, otherwise true. @@ -99,7 +99,8 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) xmin_changed = (remote_slot->catalog_xmin != slot->data.catalog_xmin); restart_lsn_changed = (remote_slot->restart_lsn != slot->data.restart_lsn); - if (!xmin_changed && !restart_lsn_changed && + if (!xmin_changed && + !restart_lsn_changed && remote_dbid == slot->data.database && remote_slot->two_phase == slot->data.two_phase && remote_slot->failover == slot->data.failover && @@ -131,7 +132,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) } /* - * Get the list of local logical slots which are synchronized from the + * Get the list of local logical slots that are synchronized from the * primary server. */ static List * @@ -302,7 +303,7 @@ reserve_wal_for_local_slot(XLogRecPtr restart_lsn) * Normally, we can determine it by using the last removed segment * number. However, if no WAL segment files have been removed by a * checkpoint since startup, we need to search for the oldest segment - * file currently existing in XLOGDIR. + * file from the current timeline existing in XLOGDIR. */ oldest_segno = XLogGetLastRemovedSegno() + 1; @@ -353,7 +354,7 @@ update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) * We do not drop the slot because the restart_lsn can be ahead of the * current location when recreating the slot in the next cycle. It may * take more time to create such a slot. Therefore, we keep this slot - * and attempt the wait and synchronization in the next cycle. + * and attempt the synchronization in the next cycle. */ return; } @@ -736,7 +737,7 @@ validate_primary_slot_name(WalReceiverConn *wrconn) /* * Check all necessary GUCs for slot synchronization are set - * appropriately, otherwise raise ERROR. + * appropriately, otherwise, raise ERROR. */ void ValidateSlotSyncParams(void) @@ -768,10 +769,7 @@ ValidateSlotSyncParams(void) errmsg("bad configuration for slot synchronization"), errhint("\"%s\" must be enabled.", "hot_standby_feedback")); - /* - * Logical decoding requires wal_level >= logical and we currently only - * synchronize logical slots. - */ + /* Logical slot sync/creation requires wal_level >= logical. */ if (wal_level < WAL_LEVEL_LOGICAL) ereport(ERROR, errcode(ERRCODE_INVALID_PARAMETER_VALUE), diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index 04738ab764..7df22a0251 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -738,7 +738,7 @@ ReplicationSlotAlter(const char *name, bool failover) /* * Do not allow users to enable failover for temporary slots as we do not - * support sync temporary slots to the standby. + * support syncing temporary slots to the standby. */ if (failover && MyReplicationSlot->data.persistency == RS_TEMPORARY) ereport(ERROR, Attachments: [text/plain] v84_0001_amit_1.patch.txt (3.5K, ../../CAA4eK1LuA9r96cYypQjWOc=E+vo5756+TAJXqsab3rRng=uasQ@mail.gmail.com/2-v84_0001_amit_1.patch.txt) download | inline diff: diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index b925c7fe7d..06103d6837 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -80,8 +80,8 @@ typedef struct RemoteSlot } RemoteSlot; /* - * If necessary, update local synced slot metadata based on the data from the - * remote slot. + * If necessary, update the local synced slot's metadata based on the data + * from the remote slot. * * If no update was needed (the data of the remote slot is the same as the * local slot) return false, otherwise true. @@ -99,7 +99,8 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) xmin_changed = (remote_slot->catalog_xmin != slot->data.catalog_xmin); restart_lsn_changed = (remote_slot->restart_lsn != slot->data.restart_lsn); - if (!xmin_changed && !restart_lsn_changed && + if (!xmin_changed && + !restart_lsn_changed && remote_dbid == slot->data.database && remote_slot->two_phase == slot->data.two_phase && remote_slot->failover == slot->data.failover && @@ -131,7 +132,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) } /* - * Get the list of local logical slots which are synchronized from the + * Get the list of local logical slots that are synchronized from the * primary server. */ static List * @@ -302,7 +303,7 @@ reserve_wal_for_local_slot(XLogRecPtr restart_lsn) * Normally, we can determine it by using the last removed segment * number. However, if no WAL segment files have been removed by a * checkpoint since startup, we need to search for the oldest segment - * file currently existing in XLOGDIR. + * file from the current timeline existing in XLOGDIR. */ oldest_segno = XLogGetLastRemovedSegno() + 1; @@ -353,7 +354,7 @@ update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) * We do not drop the slot because the restart_lsn can be ahead of the * current location when recreating the slot in the next cycle. It may * take more time to create such a slot. Therefore, we keep this slot - * and attempt the wait and synchronization in the next cycle. + * and attempt the synchronization in the next cycle. */ return; } @@ -736,7 +737,7 @@ validate_primary_slot_name(WalReceiverConn *wrconn) /* * Check all necessary GUCs for slot synchronization are set - * appropriately, otherwise raise ERROR. + * appropriately, otherwise, raise ERROR. */ void ValidateSlotSyncParams(void) @@ -768,10 +769,7 @@ ValidateSlotSyncParams(void) errmsg("bad configuration for slot synchronization"), errhint("\"%s\" must be enabled.", "hot_standby_feedback")); - /* - * Logical decoding requires wal_level >= logical and we currently only - * synchronize logical slots. - */ + /* Logical slot sync/creation requires wal_level >= logical. */ if (wal_level < WAL_LEVEL_LOGICAL) ereport(ERROR, errcode(ERRCODE_INVALID_PARAMETER_VALUE), diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index 04738ab764..7df22a0251 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -738,7 +738,7 @@ ReplicationSlotAlter(const char *name, bool failover) /* * Do not allow users to enable failover for temporary slots as we do not - * support sync temporary slots to the standby. + * support syncing temporary slots to the standby. */ if (failover && MyReplicationSlot->data.persistency == RS_TEMPORARY) ereport(ERROR, ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-12 10:03 Bertrand Drouvot <[email protected]> parent: Zhijie Hou (Fujitsu) <[email protected]> 1 sibling, 2 replies; 46+ messages in thread From: Bertrand Drouvot @ 2024-02-12 10:03 UTC (permalink / raw) To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> Hi, On Sun, Feb 11, 2024 at 01:23:19PM +0000, Zhijie Hou (Fujitsu) wrote: > On Saturday, February 10, 2024 9:10 PM Amit Kapila <[email protected]> wrote: > > > > On Sat, Feb 10, 2024 at 5:31 PM Masahiko Sawada <[email protected]> > > wrote: > > > > > > On Fri, Feb 9, 2024 at 4:08 PM Zhijie Hou (Fujitsu) > > > <[email protected]> wrote: > > > > > > > Another alternative is to register the callback when calling > > > > slotsync functions and unregister it after the function call. And > > > > register the callback in > > > > slotsyncworkmain() for the slotsync worker patch, although this may > > > > adds a few more codes. > > > > > > Another idea is that SyncReplicationSlots() calls synchronize_slots() > > > in PG_ENSURE_ERROR_CLEANUP() block instead of PG_TRY(), to make sure > > > to clear the flag in case of ERROR or FATAL. And the slotsync worker > > > uses the before_shmem_callback to clear the flag. > > > > > > > +1. This sounds like a better way to clear the flag. > > Agreed. Here is the V84 patch which addressed this. > > Apart from above, I removed the txn start/end codes from 0001 as they are used > in the slotsync worker patch. And I also ran pgindent and pgperltidy for the > patch. > Thanks! A few random comments: 001 === " For the synchronization to work, it is mandatory to have a physical replication slot between the primary and the standby, " Maybe mention "primary_slot_name" here? 002 === + <para> + Synchronize the logical failover slots from the primary server to the standby server. should we say "logical failover replication slots" instead? 003 === + If, after executing the function, + <link linkend="guc-hot-standby-feedback"> + <varname>hot_standby_feedback</varname></link> is disabled on + the standby or the physical slot configured in + <link linkend="guc-primary-slot-name"> + <varname>primary_slot_name</varname></link> is + removed, I think another option that could lead to slot invalidation is if primary_slot_name is NULL or miss-configured. Indeed hot_standby_feedback would be working (for the catalog_xmin) but only as long as the standby is up and running. 004 === + on the standby. For the synchronization to work, it is mandatory to + have a physical replication slot between the primary and the standby, should we mention primary_slot_name here? 005 === + To resume logical replication after failover from the synced logical + slots, the subscription's 'conninfo' must be altered Only in a pub/sub context but not for other ways of using the logical replication slot(s). 006 === + neither be used for logical decoding nor dropped by the user what about "nor dropped manually"? 007 === +typedef struct SlotSyncCtxStruct +{ Should we remove "Struct" from the struct name? 008 === + ereport(LOG, + errmsg("dropped replication slot \"%s\" of dbid %d", + NameStr(local_slot->data.name), + local_slot->data.database)); We emit a message when an "invalidated" slot is dropped but not when we create a slot. Shouldn't we emit a message when we create a synced slot on the standby? I think that could be confusing to see "a drop" message not followed by "a create" one when it's expected (slot valid on the primary for example). 009 === Regarding 040_standby_failover_slots_sync.pl what about adding tests for? - synced slot invalidation (and ensure it's recreated once pg_sync_replication_slots() is called and when the slot in primary is valid) - cannot enable failover for a temporary replication slot - replication slots can only be synchronized from a standby server Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-12 10:49 Amit Kapila <[email protected]> parent: Bertrand Drouvot <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: Amit Kapila @ 2024-02-12 10:49 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Mon, Feb 12, 2024 at 3:33 PM Bertrand Drouvot <[email protected]> wrote: > > A few random comments: > > > 003 === > > + If, after executing the function, > + <link linkend="guc-hot-standby-feedback"> > + <varname>hot_standby_feedback</varname></link> is disabled on > + the standby or the physical slot configured in > + <link linkend="guc-primary-slot-name"> > + <varname>primary_slot_name</varname></link> is > + removed, > > I think another option that could lead to slot invalidation is if primary_slot_name > is NULL or miss-configured. > If the primary_slot_name is NULL then the function will error out. So, not sure, if we need to say anything explicitly here. > Indeed hot_standby_feedback would be working > (for the catalog_xmin) but only as long as the standby is up and running. > ... > > 005 === > > + To resume logical replication after failover from the synced logical > + slots, the subscription's 'conninfo' must be altered > > Only in a pub/sub context but not for other ways of using the logical replication > slot(s). > Right, but what additional information do you want here? I thought we were speaking about the in-build logical replication here so this is okay. > > 008 === > > + ereport(LOG, > + errmsg("dropped replication slot \"%s\" of dbid %d", > + NameStr(local_slot->data.name), > + local_slot->data.database)); > > We emit a message when an "invalidated" slot is dropped but not when we create > a slot. Shouldn't we emit a message when we create a synced slot on the standby? > > I think that could be confusing to see "a drop" message not followed by "a create" > one when it's expected (slot valid on the primary for example). > Isn't the below message for sync-ready slot sufficient? Otherwise, in most cases, we will LOG multiple similar messages. + ereport(LOG, + errmsg("newly created slot \"%s\" is sync-ready now", + remote_slot->name)); -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-12 14:06 Bertrand Drouvot <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Bertrand Drouvot @ 2024-02-12 14:06 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> Hi, On Mon, Feb 12, 2024 at 04:19:33PM +0530, Amit Kapila wrote: > On Mon, Feb 12, 2024 at 3:33 PM Bertrand Drouvot > <[email protected]> wrote: > > > > A few random comments: > > > > > > 003 === > > > > + If, after executing the function, > > + <link linkend="guc-hot-standby-feedback"> > > + <varname>hot_standby_feedback</varname></link> is disabled on > > + the standby or the physical slot configured in > > + <link linkend="guc-primary-slot-name"> > > + <varname>primary_slot_name</varname></link> is > > + removed, > > > > I think another option that could lead to slot invalidation is if primary_slot_name > > is NULL or miss-configured. > > > > If the primary_slot_name is NULL then the function will error out. Yeah right, it had to be non NULL initially so we know there is a physical slot (if not dropped) that should prevent conflicts at the first place (should hsf be on). Please forget about comment 003 then. > > > > 005 === > > > > + To resume logical replication after failover from the synced logical > > + slots, the subscription's 'conninfo' must be altered > > > > Only in a pub/sub context but not for other ways of using the logical replication > > slot(s). > > > > Right, but what additional information do you want here? I thought we > were speaking about the in-build logical replication here so this is > okay. The "Logical Decoding Concepts" sub-chapter also mentions "Logical decoding clients" so I was not sure the part added in the patch was for in-build logical replication only. Or maybe just reword that way "In case of in-build logical replication, to resume after failover from the synced......"? > > > > > 008 === > > > > + ereport(LOG, > > + errmsg("dropped replication slot \"%s\" of dbid %d", > > + NameStr(local_slot->data.name), > > + local_slot->data.database)); > > > > We emit a message when an "invalidated" slot is dropped but not when we create > > a slot. Shouldn't we emit a message when we create a synced slot on the standby? > > > > I think that could be confusing to see "a drop" message not followed by "a create" > > one when it's expected (slot valid on the primary for example). > > > > Isn't the below message for sync-ready slot sufficient? Otherwise, in > most cases, we will LOG multiple similar messages. > > + ereport(LOG, > + errmsg("newly created slot \"%s\" is sync-ready now", > + remote_slot->name)); Yes it is sufficient if we reach it. For example during some test, I was able to go through this code path: Breakpoint 2, update_and_persist_local_synced_slot (remote_slot=0x56450e7c49c0, remote_dbid=5) at slotsync.c:340 340 ReplicationSlot *slot = MyReplicationSlot; (gdb) n 346 if (remote_slot->restart_lsn < slot->data.restart_lsn || (gdb) 347 TransactionIdPrecedes(remote_slot->catalog_xmin, (gdb) 346 if (remote_slot->restart_lsn < slot->data.restart_lsn || (gdb) 358 return; means exiting from update_and_persist_local_synced_slot() without reaching the "newly created slot" message (the slot on the primary was "inactive"). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* RE: Synchronizing slots from primary to standby @ 2024-02-13 01:15 Zhijie Hou (Fujitsu) <[email protected]> parent: Bertrand Drouvot <[email protected]> 1 sibling, 0 replies; 46+ messages in thread From: Zhijie Hou (Fujitsu) @ 2024-02-13 01:15 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Monday, February 12, 2024 6:03 PM Bertrand Drouvot <[email protected]> wrote: > > Hi, > > On Sun, Feb 11, 2024 at 01:23:19PM +0000, Zhijie Hou (Fujitsu) wrote: > > On Saturday, February 10, 2024 9:10 PM Amit Kapila > <[email protected]> wrote: > > > > > > On Sat, Feb 10, 2024 at 5:31 PM Masahiko Sawada > > > <[email protected]> > > > wrote: > > > > > > > > On Fri, Feb 9, 2024 at 4:08 PM Zhijie Hou (Fujitsu) > > > > <[email protected]> wrote: > > > > > > > > > Another alternative is to register the callback when calling > > > > > slotsync functions and unregister it after the function call. > > > > > And register the callback in > > > > > slotsyncworkmain() for the slotsync worker patch, although this > > > > > may adds a few more codes. > > > > > > > > Another idea is that SyncReplicationSlots() calls > > > > synchronize_slots() in PG_ENSURE_ERROR_CLEANUP() block instead of > > > > PG_TRY(), to make sure to clear the flag in case of ERROR or > > > > FATAL. And the slotsync worker uses the before_shmem_callback to clear > the flag. > > > > > > > > > > +1. This sounds like a better way to clear the flag. > > > > Agreed. Here is the V84 patch which addressed this. > > > > Apart from above, I removed the txn start/end codes from 0001 as they > > are used in the slotsync worker patch. And I also ran pgindent and > > pgperltidy for the patch. > > > > Thanks! > > A few random comments: Thanks for the comments. > > 001 === > > " > For > the synchronization to work, it is mandatory to have a physical replication slot > between the primary and the standby, " > > Maybe mention "primary_slot_name" here? Added. > > 002 === > > + <para> > + Synchronize the logical failover slots from the primary server to the > standby server. > > should we say "logical failover replication slots" instead? Changed. > > 003 === > > + If, after executing the function, > + <link linkend="guc-hot-standby-feedback"> > + <varname>hot_standby_feedback</varname></link> is disabled > on > + the standby or the physical slot configured in > + <link linkend="guc-primary-slot-name"> > + <varname>primary_slot_name</varname></link> is > + removed, > > I think another option that could lead to slot invalidation is if primary_slot_name > is NULL or miss-configured. Indeed hot_standby_feedback would be working > (for the catalog_xmin) but only as long as the standby is up and running. I didn't change this based on the discussion. > > 004 === > > + on the standby. For the synchronization to work, it is mandatory to > + have a physical replication slot between the primary and the > + standby, > > should we mention primary_slot_name here? Added. > > 005 === > > + To resume logical replication after failover from the synced logical > + slots, the subscription's 'conninfo' must be altered > > Only in a pub/sub context but not for other ways of using the logical replication > slot(s). I am not very sure about this, because the 3-rd part logicalrep can also have their own replication origin, so I didn't change for now, but will think over this. > > 006 === > > + neither be used for logical decoding nor dropped by the user > > what about "nor dropped manually"? Changed. > > 007 === > > +typedef struct SlotSyncCtxStruct > +{ > > Should we remove "Struct" from the struct name? The name was named based on some other comment to be consistent with LogicalReplCtxStruct, so I didn't change this. If other also prefer without struct, we can change it later. > 008 === > > + ereport(LOG, > + errmsg("dropped replication slot > \"%s\" of dbid %d", > + > NameStr(local_slot->data.name), > + > + local_slot->data.database)); > > We emit a message when an "invalidated" slot is dropped but not when we > create a slot. Shouldn't we emit a message when we create a synced slot on the > standby? > > I think that could be confusing to see "a drop" message not followed by "a > create" > one when it's expected (slot valid on the primary for example). I think we will report "sync-ready" for newly synced slot, for newly created temporary slots, I am not sure do we need to report log to them, because they will be dropped on promotion anyway. But if others also prefer to log, I am fine with that. > > 009 === > > Regarding 040_standby_failover_slots_sync.pl what about adding tests for? > > - synced slot invalidation (and ensure it's recreated once > pg_sync_replication_slots() is called and when the slot in primary is valid) Will try this in next version. > - cannot enable failover for a temporary replication slot Added. > - replication slots can only be synchronized from a standby server Added. Best Regards, Hou zj ^ permalink raw reply [nested|flat] 46+ messages in thread
* RE: Synchronizing slots from primary to standby @ 2024-02-13 01:15 Zhijie Hou (Fujitsu) <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 2 replies; 46+ messages in thread From: Zhijie Hou (Fujitsu) @ 2024-02-13 01:15 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Monday, February 12, 2024 5:40 PM Amit Kapila <[email protected]> wrote: > > On Sun, Feb 11, 2024 at 6:53 PM Zhijie Hou (Fujitsu) <[email protected]> > wrote: > > > > Agreed. Here is the V84 patch which addressed this. > > > > Few comments: > ============= > 1. Isn't the new function (pg_sync_replication_slots()) allowed to sync the slots > from physical standby to another cascading standby? > Won't it be better to simply disallow syncing slots on cascading standby to keep > it consistent with slotsync worker behavior? > > 2. > Previously, I commented to keep the declaration and definition of functions in > the same order but I see that it still doesn't match in the below case: > > @@ -44,6 +46,7 @@ extern void WalSndWakeup(bool physical, bool logical); > extern void WalSndInitStopping(void); extern void WalSndWaitStopping(void); > extern void HandleWalSndInitStopping(void); > +extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); > extern void WalSndRqstFileReload(void); > > I think we can keep the new declaration just before WalSndSignals(). > That would be more consistent. > > 3. > + <para> > + True if this is a logical slot that was synced from a primary server. > + </para> > + <para> > + On a hot standby, the slots with the synced column marked as true can > + neither be used for logical decoding nor dropped by the user. > + The value > > I don't think we need a separate para here. > > Apart from this, I have made several cosmetic changes in the attached. > Please include these in the next version unless you see any problems. Thanks for the comments, I have addressed them. Here is the new version patch which addressed above and most of Bertrand's comments. TODO: trying to add one test for the case the slot is valid on primary while the synced slots is invalidated on the standby. Best Regards, Houzj Attachments: [application/octet-stream] v85-0001-Add-a-slot-synchronization-function.patch (73.0K, ../../OS0PR01MB571696ABBAB8CF8CA5CA19A6944F2@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v85-0001-Add-a-slot-synchronization-function.patch) download | inline diff: From 71aa0d89e14000a0b16126f757d5c85729480aac Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Wed, 7 Feb 2024 10:37:31 +0530 Subject: [PATCH v856 1/2] Add a slot synchronization function. This commit introduces a new SQL function pg_sync_replication_slots() which is used to synchronize the logical replication slots from the primary server to the physical standby so that logical replication can be resumed after failover. A new 'synced' flag is introduced in pg_replication_slots view, indicating whether the slot has been synchronized from the primary server. On a standby, synced slots cannot be dropped or consumed, and any attempt to perform logical decoding on them will result in an error. The logical replication slots on the primary can be synchronized to the hot standby by enabling failover during slot creation (e.g. using the "failover" parameter of pg_create_logical_replication_slot(), or using the "failover" option of the CREATE SUBSCRIPTION command), and then calling pg_sync_replication_slots() function on the standby. For the synchronization to work, it is mandatory to have a physical replication slot between the primary and the standby, 'hot_standby_feedback' must be enabled on the standby and a valid dbname must be specified in 'primary_conninfo'. If a logical slot is invalidated on the primary, then that slot on the standby is also invalidated. If a logical slot on the primary is valid but is invalidated on the standby, then that slot is dropped but will be recreated on the standby in next pg_sync_replication_slots() call provided the slot still exists on the primary server. It is okay to recreate such slots as long as these are not consumable on the standby (which is the case currently). This situation may occur due to the following reasons: - The 'max_slot_wal_keep_size' on the standby is insufficient to retain WAL records from the restart_lsn of the slot. - 'primary_slot_name' is temporarily reset to null and the physical slot is removed. We may also see the slots invalidated and dropped on the standby if the primary changes 'wal_level' to a level lower than logical. Changing the primary 'wal_level' to a level lower than logical is only possible if the logical slots are removed on the primary server, so it's expected to see the slots being removed on the standby too (and re-created if they are re-created on the primary server). The slots synchronization status on the standby can be monitored using 'synced' column of pg_replication_slots view. --- .../test_decoding/expected/permissions.out | 3 + contrib/test_decoding/expected/slot.out | 2 + contrib/test_decoding/sql/permissions.sql | 1 + contrib/test_decoding/sql/slot.sql | 1 + doc/src/sgml/config.sgml | 9 +- doc/src/sgml/func.sgml | 34 +- doc/src/sgml/logicaldecoding.sgml | 56 ++ doc/src/sgml/protocol.sgml | 6 +- doc/src/sgml/system-views.sgml | 20 +- src/backend/catalog/system_views.sql | 3 +- src/backend/replication/logical/Makefile | 1 + src/backend/replication/logical/logical.c | 12 + src/backend/replication/logical/meson.build | 1 + src/backend/replication/logical/slotsync.c | 896 ++++++++++++++++++ src/backend/replication/slot.c | 104 +- src/backend/replication/slotfuncs.c | 72 +- src/backend/replication/walsender.c | 19 +- src/backend/storage/ipc/ipci.c | 3 + src/include/catalog/pg_proc.dat | 10 +- src/include/replication/slot.h | 19 +- src/include/replication/slotsync.h | 23 + src/include/replication/walsender.h | 3 + .../t/040_standby_failover_slots_sync.pl | 124 +++ src/test/regress/expected/rules.out | 5 +- src/tools/pgindent/typedefs.list | 2 + 25 files changed, 1394 insertions(+), 35 deletions(-) create mode 100644 src/backend/replication/logical/slotsync.c create mode 100644 src/include/replication/slotsync.h diff --git a/contrib/test_decoding/expected/permissions.out b/contrib/test_decoding/expected/permissions.out index d6eaba8c55..8d100646ce 100644 --- a/contrib/test_decoding/expected/permissions.out +++ b/contrib/test_decoding/expected/permissions.out @@ -64,6 +64,9 @@ DETAIL: Only roles with the REPLICATION attribute may use replication slots. SELECT pg_drop_replication_slot('regression_slot'); ERROR: permission denied to use replication slots DETAIL: Only roles with the REPLICATION attribute may use replication slots. +SELECT pg_sync_replication_slots(); +ERROR: permission denied to use replication slots +DETAIL: Only roles with the REPLICATION attribute may use replication slots. RESET ROLE; -- replication users can drop superuser created slots SET ROLE regress_lr_superuser; diff --git a/contrib/test_decoding/expected/slot.out b/contrib/test_decoding/expected/slot.out index 261d8886d3..349ab2d380 100644 --- a/contrib/test_decoding/expected/slot.out +++ b/contrib/test_decoding/expected/slot.out @@ -425,6 +425,8 @@ SELECT 'init' FROM pg_create_logical_replication_slot('failover_default_slot', ' init (1 row) +SELECT 'init' FROM pg_create_logical_replication_slot('failover_true_temp_slot', 'test_decoding', true, false, true); +ERROR: cannot enable failover for a temporary replication slot SELECT 'init' FROM pg_create_physical_replication_slot('physical_slot'); ?column? ---------- diff --git a/contrib/test_decoding/sql/permissions.sql b/contrib/test_decoding/sql/permissions.sql index 312b514593..94db936aee 100644 --- a/contrib/test_decoding/sql/permissions.sql +++ b/contrib/test_decoding/sql/permissions.sql @@ -29,6 +29,7 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_d INSERT INTO lr_test VALUES('lr_superuser_init'); SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); SELECT pg_drop_replication_slot('regression_slot'); +SELECT pg_sync_replication_slots(); RESET ROLE; -- replication users can drop superuser created slots diff --git a/contrib/test_decoding/sql/slot.sql b/contrib/test_decoding/sql/slot.sql index 45aeae7fd5..580e3ae3be 100644 --- a/contrib/test_decoding/sql/slot.sql +++ b/contrib/test_decoding/sql/slot.sql @@ -181,6 +181,7 @@ SELECT pg_drop_replication_slot('copied_slot2_notemp'); SELECT 'init' FROM pg_create_logical_replication_slot('failover_true_slot', 'test_decoding', false, false, true); SELECT 'init' FROM pg_create_logical_replication_slot('failover_false_slot', 'test_decoding', false, false, false); SELECT 'init' FROM pg_create_logical_replication_slot('failover_default_slot', 'test_decoding', false, false); +SELECT 'init' FROM pg_create_logical_replication_slot('failover_true_temp_slot', 'test_decoding', true, false, true); SELECT 'init' FROM pg_create_physical_replication_slot('physical_slot'); SELECT slot_name, slot_type, failover FROM pg_replication_slots; diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 61038472c5..037a3b8a64 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4612,8 +4612,13 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" <varname>primary_conninfo</varname> string, or in a separate <filename>~/.pgpass</filename> file on the standby server (use <literal>replication</literal> as the database name). - Do not specify a database name in the - <varname>primary_conninfo</varname> string. + </para> + <para> + For replication slot synchronization (see + <xref linkend="logicaldecoding-replication-slots-synchronization"/>), + it is also necessary to specify a valid <literal>dbname</literal> + in the <varname>primary_conninfo</varname> string. This will only be + used for slot synchronization. It is ignored for streaming. </para> <para> This parameter can only be set in the <filename>postgresql.conf</filename> diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 11d537b341..1b1041195c 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -28075,7 +28075,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset </row> <row> - <entry role="func_table_entry"><para role="func_signature"> + <entry id="pg-create-logical-replication-slot" role="func_table_entry"><para role="func_signature"> <indexterm> <primary>pg_create_logical_replication_slot</primary> </indexterm> @@ -28444,6 +28444,38 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset record is flushed along with its transaction. </para></entry> </row> + + <row> + <entry id="pg-sync-replication-slots" role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>pg_sync_replication_slots</primary> + </indexterm> + <function>pg_sync_replication_slots</function> () + <returnvalue>void</returnvalue> + </para> + <para> + Synchronize the logical failover replication slots from the primary + server to the standby server. This function can only be executed on the + standby server. See + <xref linkend="logicaldecoding-replication-slots-synchronization"/> for details. + </para> + + <caution> + <para> + If, after executing the function, + <link linkend="guc-hot-standby-feedback"> + <varname>hot_standby_feedback</varname></link> is disabled on + the standby or the physical slot configured in + <link linkend="guc-primary-slot-name"> + <varname>primary_slot_name</varname></link> is + removed, then it is possible that the necessary rows of the + synchronized slot will be removed by the VACUUM process on the primary + server, resulting in the synchronized slot becoming invalidated. + </para> + </caution> + </entry> + </row> + </tbody> </tgroup> </table> diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index cd152d4ced..dae2d5865e 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -358,6 +358,62 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU So if a slot is no longer required it should be dropped. </para> </caution> + + </sect2> + + <sect2 id="logicaldecoding-replication-slots-synchronization"> + <title>Replication Slot Synchronization</title> + <para> + The logical replication slots on the primary can be synchronized to + the hot standby by enabling <literal>failover</literal> during slot + creation (e.g., using the <literal>failover</literal> parameter of + <link linkend="pg-create-logical-replication-slot"> + <function>pg_create_logical_replication_slot</function></link>, or + using the <link linkend="sql-createsubscription-params-with-failover"> + <literal>failover</literal></link> option of + <command>CREATE SUBSCRIPTION</command>), and then calling + <link linkend="pg-sync-replication-slots"> + <function>pg_sync_replication_slots</function></link> + on the standby. For the synchronization to work, it is mandatory to + have a physical replication slot between the primary and the standby (e.g., + <link linkend="guc-primary-slot-name"><varname>primary_slot_name</varname></link> + should be configured on the standby), and + <link linkend="guc-hot-standby-feedback"><varname>hot_standby_feedback</varname></link> + must be enabled on the standby. It is also necessary to specify a valid + <literal>dbname</literal> in the + <link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>. + </para> + + <para> + The ability to resume logical replication after failover depends upon the + <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield> + value for the synchronized slots on the standby at the time of failover. + Only persistent slots that have attained synced state as true on the standby + before failover can be used for logical replication after failover. + Temporary slots will be dropped, therefore logical replication for those + slots cannot be resumed. For example, if the synchronized slot could not + become persistent on the standby due to a disabled subscription, then the + subscription cannot be resumed after failover even when it is enabled. + </para> + + <para> + To resume logical replication after failover from the synced logical + slots, the subscription's 'conninfo' must be altered to point to the + new primary server. This is done using + <link linkend="sql-altersubscription-params-connection"><command>ALTER SUBSCRIPTION ... CONNECTION</command></link>. + It is recommended that subscriptions are first disabled before promoting + the standby and are re-enabled after altering the connection string. + </para> + <caution> + <para> + There is a chance that the old primary is up again during the promotion + and if subscriptions are not disabled, the logical subscribers may + continue to receive data from the old primary server even after promotion + until the connection string is altered. This might result in data + inconsistency issues, preventing the logical subscribers from being + able to continue replication from the new primary server. + </para> + </caution> </sect2> <sect2 id="logicaldecoding-explanation-output-plugins"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index ed1d62f5f8..7ffddfbd82 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -2062,7 +2062,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" <term><literal>FAILOVER [ <replaceable class="parameter">boolean</replaceable> ]</literal></term> <listitem> <para> - If true, the slot is enabled to be synced to the standbys. + If true, the slot is enabled to be synced to the standbys + so that logical replication can be resumed after failover. The default is false. </para> </listitem> @@ -2162,7 +2163,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" <term><literal>FAILOVER [ <replaceable class="parameter">boolean</replaceable> ]</literal></term> <listitem> <para> - If true, the slot is enabled to be synced to the standbys. + If true, the slot is enabled to be synced to the standbys + so that logical replication can be resumed after failover. </para> </listitem> </varlistentry> diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index dd468b31ea..be90edd0e2 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2561,10 +2561,26 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <structfield>failover</structfield> <type>bool</type> </para> <para> - True if this is a logical slot enabled to be synced to the standbys. - Always false for physical slots. + True if this is a logical slot enabled to be synced to the standbys + so that logical replication can be resumed from the new primary + after failover. Always false for physical slots. </para></entry> </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>synced</structfield> <type>bool</type> + </para> + <para> + True if this is a logical slot that was synced from a primary server. + On a hot standby, the slots with the synced column marked as true can + neither be used for logical decoding nor dropped manually. The value + of this column has no meaning on the primary server; the column value on + the primary is default false for all slots but may (if leftover from a + promoted standby) also be true. + </para></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 6791bff9dd..04227a72d1 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1024,7 +1024,8 @@ CREATE VIEW pg_replication_slots AS L.safe_wal_size, L.two_phase, L.conflict_reason, - L.failover + L.failover, + L.synced FROM pg_get_replication_slots() AS L LEFT JOIN pg_database D ON (L.datoid = D.oid); diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index 2dc25e37bb..ba03eeff1c 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -25,6 +25,7 @@ OBJS = \ proto.o \ relation.o \ reorderbuffer.o \ + slotsync.o \ snapbuild.o \ tablesync.o \ worker.o diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index ca09c683f1..a53815f2ed 100644 --- a/src/backend/replication/logical/logical.c +++ b/src/backend/replication/logical/logical.c @@ -524,6 +524,18 @@ CreateDecodingContext(XLogRecPtr start_lsn, errmsg("replication slot \"%s\" was not created in this database", NameStr(slot->data.name)))); + /* + * Do not allow consumption of a "synchronized" slot until the standby + * gets promoted. + */ + if (RecoveryInProgress() && slot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot use replication slot \"%s\" for logical decoding", + NameStr(slot->data.name)), + errdetail("This slot is being synchronized from the primary server."), + errhint("Specify another replication slot.")); + /* * Check if slot has been invalidated due to max_slot_wal_keep_size. Avoid * "cannot get changes" wording in this errmsg because that'd be diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build index 1050eb2c09..3dec36a6de 100644 --- a/src/backend/replication/logical/meson.build +++ b/src/backend/replication/logical/meson.build @@ -11,6 +11,7 @@ backend_sources += files( 'proto.c', 'relation.c', 'reorderbuffer.c', + 'slotsync.c', 'snapbuild.c', 'tablesync.c', 'worker.c', diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c new file mode 100644 index 0000000000..feb04e1451 --- /dev/null +++ b/src/backend/replication/logical/slotsync.c @@ -0,0 +1,896 @@ +/*------------------------------------------------------------------------- + * slotsync.c + * Functionality for synchronizing slots to a standby server from the + * primary server. + * + * Copyright (c) 2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/slotsync.c + * + * This file contains the code for slot synchronization on a physical standby + * to fetch logical failover slots information from the primary server, create + * the slots on the standby and synchronize them. This is done by a call to SQL + * function pg_sync_replication_slots. + * + * If on physical standby, the WAL corresponding to the remote's restart_lsn + * is not available or the remote's catalog_xmin precedes the oldest xid for which + * it is guaranteed that rows wouldn't have been removed then we cannot create + * the local standby slot because that would mean moving the local slot + * backward and decoding won't be possible via such a slot. In this case, the + * slot will be marked as RS_TEMPORARY. Once the primary server catches up, + * the slot will be marked as RS_PERSISTENT (which means sync-ready) after + * which we can call pg_sync_replication_slots() periodically to perform + * syncs. + * + * Any standby synchronized slots will be dropped if they no longer need + * to be synchronized. See comment atop drop_local_obsolete_slots() for more + * details. + *--------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/xlog_internal.h" +#include "access/xlogrecovery.h" +#include "catalog/pg_database.h" +#include "commands/dbcommands.h" +#include "replication/logical.h" +#include "replication/slotsync.h" +#include "storage/ipc.h" +#include "storage/lmgr.h" +#include "storage/procarray.h" +#include "utils/builtins.h" +#include "utils/pg_lsn.h" + +/* Struct for sharing information to control slot synchronization. */ +typedef struct SlotSyncCtxStruct +{ + /* prevents concurrent slot syncs to avoid slot overwrites */ + bool syncing; + slock_t mutex; +} SlotSyncCtxStruct; + +SlotSyncCtxStruct *SlotSyncCtx = NULL; + +/* + * Flag to tell if we are syncing replication slots. Unlike the 'syncing' flag + * in SlotSyncCtxStruct, this flag is true only if the current process is + * performing slot synchronization. + */ +static bool syncing_slots = false; + +/* + * Structure to hold information fetched from the primary server about a logical + * replication slot. + */ +typedef struct RemoteSlot +{ + char *name; + char *plugin; + char *database; + bool two_phase; + bool failover; + XLogRecPtr restart_lsn; + XLogRecPtr confirmed_lsn; + TransactionId catalog_xmin; + + /* RS_INVAL_NONE if valid, or the reason of invalidation */ + ReplicationSlotInvalidationCause invalidated; +} RemoteSlot; + +/* + * If necessary, update the local synced slot's metadata based on the data + * from the remote slot. + * + * If no update was needed (the data of the remote slot is the same as the + * local slot) return false, otherwise true. + */ +static bool +update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot = MyReplicationSlot; + bool xmin_changed; + bool restart_lsn_changed; + NameData plugin_name; + + Assert(slot->data.invalidated == RS_INVAL_NONE); + + xmin_changed = (remote_slot->catalog_xmin != slot->data.catalog_xmin); + restart_lsn_changed = (remote_slot->restart_lsn != slot->data.restart_lsn); + + if (!xmin_changed && + !restart_lsn_changed && + remote_dbid == slot->data.database && + remote_slot->two_phase == slot->data.two_phase && + remote_slot->failover == slot->data.failover && + remote_slot->confirmed_lsn == slot->data.confirmed_flush && + strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) + return false; + + /* Avoid expensive operations while holding a spinlock. */ + namestrcpy(&plugin_name, remote_slot->plugin); + + SpinLockAcquire(&slot->mutex); + slot->data.plugin = plugin_name; + slot->data.database = remote_dbid; + slot->data.two_phase = remote_slot->two_phase; + slot->data.failover = remote_slot->failover; + slot->data.restart_lsn = remote_slot->restart_lsn; + slot->data.confirmed_flush = remote_slot->confirmed_lsn; + slot->data.catalog_xmin = remote_slot->catalog_xmin; + slot->effective_catalog_xmin = remote_slot->catalog_xmin; + SpinLockRelease(&slot->mutex); + + if (xmin_changed) + ReplicationSlotsComputeRequiredXmin(false); + + if (restart_lsn_changed) + ReplicationSlotsComputeRequiredLSN(); + + return true; +} + +/* + * Get the list of local logical slots that are synchronized from the + * primary server. + */ +static List * +get_local_synced_slots(void) +{ + List *local_slots = NIL; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + + for (int i = 0; i < max_replication_slots; i++) + { + ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i]; + + /* Check if it is a synchronized slot */ + if (s->in_use && s->data.synced) + { + Assert(SlotIsLogical(s)); + local_slots = lappend(local_slots, s); + } + } + + LWLockRelease(ReplicationSlotControlLock); + + return local_slots; +} + +/* + * Helper function to check if local_slot is required to be retained. + * + * Return false either if local_slot does not exist in the remote_slots list + * or is invalidated while the corresponding remote slot is still valid, + * otherwise true. + */ +static bool +local_sync_slot_required(ReplicationSlot *local_slot, List *remote_slots) +{ + bool remote_exists = false; + bool locally_invalidated = false; + + foreach_ptr(RemoteSlot, remote_slot, remote_slots) + { + if (strcmp(remote_slot->name, NameStr(local_slot->data.name)) == 0) + { + remote_exists = true; + + /* + * If remote slot is not invalidated but local slot is marked as + * invalidated, then set locally_invalidated flag. + */ + SpinLockAcquire(&local_slot->mutex); + locally_invalidated = + (remote_slot->invalidated == RS_INVAL_NONE) && + (local_slot->data.invalidated != RS_INVAL_NONE); + SpinLockRelease(&local_slot->mutex); + + break; + } + } + + return (remote_exists && !locally_invalidated); +} + +/* + * Drop local obsolete slots. + * + * Drop the local slots that no longer need to be synced i.e. these either do + * not exist on the primary or are no longer enabled for failover. + * + * Additionally, drop any slots that are valid on the primary but got + * invalidated on the standby. This situation may occur due to the following + * reasons: + * - The 'max_slot_wal_keep_size' on the standby is insufficient to retain WAL + * records from the restart_lsn of the slot. + * - 'primary_slot_name' is temporarily reset to null and the physical slot is + * removed. + * These dropped slots will get recreated in next sync-cycle and it is okay to + * drop and recreate such slots as long as these are not consumable on the + * standby (which is the case currently). + * + * Note: Change of 'wal_level' on the primary server to a level lower than + * logical may also result in slot invalidation and removal on the standby. + * This is because such 'wal_level' change is only possible if the logical + * slots are removed on the primary server, so it's expected to see the + * slots being invalidated and removed on the standby too (and re-created + * if they are re-created on the primary server). + */ +static void +drop_local_obsolete_slots(List *remote_slot_list) +{ + List *local_slots = get_local_synced_slots(); + + foreach_ptr(ReplicationSlot, local_slot, local_slots) + { + /* Drop the local slot if it is not required to be retained. */ + if (!local_sync_slot_required(local_slot, remote_slot_list)) + { + bool synced_slot; + + /* + * Use shared lock to prevent a conflict with + * ReplicationSlotsDropDBSlots(), trying to drop the same slot + * during a drop-database operation. + */ + LockSharedObject(DatabaseRelationId, local_slot->data.database, + 0, AccessShareLock); + + /* + * In the small window between getting the slot to drop and + * locking the database, there is a possibility of a parallel + * database drop by the startup process and the creation of a new + * slot by the user. This new user-created slot may end up using + * the same shared memory as that of 'local_slot'. Thus check if + * local_slot is still the synced one before performing actual + * drop. + */ + SpinLockAcquire(&local_slot->mutex); + synced_slot = local_slot->in_use && local_slot->data.synced; + SpinLockRelease(&local_slot->mutex); + + if (synced_slot) + { + ReplicationSlotAcquire(NameStr(local_slot->data.name), true); + ReplicationSlotDropAcquired(); + } + + UnlockSharedObject(DatabaseRelationId, local_slot->data.database, + 0, AccessShareLock); + + ereport(LOG, + errmsg("dropped replication slot \"%s\" of dbid %d", + NameStr(local_slot->data.name), + local_slot->data.database)); + } + } +} + +/* + * Reserve WAL for the currently active local slot using the specified WAL + * location (restart_lsn). + * + * If the given WAL location has been removed, reserve WAL using the oldest + * existing WAL segment. + */ +static void +reserve_wal_for_local_slot(XLogRecPtr restart_lsn) +{ + XLogSegNo oldest_segno; + XLogSegNo segno; + ReplicationSlot *slot = MyReplicationSlot; + + Assert(slot != NULL); + Assert(XLogRecPtrIsInvalid(slot->data.restart_lsn)); + + while (true) + { + SpinLockAcquire(&slot->mutex); + slot->data.restart_lsn = restart_lsn; + SpinLockRelease(&slot->mutex); + + /* Prevent WAL removal as fast as possible */ + ReplicationSlotsComputeRequiredLSN(); + + XLByteToSeg(slot->data.restart_lsn, segno, wal_segment_size); + + /* + * Find the oldest existing WAL segment file. + * + * Normally, we can determine it by using the last removed segment + * number. However, if no WAL segment files have been removed by a + * checkpoint since startup, we need to search for the oldest segment + * file from the current timeline existing in XLOGDIR. + */ + oldest_segno = XLogGetLastRemovedSegno() + 1; + + if (oldest_segno == 1) + { + TimeLineID cur_timeline; + + GetWalRcvFlushRecPtr(NULL, &cur_timeline); + oldest_segno = XLogGetOldestSegno(cur_timeline); + } + + /* + * If all required WAL is still there, great, otherwise retry. The + * slot should prevent further removal of WAL, unless there's a + * concurrent ReplicationSlotsComputeRequiredLSN() after we've written + * the new restart_lsn above, so normally we should never need to loop + * more than twice. + */ + if (segno >= oldest_segno) + break; + + /* Retry using the location of the oldest wal segment */ + XLogSegNoOffsetToRecPtr(oldest_segno, 0, wal_segment_size, restart_lsn); + } +} + +/* + * If the remote restart_lsn and catalog_xmin have caught up with the + * local ones, then update the LSNs and persist the local synced slot for + * future synchronization; otherwise, do nothing. + */ +static void +update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot = MyReplicationSlot; + + /* + * Check if the primary server has caught up. Refer to the comment atop + * the file for details on this check. + */ + if (remote_slot->restart_lsn < slot->data.restart_lsn || + TransactionIdPrecedes(remote_slot->catalog_xmin, + slot->data.catalog_xmin)) + { + /* + * The remote slot didn't catch up to locally reserved position. + * + * We do not drop the slot because the restart_lsn can be ahead of the + * current location when recreating the slot in the next cycle. It may + * take more time to create such a slot. Therefore, we keep this slot + * and attempt the synchronization in the next cycle. + */ + return; + } + + /* First time slot update, the function must return true */ + if (!update_local_synced_slot(remote_slot, remote_dbid)) + elog(ERROR, "failed to update slot"); + + ReplicationSlotPersist(); + + ereport(LOG, + errmsg("newly created slot \"%s\" is sync-ready now", + remote_slot->name)); +} + +/* + * Synchronize a single slot to the given position. + * + * This creates a new slot if there is no existing one and updates the + * metadata of the slot as per the data received from the primary server. + * + * The slot is created as a temporary slot and stays in the same state until the + * the remote_slot catches up with locally reserved position and local slot is + * updated. The slot is then persisted and is considered as sync-ready for + * periodic syncs. + */ +static void +synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot; + XLogRecPtr latestFlushPtr; + + /* + * Make sure that concerned WAL is received and flushed before syncing + * slot to target lsn received from the primary server. + */ + latestFlushPtr = GetStandbyFlushRecPtr(NULL); + if (remote_slot->confirmed_lsn > latestFlushPtr) + elog(ERROR, + "skipping slot synchronization as the received slot sync" + " LSN %X/%X for slot \"%s\" is ahead of the standby position %X/%X", + LSN_FORMAT_ARGS(remote_slot->confirmed_lsn), + remote_slot->name, + LSN_FORMAT_ARGS(latestFlushPtr)); + + /* Search for the named slot */ + if ((slot = SearchNamedReplicationSlot(remote_slot->name, true))) + { + bool synced; + + SpinLockAcquire(&slot->mutex); + synced = slot->data.synced; + SpinLockRelease(&slot->mutex); + + /* User-created slot with the same name exists, raise ERROR. */ + if (!synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("exiting from slot synchronization because same" + " name slot \"%s\" already exists on the standby", + remote_slot->name)); + + /* + * The slot has been synchronized before. + * + * It is important to acquire the slot here before checking + * invalidation. If we don't acquire the slot first, there could be a + * race condition that the local slot could be invalidated just after + * checking the 'invalidated' flag here and we could end up + * overwriting 'invalidated' flag to remote_slot's value. See + * InvalidatePossiblyObsoleteSlot() where it invalidates slot directly + * if the slot is not acquired by other processes. + */ + ReplicationSlotAcquire(remote_slot->name, true); + + Assert(slot == MyReplicationSlot); + + /* + * Copy the invalidation cause from remote only if local slot is not + * invalidated locally, we don't want to overwrite existing one. + */ + if (slot->data.invalidated == RS_INVAL_NONE && + remote_slot->invalidated != RS_INVAL_NONE) + { + SpinLockAcquire(&slot->mutex); + slot->data.invalidated = remote_slot->invalidated; + SpinLockRelease(&slot->mutex); + + /* Make sure the invalidated state persists across server restart */ + ReplicationSlotMarkDirty(); + ReplicationSlotSave(); + } + + /* Skip the sync of an invalidated slot */ + if (slot->data.invalidated != RS_INVAL_NONE) + { + ReplicationSlotRelease(); + return; + } + + /* Slot not ready yet, let's attempt to make it sync-ready now. */ + if (slot->data.persistency == RS_TEMPORARY) + { + update_and_persist_local_synced_slot(remote_slot, remote_dbid); + } + + /* Slot ready for sync, so sync it. */ + else + { + /* + * Sanity check: As long as the invalidations are handled + * appropriately as above, this should never happen. + */ + if (remote_slot->restart_lsn < slot->data.restart_lsn) + elog(ERROR, + "cannot synchronize local slot \"%s\" LSN(%X/%X)" + " to remote slot's LSN(%X/%X) as synchronization" + " would move it backwards", remote_slot->name, + LSN_FORMAT_ARGS(slot->data.restart_lsn), + LSN_FORMAT_ARGS(remote_slot->restart_lsn)); + + /* Make sure the slot changes persist across server restart */ + if (update_local_synced_slot(remote_slot, remote_dbid)) + { + ReplicationSlotMarkDirty(); + ReplicationSlotSave(); + } + } + } + /* Otherwise create the slot first. */ + else + { + NameData plugin_name; + TransactionId xmin_horizon = InvalidTransactionId; + + /* Skip creating the local slot if remote_slot is invalidated already */ + if (remote_slot->invalidated != RS_INVAL_NONE) + return; + + ReplicationSlotCreate(remote_slot->name, true, RS_TEMPORARY, + remote_slot->two_phase, + remote_slot->failover, + true); + + /* For shorter lines. */ + slot = MyReplicationSlot; + + /* Avoid expensive operations while holding a spinlock. */ + namestrcpy(&plugin_name, remote_slot->plugin); + + SpinLockAcquire(&slot->mutex); + slot->data.database = remote_dbid; + slot->data.plugin = plugin_name; + SpinLockRelease(&slot->mutex); + + reserve_wal_for_local_slot(remote_slot->restart_lsn); + + LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE); + xmin_horizon = GetOldestSafeDecodingTransactionId(true); + SpinLockAcquire(&slot->mutex); + slot->effective_catalog_xmin = xmin_horizon; + slot->data.catalog_xmin = xmin_horizon; + SpinLockRelease(&slot->mutex); + ReplicationSlotsComputeRequiredXmin(true); + LWLockRelease(ProcArrayLock); + + update_and_persist_local_synced_slot(remote_slot, remote_dbid); + } + + ReplicationSlotRelease(); +} + +/* + * Synchronize slots. + * + * Gets the failover logical slots info from the primary server and updates + * the slots locally. Creates the slots if not present on the standby. + */ +static void +synchronize_slots(WalReceiverConn *wrconn) +{ +#define SLOTSYNC_COLUMN_COUNT 9 + Oid slotRow[SLOTSYNC_COLUMN_COUNT] = {TEXTOID, TEXTOID, LSNOID, + LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID}; + + WalRcvExecResult *res; + TupleTableSlot *tupslot; + StringInfoData s; + List *remote_slot_list = NIL; + + SpinLockAcquire(&SlotSyncCtx->mutex); + if (SlotSyncCtx->syncing) + { + SpinLockRelease(&SlotSyncCtx->mutex); + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot synchronize replication slots concurrently")); + } + + SlotSyncCtx->syncing = true; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = true; + + initStringInfo(&s); + + /* Construct query to fetch slots with failover enabled. */ + appendStringInfo(&s, + "SELECT slot_name, plugin, confirmed_flush_lsn," + " restart_lsn, catalog_xmin, two_phase, failover," + " database, conflict_reason" + " FROM pg_catalog.pg_replication_slots" + " WHERE failover and NOT temporary"); + + /* Execute the query */ + res = walrcv_exec(wrconn, s.data, SLOTSYNC_COLUMN_COUNT, slotRow); + pfree(s.data); + + if (res->status != WALRCV_OK_TUPLES) + ereport(ERROR, + errmsg("could not fetch failover logical slots info from the primary server: %s", + res->err)); + + /* Construct the remote_slot tuple and synchronize each slot locally */ + tupslot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple); + while (tuplestore_gettupleslot(res->tuplestore, true, false, tupslot)) + { + bool isnull; + RemoteSlot *remote_slot = palloc0(sizeof(RemoteSlot)); + Datum d; + int col = 0; + + remote_slot->name = TextDatumGetCString(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->plugin = TextDatumGetCString(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + /* + * It is possible to get null values for LSN and Xmin if slot is + * invalidated on the primary server, so handle accordingly. + */ + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->confirmed_lsn = isnull ? InvalidXLogRecPtr : + DatumGetLSN(d); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->restart_lsn = isnull ? InvalidXLogRecPtr : DatumGetLSN(d); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->catalog_xmin = isnull ? InvalidTransactionId : + DatumGetTransactionId(d); + + remote_slot->two_phase = DatumGetBool(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->failover = DatumGetBool(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->database = TextDatumGetCString(slot_getattr(tupslot, + ++col, &isnull)); + Assert(!isnull); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->invalidated = isnull ? RS_INVAL_NONE : + GetSlotInvalidationCause(TextDatumGetCString(d)); + + /* Sanity check */ + Assert(col == SLOTSYNC_COLUMN_COUNT); + + /* + * If restart_lsn, confirmed_lsn or catalog_xmin is invalid but the + * slot is valid, that means we have fetched the remote_slot in its + * RS_EPHEMERAL state. In such a case, don't sync it; we can always + * sync it in the next sync cycle when the remote_slot is persisted + * and has valid lsn(s) and xmin values. + * + * XXX: In future, if we plan to expose 'slot->data.persistency' in + * pg_replication_slots view, then we can avoid fetching RS_EPHEMERAL + * slots in the first place. + */ + if ((XLogRecPtrIsInvalid(remote_slot->restart_lsn) || + XLogRecPtrIsInvalid(remote_slot->confirmed_lsn) || + !TransactionIdIsValid(remote_slot->catalog_xmin)) && + remote_slot->invalidated == RS_INVAL_NONE) + pfree(remote_slot); + else + /* Create list of remote slots */ + remote_slot_list = lappend(remote_slot_list, remote_slot); + + ExecClearTuple(tupslot); + } + + /* Drop local slots that no longer need to be synced. */ + drop_local_obsolete_slots(remote_slot_list); + + /* Now sync the slots locally */ + foreach_ptr(RemoteSlot, remote_slot, remote_slot_list) + { + Oid remote_dbid = get_database_oid(remote_slot->database, false); + + /* + * Use shared lock to prevent a conflict with + * ReplicationSlotsDropDBSlots(), trying to drop the same slot during + * a drop-database operation. + */ + LockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); + + synchronize_one_slot(remote_slot, remote_dbid); + + UnlockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); + } + + /* We are done, free remote_slot_list elements */ + list_free_deep(remote_slot_list); + + walrcv_clear_result(res); + + SpinLockAcquire(&SlotSyncCtx->mutex); + SlotSyncCtx->syncing = false; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = false; +} + +/* + * Checks the primary server info. + * + * Using the specified primary server connection, check whether we are a + * cascading standby. It also validates primary_slot_name for non-cascading + * standbys. + */ +static void +check_primary_info(WalReceiverConn *wrconn, int elevel) +{ +#define PRIMARY_INFO_OUTPUT_COL_COUNT 2 + WalRcvExecResult *res; + Oid slotRow[PRIMARY_INFO_OUTPUT_COL_COUNT] = {BOOLOID, BOOLOID}; + StringInfoData cmd; + bool isnull; + TupleTableSlot *tupslot; + bool remote_in_recovery; + bool primary_info_valid; + + initStringInfo(&cmd); + appendStringInfo(&cmd, + "SELECT pg_is_in_recovery(), count(*) = 1" + " FROM pg_catalog.pg_replication_slots" + " WHERE slot_type='physical' AND slot_name=%s", + quote_literal_cstr(PrimarySlotName)); + + res = walrcv_exec(wrconn, cmd.data, PRIMARY_INFO_OUTPUT_COL_COUNT, slotRow); + pfree(cmd.data); + + if (res->status != WALRCV_OK_TUPLES) + ereport(ERROR, + errmsg("could not fetch primary_slot_name \"%s\" info from the primary server: %s", + PrimarySlotName, res->err), + errhint("Check if \"primary_slot_name\" is configured correctly.")); + + tupslot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple); + if (!tuplestore_gettupleslot(res->tuplestore, true, false, tupslot)) + elog(ERROR, + "failed to fetch tuple for the primary server slot specified by \"primary_slot_name\""); + + remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull)); + Assert(!isnull); + + if (remote_in_recovery) + ereport(elevel, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot synchronize replication slots from a standby server")); + + primary_info_valid = DatumGetBool(slot_getattr(tupslot, 2, &isnull)); + Assert(!isnull); + + if (!primary_info_valid) + ereport(elevel, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + /* translator: second %s is a GUC variable name */ + errdetail("The replication slot \"%s\" specified by \"%s\" does not exist on the primary server.", + PrimarySlotName, "primary_slot_name")); + + ExecClearTuple(tupslot); + walrcv_clear_result(res); +} + +/* + * Check all necessary GUCs for slot synchronization are set + * appropriately, otherwise, raise ERROR. + */ +void +ValidateSlotSyncParams(void) +{ + char *dbname; + + /* + * A physical replication slot(primary_slot_name) is required on the + * primary to ensure that the rows needed by the standby are not removed + * after restarting, so that the synchronized slot on the standby will not + * be invalidated. + */ + if (PrimarySlotName == NULL || *PrimarySlotName == '\0') + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be defined.", "primary_slot_name")); + + /* + * hot_standby_feedback must be enabled to cooperate with the physical + * replication slot, which allows informing the primary about the xmin and + * catalog_xmin values on the standby. + */ + if (!hot_standby_feedback) + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be enabled.", "hot_standby_feedback")); + + /* Logical slot sync/creation requires wal_level >= logical. */ + if (wal_level < WAL_LEVEL_LOGICAL) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"wal_level\" must be >= logical.")); + + /* + * The primary_conninfo is required to make connection to primary for + * getting slots information. + */ + if (PrimaryConnInfo == NULL || *PrimaryConnInfo == '\0') + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be defined.", "primary_conninfo")); + + /* + * The slot synchronization needs a database connection for walrcv_exec to + * work. + */ + dbname = walrcv_get_dbname_from_conninfo(PrimaryConnInfo); + if (dbname == NULL) + ereport(ERROR, + + /* + * translator: 'dbname' is a specific option; %s is a GUC variable + * name + */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("'dbname' must be specified in \"%s\".", "primary_conninfo")); +} + +/* + * Is current process syncing replication slots ? + */ +bool +IsSyncingReplicationSlots(void) +{ + return syncing_slots; +} + +/* + * Amount of shared memory required for slot synchronization. + */ +Size +SlotSyncShmemSize(void) +{ + return sizeof(SlotSyncCtxStruct); +} + +/* + * Allocate and initialize the shared memory of slot synchronization. + */ +void +SlotSyncShmemInit(void) +{ + bool found; + + SlotSyncCtx = (SlotSyncCtxStruct *) + ShmemInitStruct("Slot Sync Data", SlotSyncShmemSize(), &found); + + if (!found) + { + SlotSyncCtx->syncing = false; + SpinLockInit(&SlotSyncCtx->mutex); + } +} + +/* + * Error cleanup callback for slot synchronization. + */ +static void +slotsync_failure_callback(int code, Datum arg) +{ + WalReceiverConn *wrconn = (WalReceiverConn *) DatumGetPointer(arg); + + if (syncing_slots) + { + /* + * If syncing_slots is true, it indicates that the process errored out + * without resetting the flag. So, we need to clean up shared memory + * and reset the flag here. + */ + SpinLockAcquire(&SlotSyncCtx->mutex); + SlotSyncCtx->syncing = false; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = false; + } + + walrcv_disconnect(wrconn); +} + +/* + * Synchronize the failover enabled replication slots using the specified + * primary server connection. + */ +void +SyncReplicationSlots(WalReceiverConn *wrconn) +{ + PG_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); + { + check_primary_info(wrconn, ERROR); + + synchronize_slots(wrconn); + } + PG_END_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); + + walrcv_disconnect(wrconn); +} diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index fd4e96c9d6..7df22a0251 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -46,6 +46,7 @@ #include "common/string.h" #include "miscadmin.h" #include "pgstat.h" +#include "replication/slotsync.h" #include "replication/slot.h" #include "storage/fd.h" #include "storage/ipc.h" @@ -103,7 +104,6 @@ int max_replication_slots = 10; /* the maximum number of replication * slots */ static void ReplicationSlotShmemExit(int code, Datum arg); -static void ReplicationSlotDropAcquired(void); static void ReplicationSlotDropPtr(ReplicationSlot *slot); /* internal persistency functions */ @@ -250,11 +250,12 @@ ReplicationSlotValidateName(const char *name, int elevel) * user will only get commit prepared. * failover: If enabled, allows the slot to be synced to standbys so * that logical replication can be resumed after failover. + * synced: True if the slot is synchronized from the primary server. */ void ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotPersistency persistency, - bool two_phase, bool failover) + bool two_phase, bool failover, bool synced) { ReplicationSlot *slot = NULL; int i; @@ -263,6 +264,34 @@ ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotValidateName(name, ERROR); + if (failover) + { + /* + * Do not allow users to create failover enabled temporary slots, + * because temporary slots will not be synced to the standby. + * + * However, failover enabled temporary slots can be created during + * slot synchronization. See the comments atop slotsync.c for details. + */ + if (persistency == RS_TEMPORARY && !IsSyncingReplicationSlots()) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a temporary replication slot")); + + /* + * Do not allow users to create the failover enabled slots on the + * standby as we do not support sync to the cascading standby. + * + * However, failover enabled slots can be created during slot + * synchronization because we need to retain the same values as the + * remote slot. + */ + if (RecoveryInProgress() && !IsSyncingReplicationSlots()) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a replication slot created on the standby")); + } + /* * If some other backend ran this code concurrently with us, we'd likely * both allocate the same slot, and that would be bad. We'd also be at @@ -315,6 +344,7 @@ ReplicationSlotCreate(const char *name, bool db_specific, slot->data.two_phase = two_phase; slot->data.two_phase_at = InvalidXLogRecPtr; slot->data.failover = failover; + slot->data.synced = synced; /* and then data only present in shared memory */ slot->just_dirtied = false; @@ -677,6 +707,16 @@ ReplicationSlotDrop(const char *name, bool nowait) ReplicationSlotAcquire(name, nowait); + /* + * Do not allow users to drop the slots which are currently being synced + * from the primary to the standby. + */ + if (RecoveryInProgress() && MyReplicationSlot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot drop replication slot \"%s\"", name), + errdetail("This slot is being synced from the primary server.")); + ReplicationSlotDropAcquired(); } @@ -696,6 +736,38 @@ ReplicationSlotAlter(const char *name, bool failover) errmsg("cannot use %s with a physical replication slot", "ALTER_REPLICATION_SLOT")); + /* + * Do not allow users to enable failover for temporary slots as we do not + * support syncing temporary slots to the standby. + */ + if (failover && MyReplicationSlot->data.persistency == RS_TEMPORARY) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a temporary replication slot")); + + if (RecoveryInProgress()) + { + /* + * Do not allow users to alter the slots which are currently being + * synced from the primary to the standby. + */ + if (MyReplicationSlot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot alter replication slot \"%s\"", name), + errdetail("This slot is being synced from the primary server.")); + + /* + * Do not allow users to enable failover on the standby as we do not + * support sync to the cascading standby. + */ + if (failover) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a replication slot" + " on the standby")); + } + if (MyReplicationSlot->data.failover != failover) { SpinLockAcquire(&MyReplicationSlot->mutex); @@ -712,7 +784,7 @@ ReplicationSlotAlter(const char *name, bool failover) /* * Permanently drop the currently acquired replication slot. */ -static void +void ReplicationSlotDropAcquired(void) { ReplicationSlot *slot = MyReplicationSlot; @@ -868,8 +940,8 @@ ReplicationSlotMarkDirty(void) } /* - * Convert a slot that's marked as RS_EPHEMERAL to a RS_PERSISTENT slot, - * guaranteeing it will be there after an eventual crash. + * Convert a slot that's marked as RS_EPHEMERAL or RS_TEMPORARY to a + * RS_PERSISTENT slot, guaranteeing it will be there after an eventual crash. */ void ReplicationSlotPersist(void) @@ -2189,3 +2261,25 @@ RestoreSlotFromDisk(const char *name) (errmsg("too many replication slots active before shutdown"), errhint("Increase max_replication_slots and try again."))); } + +/* + * Maps the pg_replication_slots.conflict_reason text value to + * ReplicationSlotInvalidationCause enum value + */ +ReplicationSlotInvalidationCause +GetSlotInvalidationCause(char *conflict_reason) +{ + Assert(conflict_reason); + + if (strcmp(conflict_reason, SLOT_INVAL_WAL_REMOVED_TEXT) == 0) + return RS_INVAL_WAL_REMOVED; + else if (strcmp(conflict_reason, SLOT_INVAL_HORIZON_TEXT) == 0) + return RS_INVAL_HORIZON; + else if (strcmp(conflict_reason, SLOT_INVAL_WAL_LEVEL_TEXT) == 0) + return RS_INVAL_WAL_LEVEL; + else + Assert(0); + + /* Keep compiler quiet */ + return RS_INVAL_NONE; +} diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index eb685089b3..527e0c8ed5 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -21,7 +21,9 @@ #include "replication/decode.h" #include "replication/logical.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "utils/builtins.h" +#include "utils/guc.h" #include "utils/inval.h" #include "utils/pg_lsn.h" #include "utils/resowner.h" @@ -43,7 +45,7 @@ create_physical_replication_slot(char *name, bool immediately_reserve, /* acquire replication slot, this will check for conflicting names */ ReplicationSlotCreate(name, false, temporary ? RS_TEMPORARY : RS_PERSISTENT, false, - false); + false, false); if (immediately_reserve) { @@ -136,7 +138,7 @@ create_logical_replication_slot(char *name, char *plugin, */ ReplicationSlotCreate(name, true, temporary ? RS_TEMPORARY : RS_EPHEMERAL, two_phase, - failover); + failover, false); /* * Create logical decoding context to find start point or, if we don't @@ -237,7 +239,7 @@ pg_drop_replication_slot(PG_FUNCTION_ARGS) Datum pg_get_replication_slots(PG_FUNCTION_ARGS) { -#define PG_GET_REPLICATION_SLOTS_COLS 16 +#define PG_GET_REPLICATION_SLOTS_COLS 17 ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; XLogRecPtr currlsn; int slotno; @@ -418,21 +420,23 @@ pg_get_replication_slots(PG_FUNCTION_ARGS) break; case RS_INVAL_WAL_REMOVED: - values[i++] = CStringGetTextDatum("wal_removed"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_WAL_REMOVED_TEXT); break; case RS_INVAL_HORIZON: - values[i++] = CStringGetTextDatum("rows_removed"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_HORIZON_TEXT); break; case RS_INVAL_WAL_LEVEL: - values[i++] = CStringGetTextDatum("wal_level_insufficient"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_WAL_LEVEL_TEXT); break; } } values[i++] = BoolGetDatum(slot_contents.data.failover); + values[i++] = BoolGetDatum(slot_contents.data.synced); + Assert(i == PG_GET_REPLICATION_SLOTS_COLS); tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, @@ -700,7 +704,6 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) XLogRecPtr src_restart_lsn; bool src_islogical; bool temporary; - bool failover; char *plugin; Datum values[2]; bool nulls[2]; @@ -756,7 +759,6 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) src_islogical = SlotIsLogical(&first_slot_contents); src_restart_lsn = first_slot_contents.data.restart_lsn; temporary = (first_slot_contents.data.persistency == RS_TEMPORARY); - failover = first_slot_contents.data.failover; plugin = logical_slot ? NameStr(first_slot_contents.data.plugin) : NULL; /* Check type of replication slot */ @@ -791,12 +793,20 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) * We must not try to read WAL, since we haven't reserved it yet -- * hence pass find_startpoint false. confirmed_flush will be set * below, by copying from the source slot. + * + * To avoid potential issues with the slot synchronization where the + * restart_lsn of a replication slot can go backward, we set the + * failover option to false here. This situation occurs when a slot + * on the primary server is dropped and immediately replaced with a + * new slot of the same name, created by copying from another existing + * slot. However, the slot synchronization will only observe the + * restart_lsn of the same slot going backward. */ create_logical_replication_slot(NameStr(*dst_name), plugin, temporary, false, - failover, + false, src_restart_lsn, false); } @@ -943,3 +953,47 @@ pg_copy_physical_replication_slot_b(PG_FUNCTION_ARGS) { return copy_replication_slot(fcinfo, false); } + +/* + * Synchronize failover enabled replication slots to a standby server + * from the primary server. + */ +Datum +pg_sync_replication_slots(PG_FUNCTION_ARGS) +{ + WalReceiverConn *wrconn; + char *err; + StringInfoData app_name; + + CheckSlotPermissions(); + + if (!RecoveryInProgress()) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("replication slots can only be synchronized to a standby server")); + + /* Load the libpq-specific functions */ + load_file("libpqwalreceiver", false); + + ValidateSlotSyncParams(); + + initStringInfo(&app_name); + if (cluster_name[0]) + appendStringInfo(&app_name, "%s_slotsync", cluster_name); + else + appendStringInfoString(&app_name, "slotsync"); + + /* Connect to the primary server. */ + wrconn = walrcv_connect(PrimaryConnInfo, false, false, false, + app_name.data, &err); + pfree(app_name.data); + + if (!wrconn) + ereport(ERROR, + errcode(ERRCODE_CONNECTION_FAILURE), + errmsg("could not connect to the primary server: %s", err)); + + SyncReplicationSlots(wrconn); + + PG_RETURN_VOID(); +} diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 77c8baa32a..2d94379f1a 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -72,6 +72,7 @@ #include "postmaster/interrupt.h" #include "replication/decode.h" #include "replication/logical.h" +#include "replication/slotsync.h" #include "replication/slot.h" #include "replication/snapbuild.h" #include "replication/syncrep.h" @@ -243,7 +244,6 @@ static void WalSndShutdown(void) pg_attribute_noreturn(); static void XLogSendPhysical(void); static void XLogSendLogical(void); static void WalSndDone(WalSndSendDataCallback send_data); -static XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); static void IdentifySystem(void); static void UploadManifest(void); static bool HandleUploadManifestPacket(StringInfo buf, off_t *offset, @@ -1224,7 +1224,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) { ReplicationSlotCreate(cmd->slotname, false, cmd->temporary ? RS_TEMPORARY : RS_PERSISTENT, - false, false); + false, false, false); if (reserve_wal) { @@ -1255,7 +1255,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) */ ReplicationSlotCreate(cmd->slotname, true, cmd->temporary ? RS_TEMPORARY : RS_EPHEMERAL, - two_phase, failover); + two_phase, failover, false); /* * Do options check early so that we can bail before calling the @@ -3375,14 +3375,17 @@ WalSndDone(WalSndSendDataCallback send_data) } /* - * Returns the latest point in WAL that has been safely flushed to disk, and - * can be sent to the standby. This should only be called when in recovery, - * ie. we're streaming to a cascaded standby. + * Returns the latest point in WAL that has been safely flushed to disk. + * This should only be called when in recovery. + * + * This is called either by cascading walsender to find WAL postion to be sent + * to a cascaded standby or by slot synchronization function to validate remote + * slot's lsn before syncing it locally. * * As a side-effect, *tli is updated to the TLI of the last * replayed WAL record. */ -static XLogRecPtr +XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli) { XLogRecPtr replayPtr; @@ -3391,6 +3394,8 @@ GetStandbyFlushRecPtr(TimeLineID *tli) TimeLineID receiveTLI; XLogRecPtr result; + Assert(am_cascading_walsender || IsSyncingReplicationSlots()); + /* * We can safely send what's already been replayed. Also, if walreceiver * is streaming WAL from the same timeline, we can send anything that it diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index 7084e18861..7e7941d625 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -36,6 +36,7 @@ #include "replication/logicallauncher.h" #include "replication/origin.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/walreceiver.h" #include "replication/walsender.h" #include "storage/bufmgr.h" @@ -153,6 +154,7 @@ CalculateShmemSize(int *num_semaphores) size = add_size(size, StatsShmemSize()); size = add_size(size, WaitEventExtensionShmemSize()); size = add_size(size, InjectionPointShmemSize()); + size = add_size(size, SlotSyncShmemSize()); #ifdef EXEC_BACKEND size = add_size(size, ShmemBackendArraySize()); #endif @@ -347,6 +349,7 @@ CreateOrAttachShmemStructs(void) WalSummarizerShmemInit(); PgArchShmemInit(); ApplyLauncherShmemInit(); + SlotSyncShmemInit(); /* * Set up other modules that need some shared memory space diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 29af4ce65d..9c120fc2b7 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11127,9 +11127,9 @@ proname => 'pg_get_replication_slots', prorows => '10', proisstrict => 'f', proretset => 't', provolatile => 's', prorettype => 'record', proargtypes => '', - proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,text,bool}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,conflict_reason,failover}', + proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,text,bool,bool}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,conflict_reason,failover,synced}', prosrc => 'pg_get_replication_slots' }, { oid => '3786', descr => 'set up a logical replication slot', proname => 'pg_create_logical_replication_slot', provolatile => 'v', @@ -11212,6 +11212,10 @@ proname => 'pg_logical_emit_message', provolatile => 'v', proparallel => 'u', prorettype => 'pg_lsn', proargtypes => 'bool text bytea bool', prosrc => 'pg_logical_emit_message_bytea' }, +{ oid => '9929', descr => 'sync replication slots from the primary to the standby', + proname => 'pg_sync_replication_slots', provolatile => 'v', proparallel => 'u', + prorettype => 'void', proargtypes => '', + prosrc => 'pg_sync_replication_slots' }, # event triggers { oid => '3566', descr => 'list objects dropped by the current command', diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index da4c776492..e706ca834c 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -52,6 +52,14 @@ typedef enum ReplicationSlotInvalidationCause RS_INVAL_WAL_LEVEL, } ReplicationSlotInvalidationCause; +/* + * The possible values for 'conflict_reason' returned in + * pg_get_replication_slots. + */ +#define SLOT_INVAL_WAL_REMOVED_TEXT "wal_removed" +#define SLOT_INVAL_HORIZON_TEXT "rows_removed" +#define SLOT_INVAL_WAL_LEVEL_TEXT "wal_level_insufficient" + /* * On-Disk data of a replication slot, preserved across restarts. */ @@ -112,6 +120,11 @@ typedef struct ReplicationSlotPersistentData /* plugin name */ NameData plugin; + /* + * Was this slot synchronized from the primary server? + */ + char synced; + /* * Is this a failover slot (sync candidate for standbys)? Only relevant * for logical slots on the primary server. @@ -224,9 +237,11 @@ extern void ReplicationSlotsShmemInit(void); /* management of individual slots */ extern void ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotPersistency persistency, - bool two_phase, bool failover); + bool two_phase, bool failover, + bool synced); extern void ReplicationSlotPersist(void); extern void ReplicationSlotDrop(const char *name, bool nowait); +extern void ReplicationSlotDropAcquired(void); extern void ReplicationSlotAlter(const char *name, bool failover); extern void ReplicationSlotAcquire(const char *name, bool nowait); @@ -259,5 +274,7 @@ extern void CheckPointReplicationSlots(bool is_shutdown); extern void CheckSlotRequirements(void); extern void CheckSlotPermissions(void); +extern ReplicationSlotInvalidationCause + GetSlotInvalidationCause(char *conflict_reason); #endif /* SLOT_H */ diff --git a/src/include/replication/slotsync.h b/src/include/replication/slotsync.h new file mode 100644 index 0000000000..e86d8a47b8 --- /dev/null +++ b/src/include/replication/slotsync.h @@ -0,0 +1,23 @@ +/*------------------------------------------------------------------------- + * + * slotsync.h + * Exports for slot synchronization. + * + * Portions Copyright (c) 2016-2024, PostgreSQL Global Development Group + * + * src/include/replication/slotsync.h + * + *------------------------------------------------------------------------- + */ +#ifndef SLOTSYNC_H +#define SLOTSYNC_H + +#include "replication/walreceiver.h" + +extern void ValidateSlotSyncParams(void); +extern bool IsSyncingReplicationSlots(void); +extern Size SlotSyncShmemSize(void); +extern void SlotSyncShmemInit(void); +extern void SyncReplicationSlots(WalReceiverConn *wrconn); + +#endif /* SLOTSYNC_H */ diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h index 1b58d50b3b..0c3996e926 100644 --- a/src/include/replication/walsender.h +++ b/src/include/replication/walsender.h @@ -12,6 +12,8 @@ #ifndef _WALSENDER_H #define _WALSENDER_H +#include "access/xlogdefs.h" + /* * What to do with a snapshot in create replication slot command. */ @@ -37,6 +39,7 @@ extern void InitWalSender(void); extern bool exec_replication_command(const char *cmd_string); extern void WalSndErrorCleanup(void); extern void WalSndResourceCleanup(bool isCommit); +extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); extern void WalSndSignals(void); extern Size WalSndShmemSize(void); extern void WalSndShmemInit(void); diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index bc58ff4cab..5ee2996c31 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -97,4 +97,128 @@ my ($result, $stdout, $stderr) = $subscriber1->psql('postgres', ok( $stderr =~ /ERROR: cannot set failover for enabled subscription/, "altering failover is not allowed for enabled subscription"); +################################################## +# Test that pg_sync_replication_slots() cannot be executed on the primary +# server. +################################################## + +($result, $stdout, $stderr) = $subscriber1->psql('postgres', + "SELECT pg_sync_replication_slots();"); +ok( $stderr =~ /ERROR: replication slots can only be synchronized to a standby server/, + "cannot sync slots on the primary server"); + +################################################## +# Test logical failover slots on the standby +# Configure standby1 to replicate and synchronize logical slots configured +# for failover on the primary +# +# failover slot lsub1_slot ->| ----> subscriber1 (connected via logical replication) +# failover slot lsub2_slot | inactive +# primary ---> | +# physical slot sb1_slot --->| ----> standby1 (connected via streaming replication) +# | lsub1_slot, lsub2_slot (synced_slot) +################################################## + +my $primary = $publisher; +my $backup_name = 'backup'; +$primary->backup($backup_name); + +# Create a standby +my $standby1 = PostgreSQL::Test::Cluster->new('standby1'); +$standby1->init_from_backup( + $primary, $backup_name, + has_streaming => 1, + has_restoring => 1); + +my $connstr_1 = $primary->connstr; +$standby1->append_conf( + 'postgresql.conf', qq( +hot_standby_feedback = on +primary_slot_name = 'sb1_slot' +primary_conninfo = '$connstr_1 dbname=postgres' +)); + +$primary->psql('postgres', + q{SELECT pg_create_logical_replication_slot('lsub2_slot', 'test_decoding', false, false, true);} +); + +$primary->psql('postgres', + q{SELECT pg_create_physical_replication_slot('sb1_slot');}); + +my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; +my $offset = -s $standby1->logfile; + +# Start the standby so that slot syncing can begin +$standby1->start; + +$primary->wait_for_catchup('regress_mysub1'); + +# Do not allow any further advancement of the restart_lsn for the lsub1_slot. +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); + +# Wait for the replication slot to become inactive on the publisher +$primary->poll_query_until( + 'postgres', + "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", + 1); + +# Wait for the standby to catch up so that the standby is not lagging behind +# the subscriber. +$primary->wait_for_replay_catchup($standby1); + +# Synchronize the primary server slots to the standby. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Confirm that the logical failover slots are created on the standby and are +# flagged as 'synced' +is( $standby1->safe_psql( + 'postgres', + q{SELECT count(*) = 2 FROM pg_replication_slots WHERE slot_name IN ('lsub1_slot', 'lsub2_slot') AND synced;} + ), + "t", + 'logical slots have synced as true on standby'); + +################################################## +# Test that the synchronized slot will be dropped if the corresponding remote +# slot on the primary server has been dropped. +################################################## + +$primary->psql('postgres', "SELECT pg_drop_replication_slot('lsub2_slot');"); + +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +is( $standby1->safe_psql( + 'postgres', + q{SELECT count(*) = 0 FROM pg_replication_slots WHERE slot_name = 'lsub2_slot';} + ), + "t", + 'synchronized slot has been dropped'); + +################################################## +# Test that a synchronized slot can not be decoded, altered or dropped by the +# user +################################################## + +# Attempting to perform logical decoding on a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql('postgres', + "select * from pg_logical_slot_get_changes('lsub1_slot', NULL, NULL);"); +ok( $stderr =~ + /ERROR: cannot use replication slot "lsub1_slot" for logical decoding/, + "logical decoding is not allowed on synced slot"); + +# Attempting to alter a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql( + 'postgres', + qq[ALTER_REPLICATION_SLOT lsub1_slot (failover);], + replication => 'database'); +ok($stderr =~ /ERROR: cannot alter replication slot "lsub1_slot"/, + "synced slot on standby cannot be altered"); + +# Attempting to drop a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql('postgres', + "SELECT pg_drop_replication_slot('lsub1_slot');"); +ok($stderr =~ /ERROR: cannot drop replication slot "lsub1_slot"/, + "synced slot on standby cannot be dropped"); + done_testing(); diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index abc944e8b8..b7488d760e 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1474,8 +1474,9 @@ pg_replication_slots| SELECT l.slot_name, l.safe_wal_size, l.two_phase, l.conflict_reason, - l.failover - FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, conflict_reason, failover) + l.failover, + l.synced + FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, conflict_reason, failover, synced) LEFT JOIN pg_database d ON ((l.datoid = d.oid))); pg_roles| SELECT pg_authid.rolname, pg_authid.rolsuper, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 91433d439b..d808aad8b0 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2325,6 +2325,7 @@ RelocationBufferInfo RelptrFreePageBtree RelptrFreePageManager RelptrFreePageSpanLeader +RemoteSlot RenameStmt ReopenPtrType ReorderBuffer @@ -2584,6 +2585,7 @@ SlabBlock SlabContext SlabSlot SlotNumber +SlotSyncCtxStruct SlruCtl SlruCtlData SlruErrorCause -- 2.43.0 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-13 03:38 shveta malik <[email protected]> parent: Zhijie Hou (Fujitsu) <[email protected]> 1 sibling, 0 replies; 46+ messages in thread From: shveta malik @ 2024-02-13 03:38 UTC (permalink / raw) To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>; shveta malik <[email protected]> On Tue, Feb 13, 2024 at 6:45 AM Zhijie Hou (Fujitsu) <[email protected]> wrote: > > On Monday, February 12, 2024 5:40 PM Amit Kapila <[email protected]> wrote: > > Thanks for the comments, I have addressed them. > > Here is the new version patch which addressed above and > most of Bertrand's comments. Thanks for the patch. I am trying to run valgrind on patch001. I followed the steps given in [1]. It ended up generating 850 log files. Is there a way to figure out that we have a memory related problem w/o going through each log file manually? I also tried running the steps with '-leak-check=summary' (in the first run, it was '-leak-check=no' as suggested in wiki) with and without the patch and tried comparing those manually for a few of them. I found o/p more or less the same. But this is a mammoth task if we have to do it manually for 850 files. So any pointers here? For reference: Sample log file with '-leak-check=no' ==00:00:08:44.321 250746== HEAP SUMMARY: ==00:00:08:44.321 250746== in use at exit: 1,298,274 bytes in 290 blocks ==00:00:08:44.321 250746== total heap usage: 11,958 allocs, 7,005 frees, 8,175,630 bytes allocated ==00:00:08:44.321 250746== ==00:00:08:44.321 250746== For a detailed leak analysis, rerun with: --leak-check=full ==00:00:08:44.321 250746== ==00:00:08:44.321 250746== For lists of detected and suppressed errors, rerun with: -s ==00:00:08:44.321 250746== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) Sample log file with '-leak-check=summary' ==00:00:00:27.300 265785== HEAP SUMMARY: ==00:00:00:27.300 265785== in use at exit: 1,929,907 bytes in 310 blocks ==00:00:00:27.300 265785== total heap usage: 71,677 allocs, 7,754 frees, 95,750,897 bytes allocated ==00:00:00:27.300 265785== ==00:00:00:27.394 265785== LEAK SUMMARY: ==00:00:00:27.394 265785== definitely lost: 20,507 bytes in 171 blocks ==00:00:00:27.394 265785== indirectly lost: 16,419 bytes in 61 blocks ==00:00:00:27.394 265785== possibly lost: 354,670 bytes in 905 blocks ==00:00:00:27.394 265785== still reachable: 592,586 bytes in 1,473 blocks ==00:00:00:27.394 265785== suppressed: 0 bytes in 0 blocks ==00:00:00:27.394 265785== Rerun with --leak-check=full to see details of leaked memory ==00:00:00:27.394 265785== ==00:00:00:27.394 265785== For lists of detected and suppressed errors, rerun with: -s ==00:00:00:27.394 265785== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) [1]: https://wiki.postgresql.org/wiki/Valgrind thanks Shveta ^ permalink raw reply [nested|flat] 46+ messages in thread
* RE: Synchronizing slots from primary to standby @ 2024-02-13 04:08 Zhijie Hou (Fujitsu) <[email protected]> parent: Zhijie Hou (Fujitsu) <[email protected]> 1 sibling, 2 replies; 46+ messages in thread From: Zhijie Hou (Fujitsu) @ 2024-02-13 04:08 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Tuesday, February 13, 2024 9:16 AM Zhijie Hou (Fujitsu) <[email protected]> wrote: > > Here is the new version patch which addressed above and most of Bertrand's > comments. > > TODO: trying to add one test for the case the slot is valid on primary while the > synced slots is invalidated on the standby. Here is the V85_2 patch set that added the test and fixed one typo, there are no other code changes. Best Regards, Hou zj Attachments: [application/octet-stream] v85_2-0001-Add-a-slot-synchronization-function.patch (75.5K, ../../OS0PR01MB57160CFA75A6546E5AFD1708944F2@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v85_2-0001-Add-a-slot-synchronization-function.patch) download | inline diff: From 49c2a3ba3eca23b9ee6d6bc6d4490d156d79c96b Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Wed, 7 Feb 2024 10:37:31 +0530 Subject: [PATCH v85] Add a slot synchronization function. This commit introduces a new SQL function pg_sync_replication_slots() which is used to synchronize the logical replication slots from the primary server to the physical standby so that logical replication can be resumed after failover. A new 'synced' flag is introduced in pg_replication_slots view, indicating whether the slot has been synchronized from the primary server. On a standby, synced slots cannot be dropped or consumed, and any attempt to perform logical decoding on them will result in an error. The logical replication slots on the primary can be synchronized to the hot standby by enabling failover during slot creation (e.g. using the "failover" parameter of pg_create_logical_replication_slot(), or using the "failover" option of the CREATE SUBSCRIPTION command), and then calling pg_sync_replication_slots() function on the standby. For the synchronization to work, it is mandatory to have a physical replication slot between the primary and the standby, 'hot_standby_feedback' must be enabled on the standby and a valid dbname must be specified in 'primary_conninfo'. If a logical slot is invalidated on the primary, then that slot on the standby is also invalidated. If a logical slot on the primary is valid but is invalidated on the standby, then that slot is dropped but will be recreated on the standby in next pg_sync_replication_slots() call provided the slot still exists on the primary server. It is okay to recreate such slots as long as these are not consumable on the standby (which is the case currently). This situation may occur due to the following reasons: - The 'max_slot_wal_keep_size' on the standby is insufficient to retain WAL records from the restart_lsn of the slot. - 'primary_slot_name' is temporarily reset to null and the physical slot is removed. We may also see the slots invalidated and dropped on the standby if the primary changes 'wal_level' to a level lower than logical. Changing the primary 'wal_level' to a level lower than logical is only possible if the logical slots are removed on the primary server, so it's expected to see the slots being removed on the standby too (and re-created if they are re-created on the primary server). The slots synchronization status on the standby can be monitored using 'synced' column of pg_replication_slots view. --- .../test_decoding/expected/permissions.out | 3 + contrib/test_decoding/expected/slot.out | 2 + contrib/test_decoding/sql/permissions.sql | 1 + contrib/test_decoding/sql/slot.sql | 1 + doc/src/sgml/config.sgml | 9 +- doc/src/sgml/func.sgml | 34 +- doc/src/sgml/logicaldecoding.sgml | 56 ++ doc/src/sgml/protocol.sgml | 6 +- doc/src/sgml/system-views.sgml | 20 +- src/backend/catalog/system_views.sql | 3 +- src/backend/replication/logical/Makefile | 1 + src/backend/replication/logical/logical.c | 12 + src/backend/replication/logical/meson.build | 1 + src/backend/replication/logical/slotsync.c | 896 ++++++++++++++++++ src/backend/replication/slot.c | 104 +- src/backend/replication/slotfuncs.c | 72 +- src/backend/replication/walsender.c | 19 +- src/backend/storage/ipc/ipci.c | 3 + src/include/catalog/pg_proc.dat | 10 +- src/include/replication/slot.h | 19 +- src/include/replication/slotsync.h | 23 + src/include/replication/walsender.h | 3 + .../t/040_standby_failover_slots_sync.pl | 193 ++++ src/test/regress/expected/rules.out | 5 +- src/tools/pgindent/typedefs.list | 2 + 25 files changed, 1463 insertions(+), 35 deletions(-) create mode 100644 src/backend/replication/logical/slotsync.c create mode 100644 src/include/replication/slotsync.h diff --git a/contrib/test_decoding/expected/permissions.out b/contrib/test_decoding/expected/permissions.out index d6eaba8c55..8d100646ce 100644 --- a/contrib/test_decoding/expected/permissions.out +++ b/contrib/test_decoding/expected/permissions.out @@ -64,6 +64,9 @@ DETAIL: Only roles with the REPLICATION attribute may use replication slots. SELECT pg_drop_replication_slot('regression_slot'); ERROR: permission denied to use replication slots DETAIL: Only roles with the REPLICATION attribute may use replication slots. +SELECT pg_sync_replication_slots(); +ERROR: permission denied to use replication slots +DETAIL: Only roles with the REPLICATION attribute may use replication slots. RESET ROLE; -- replication users can drop superuser created slots SET ROLE regress_lr_superuser; diff --git a/contrib/test_decoding/expected/slot.out b/contrib/test_decoding/expected/slot.out index 261d8886d3..349ab2d380 100644 --- a/contrib/test_decoding/expected/slot.out +++ b/contrib/test_decoding/expected/slot.out @@ -425,6 +425,8 @@ SELECT 'init' FROM pg_create_logical_replication_slot('failover_default_slot', ' init (1 row) +SELECT 'init' FROM pg_create_logical_replication_slot('failover_true_temp_slot', 'test_decoding', true, false, true); +ERROR: cannot enable failover for a temporary replication slot SELECT 'init' FROM pg_create_physical_replication_slot('physical_slot'); ?column? ---------- diff --git a/contrib/test_decoding/sql/permissions.sql b/contrib/test_decoding/sql/permissions.sql index 312b514593..94db936aee 100644 --- a/contrib/test_decoding/sql/permissions.sql +++ b/contrib/test_decoding/sql/permissions.sql @@ -29,6 +29,7 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_d INSERT INTO lr_test VALUES('lr_superuser_init'); SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); SELECT pg_drop_replication_slot('regression_slot'); +SELECT pg_sync_replication_slots(); RESET ROLE; -- replication users can drop superuser created slots diff --git a/contrib/test_decoding/sql/slot.sql b/contrib/test_decoding/sql/slot.sql index 45aeae7fd5..580e3ae3be 100644 --- a/contrib/test_decoding/sql/slot.sql +++ b/contrib/test_decoding/sql/slot.sql @@ -181,6 +181,7 @@ SELECT pg_drop_replication_slot('copied_slot2_notemp'); SELECT 'init' FROM pg_create_logical_replication_slot('failover_true_slot', 'test_decoding', false, false, true); SELECT 'init' FROM pg_create_logical_replication_slot('failover_false_slot', 'test_decoding', false, false, false); SELECT 'init' FROM pg_create_logical_replication_slot('failover_default_slot', 'test_decoding', false, false); +SELECT 'init' FROM pg_create_logical_replication_slot('failover_true_temp_slot', 'test_decoding', true, false, true); SELECT 'init' FROM pg_create_physical_replication_slot('physical_slot'); SELECT slot_name, slot_type, failover FROM pg_replication_slots; diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 61038472c5..037a3b8a64 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4612,8 +4612,13 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" <varname>primary_conninfo</varname> string, or in a separate <filename>~/.pgpass</filename> file on the standby server (use <literal>replication</literal> as the database name). - Do not specify a database name in the - <varname>primary_conninfo</varname> string. + </para> + <para> + For replication slot synchronization (see + <xref linkend="logicaldecoding-replication-slots-synchronization"/>), + it is also necessary to specify a valid <literal>dbname</literal> + in the <varname>primary_conninfo</varname> string. This will only be + used for slot synchronization. It is ignored for streaming. </para> <para> This parameter can only be set in the <filename>postgresql.conf</filename> diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 11d537b341..1b1041195c 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -28075,7 +28075,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset </row> <row> - <entry role="func_table_entry"><para role="func_signature"> + <entry id="pg-create-logical-replication-slot" role="func_table_entry"><para role="func_signature"> <indexterm> <primary>pg_create_logical_replication_slot</primary> </indexterm> @@ -28444,6 +28444,38 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset record is flushed along with its transaction. </para></entry> </row> + + <row> + <entry id="pg-sync-replication-slots" role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>pg_sync_replication_slots</primary> + </indexterm> + <function>pg_sync_replication_slots</function> () + <returnvalue>void</returnvalue> + </para> + <para> + Synchronize the logical failover replication slots from the primary + server to the standby server. This function can only be executed on the + standby server. See + <xref linkend="logicaldecoding-replication-slots-synchronization"/> for details. + </para> + + <caution> + <para> + If, after executing the function, + <link linkend="guc-hot-standby-feedback"> + <varname>hot_standby_feedback</varname></link> is disabled on + the standby or the physical slot configured in + <link linkend="guc-primary-slot-name"> + <varname>primary_slot_name</varname></link> is + removed, then it is possible that the necessary rows of the + synchronized slot will be removed by the VACUUM process on the primary + server, resulting in the synchronized slot becoming invalidated. + </para> + </caution> + </entry> + </row> + </tbody> </tgroup> </table> diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index cd152d4ced..dae2d5865e 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -358,6 +358,62 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU So if a slot is no longer required it should be dropped. </para> </caution> + + </sect2> + + <sect2 id="logicaldecoding-replication-slots-synchronization"> + <title>Replication Slot Synchronization</title> + <para> + The logical replication slots on the primary can be synchronized to + the hot standby by enabling <literal>failover</literal> during slot + creation (e.g., using the <literal>failover</literal> parameter of + <link linkend="pg-create-logical-replication-slot"> + <function>pg_create_logical_replication_slot</function></link>, or + using the <link linkend="sql-createsubscription-params-with-failover"> + <literal>failover</literal></link> option of + <command>CREATE SUBSCRIPTION</command>), and then calling + <link linkend="pg-sync-replication-slots"> + <function>pg_sync_replication_slots</function></link> + on the standby. For the synchronization to work, it is mandatory to + have a physical replication slot between the primary and the standby (e.g., + <link linkend="guc-primary-slot-name"><varname>primary_slot_name</varname></link> + should be configured on the standby), and + <link linkend="guc-hot-standby-feedback"><varname>hot_standby_feedback</varname></link> + must be enabled on the standby. It is also necessary to specify a valid + <literal>dbname</literal> in the + <link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>. + </para> + + <para> + The ability to resume logical replication after failover depends upon the + <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield> + value for the synchronized slots on the standby at the time of failover. + Only persistent slots that have attained synced state as true on the standby + before failover can be used for logical replication after failover. + Temporary slots will be dropped, therefore logical replication for those + slots cannot be resumed. For example, if the synchronized slot could not + become persistent on the standby due to a disabled subscription, then the + subscription cannot be resumed after failover even when it is enabled. + </para> + + <para> + To resume logical replication after failover from the synced logical + slots, the subscription's 'conninfo' must be altered to point to the + new primary server. This is done using + <link linkend="sql-altersubscription-params-connection"><command>ALTER SUBSCRIPTION ... CONNECTION</command></link>. + It is recommended that subscriptions are first disabled before promoting + the standby and are re-enabled after altering the connection string. + </para> + <caution> + <para> + There is a chance that the old primary is up again during the promotion + and if subscriptions are not disabled, the logical subscribers may + continue to receive data from the old primary server even after promotion + until the connection string is altered. This might result in data + inconsistency issues, preventing the logical subscribers from being + able to continue replication from the new primary server. + </para> + </caution> </sect2> <sect2 id="logicaldecoding-explanation-output-plugins"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index ed1d62f5f8..7ffddfbd82 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -2062,7 +2062,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" <term><literal>FAILOVER [ <replaceable class="parameter">boolean</replaceable> ]</literal></term> <listitem> <para> - If true, the slot is enabled to be synced to the standbys. + If true, the slot is enabled to be synced to the standbys + so that logical replication can be resumed after failover. The default is false. </para> </listitem> @@ -2162,7 +2163,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" <term><literal>FAILOVER [ <replaceable class="parameter">boolean</replaceable> ]</literal></term> <listitem> <para> - If true, the slot is enabled to be synced to the standbys. + If true, the slot is enabled to be synced to the standbys + so that logical replication can be resumed after failover. </para> </listitem> </varlistentry> diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index dd468b31ea..be90edd0e2 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2561,10 +2561,26 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <structfield>failover</structfield> <type>bool</type> </para> <para> - True if this is a logical slot enabled to be synced to the standbys. - Always false for physical slots. + True if this is a logical slot enabled to be synced to the standbys + so that logical replication can be resumed from the new primary + after failover. Always false for physical slots. </para></entry> </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>synced</structfield> <type>bool</type> + </para> + <para> + True if this is a logical slot that was synced from a primary server. + On a hot standby, the slots with the synced column marked as true can + neither be used for logical decoding nor dropped manually. The value + of this column has no meaning on the primary server; the column value on + the primary is default false for all slots but may (if leftover from a + promoted standby) also be true. + </para></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 6791bff9dd..04227a72d1 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1024,7 +1024,8 @@ CREATE VIEW pg_replication_slots AS L.safe_wal_size, L.two_phase, L.conflict_reason, - L.failover + L.failover, + L.synced FROM pg_get_replication_slots() AS L LEFT JOIN pg_database D ON (L.datoid = D.oid); diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index 2dc25e37bb..ba03eeff1c 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -25,6 +25,7 @@ OBJS = \ proto.o \ relation.o \ reorderbuffer.o \ + slotsync.o \ snapbuild.o \ tablesync.o \ worker.o diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index ca09c683f1..a53815f2ed 100644 --- a/src/backend/replication/logical/logical.c +++ b/src/backend/replication/logical/logical.c @@ -524,6 +524,18 @@ CreateDecodingContext(XLogRecPtr start_lsn, errmsg("replication slot \"%s\" was not created in this database", NameStr(slot->data.name)))); + /* + * Do not allow consumption of a "synchronized" slot until the standby + * gets promoted. + */ + if (RecoveryInProgress() && slot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot use replication slot \"%s\" for logical decoding", + NameStr(slot->data.name)), + errdetail("This slot is being synchronized from the primary server."), + errhint("Specify another replication slot.")); + /* * Check if slot has been invalidated due to max_slot_wal_keep_size. Avoid * "cannot get changes" wording in this errmsg because that'd be diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build index 1050eb2c09..3dec36a6de 100644 --- a/src/backend/replication/logical/meson.build +++ b/src/backend/replication/logical/meson.build @@ -11,6 +11,7 @@ backend_sources += files( 'proto.c', 'relation.c', 'reorderbuffer.c', + 'slotsync.c', 'snapbuild.c', 'tablesync.c', 'worker.c', diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c new file mode 100644 index 0000000000..feb04e1451 --- /dev/null +++ b/src/backend/replication/logical/slotsync.c @@ -0,0 +1,896 @@ +/*------------------------------------------------------------------------- + * slotsync.c + * Functionality for synchronizing slots to a standby server from the + * primary server. + * + * Copyright (c) 2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/slotsync.c + * + * This file contains the code for slot synchronization on a physical standby + * to fetch logical failover slots information from the primary server, create + * the slots on the standby and synchronize them. This is done by a call to SQL + * function pg_sync_replication_slots. + * + * If on physical standby, the WAL corresponding to the remote's restart_lsn + * is not available or the remote's catalog_xmin precedes the oldest xid for which + * it is guaranteed that rows wouldn't have been removed then we cannot create + * the local standby slot because that would mean moving the local slot + * backward and decoding won't be possible via such a slot. In this case, the + * slot will be marked as RS_TEMPORARY. Once the primary server catches up, + * the slot will be marked as RS_PERSISTENT (which means sync-ready) after + * which we can call pg_sync_replication_slots() periodically to perform + * syncs. + * + * Any standby synchronized slots will be dropped if they no longer need + * to be synchronized. See comment atop drop_local_obsolete_slots() for more + * details. + *--------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/xlog_internal.h" +#include "access/xlogrecovery.h" +#include "catalog/pg_database.h" +#include "commands/dbcommands.h" +#include "replication/logical.h" +#include "replication/slotsync.h" +#include "storage/ipc.h" +#include "storage/lmgr.h" +#include "storage/procarray.h" +#include "utils/builtins.h" +#include "utils/pg_lsn.h" + +/* Struct for sharing information to control slot synchronization. */ +typedef struct SlotSyncCtxStruct +{ + /* prevents concurrent slot syncs to avoid slot overwrites */ + bool syncing; + slock_t mutex; +} SlotSyncCtxStruct; + +SlotSyncCtxStruct *SlotSyncCtx = NULL; + +/* + * Flag to tell if we are syncing replication slots. Unlike the 'syncing' flag + * in SlotSyncCtxStruct, this flag is true only if the current process is + * performing slot synchronization. + */ +static bool syncing_slots = false; + +/* + * Structure to hold information fetched from the primary server about a logical + * replication slot. + */ +typedef struct RemoteSlot +{ + char *name; + char *plugin; + char *database; + bool two_phase; + bool failover; + XLogRecPtr restart_lsn; + XLogRecPtr confirmed_lsn; + TransactionId catalog_xmin; + + /* RS_INVAL_NONE if valid, or the reason of invalidation */ + ReplicationSlotInvalidationCause invalidated; +} RemoteSlot; + +/* + * If necessary, update the local synced slot's metadata based on the data + * from the remote slot. + * + * If no update was needed (the data of the remote slot is the same as the + * local slot) return false, otherwise true. + */ +static bool +update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot = MyReplicationSlot; + bool xmin_changed; + bool restart_lsn_changed; + NameData plugin_name; + + Assert(slot->data.invalidated == RS_INVAL_NONE); + + xmin_changed = (remote_slot->catalog_xmin != slot->data.catalog_xmin); + restart_lsn_changed = (remote_slot->restart_lsn != slot->data.restart_lsn); + + if (!xmin_changed && + !restart_lsn_changed && + remote_dbid == slot->data.database && + remote_slot->two_phase == slot->data.two_phase && + remote_slot->failover == slot->data.failover && + remote_slot->confirmed_lsn == slot->data.confirmed_flush && + strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) + return false; + + /* Avoid expensive operations while holding a spinlock. */ + namestrcpy(&plugin_name, remote_slot->plugin); + + SpinLockAcquire(&slot->mutex); + slot->data.plugin = plugin_name; + slot->data.database = remote_dbid; + slot->data.two_phase = remote_slot->two_phase; + slot->data.failover = remote_slot->failover; + slot->data.restart_lsn = remote_slot->restart_lsn; + slot->data.confirmed_flush = remote_slot->confirmed_lsn; + slot->data.catalog_xmin = remote_slot->catalog_xmin; + slot->effective_catalog_xmin = remote_slot->catalog_xmin; + SpinLockRelease(&slot->mutex); + + if (xmin_changed) + ReplicationSlotsComputeRequiredXmin(false); + + if (restart_lsn_changed) + ReplicationSlotsComputeRequiredLSN(); + + return true; +} + +/* + * Get the list of local logical slots that are synchronized from the + * primary server. + */ +static List * +get_local_synced_slots(void) +{ + List *local_slots = NIL; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + + for (int i = 0; i < max_replication_slots; i++) + { + ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i]; + + /* Check if it is a synchronized slot */ + if (s->in_use && s->data.synced) + { + Assert(SlotIsLogical(s)); + local_slots = lappend(local_slots, s); + } + } + + LWLockRelease(ReplicationSlotControlLock); + + return local_slots; +} + +/* + * Helper function to check if local_slot is required to be retained. + * + * Return false either if local_slot does not exist in the remote_slots list + * or is invalidated while the corresponding remote slot is still valid, + * otherwise true. + */ +static bool +local_sync_slot_required(ReplicationSlot *local_slot, List *remote_slots) +{ + bool remote_exists = false; + bool locally_invalidated = false; + + foreach_ptr(RemoteSlot, remote_slot, remote_slots) + { + if (strcmp(remote_slot->name, NameStr(local_slot->data.name)) == 0) + { + remote_exists = true; + + /* + * If remote slot is not invalidated but local slot is marked as + * invalidated, then set locally_invalidated flag. + */ + SpinLockAcquire(&local_slot->mutex); + locally_invalidated = + (remote_slot->invalidated == RS_INVAL_NONE) && + (local_slot->data.invalidated != RS_INVAL_NONE); + SpinLockRelease(&local_slot->mutex); + + break; + } + } + + return (remote_exists && !locally_invalidated); +} + +/* + * Drop local obsolete slots. + * + * Drop the local slots that no longer need to be synced i.e. these either do + * not exist on the primary or are no longer enabled for failover. + * + * Additionally, drop any slots that are valid on the primary but got + * invalidated on the standby. This situation may occur due to the following + * reasons: + * - The 'max_slot_wal_keep_size' on the standby is insufficient to retain WAL + * records from the restart_lsn of the slot. + * - 'primary_slot_name' is temporarily reset to null and the physical slot is + * removed. + * These dropped slots will get recreated in next sync-cycle and it is okay to + * drop and recreate such slots as long as these are not consumable on the + * standby (which is the case currently). + * + * Note: Change of 'wal_level' on the primary server to a level lower than + * logical may also result in slot invalidation and removal on the standby. + * This is because such 'wal_level' change is only possible if the logical + * slots are removed on the primary server, so it's expected to see the + * slots being invalidated and removed on the standby too (and re-created + * if they are re-created on the primary server). + */ +static void +drop_local_obsolete_slots(List *remote_slot_list) +{ + List *local_slots = get_local_synced_slots(); + + foreach_ptr(ReplicationSlot, local_slot, local_slots) + { + /* Drop the local slot if it is not required to be retained. */ + if (!local_sync_slot_required(local_slot, remote_slot_list)) + { + bool synced_slot; + + /* + * Use shared lock to prevent a conflict with + * ReplicationSlotsDropDBSlots(), trying to drop the same slot + * during a drop-database operation. + */ + LockSharedObject(DatabaseRelationId, local_slot->data.database, + 0, AccessShareLock); + + /* + * In the small window between getting the slot to drop and + * locking the database, there is a possibility of a parallel + * database drop by the startup process and the creation of a new + * slot by the user. This new user-created slot may end up using + * the same shared memory as that of 'local_slot'. Thus check if + * local_slot is still the synced one before performing actual + * drop. + */ + SpinLockAcquire(&local_slot->mutex); + synced_slot = local_slot->in_use && local_slot->data.synced; + SpinLockRelease(&local_slot->mutex); + + if (synced_slot) + { + ReplicationSlotAcquire(NameStr(local_slot->data.name), true); + ReplicationSlotDropAcquired(); + } + + UnlockSharedObject(DatabaseRelationId, local_slot->data.database, + 0, AccessShareLock); + + ereport(LOG, + errmsg("dropped replication slot \"%s\" of dbid %d", + NameStr(local_slot->data.name), + local_slot->data.database)); + } + } +} + +/* + * Reserve WAL for the currently active local slot using the specified WAL + * location (restart_lsn). + * + * If the given WAL location has been removed, reserve WAL using the oldest + * existing WAL segment. + */ +static void +reserve_wal_for_local_slot(XLogRecPtr restart_lsn) +{ + XLogSegNo oldest_segno; + XLogSegNo segno; + ReplicationSlot *slot = MyReplicationSlot; + + Assert(slot != NULL); + Assert(XLogRecPtrIsInvalid(slot->data.restart_lsn)); + + while (true) + { + SpinLockAcquire(&slot->mutex); + slot->data.restart_lsn = restart_lsn; + SpinLockRelease(&slot->mutex); + + /* Prevent WAL removal as fast as possible */ + ReplicationSlotsComputeRequiredLSN(); + + XLByteToSeg(slot->data.restart_lsn, segno, wal_segment_size); + + /* + * Find the oldest existing WAL segment file. + * + * Normally, we can determine it by using the last removed segment + * number. However, if no WAL segment files have been removed by a + * checkpoint since startup, we need to search for the oldest segment + * file from the current timeline existing in XLOGDIR. + */ + oldest_segno = XLogGetLastRemovedSegno() + 1; + + if (oldest_segno == 1) + { + TimeLineID cur_timeline; + + GetWalRcvFlushRecPtr(NULL, &cur_timeline); + oldest_segno = XLogGetOldestSegno(cur_timeline); + } + + /* + * If all required WAL is still there, great, otherwise retry. The + * slot should prevent further removal of WAL, unless there's a + * concurrent ReplicationSlotsComputeRequiredLSN() after we've written + * the new restart_lsn above, so normally we should never need to loop + * more than twice. + */ + if (segno >= oldest_segno) + break; + + /* Retry using the location of the oldest wal segment */ + XLogSegNoOffsetToRecPtr(oldest_segno, 0, wal_segment_size, restart_lsn); + } +} + +/* + * If the remote restart_lsn and catalog_xmin have caught up with the + * local ones, then update the LSNs and persist the local synced slot for + * future synchronization; otherwise, do nothing. + */ +static void +update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot = MyReplicationSlot; + + /* + * Check if the primary server has caught up. Refer to the comment atop + * the file for details on this check. + */ + if (remote_slot->restart_lsn < slot->data.restart_lsn || + TransactionIdPrecedes(remote_slot->catalog_xmin, + slot->data.catalog_xmin)) + { + /* + * The remote slot didn't catch up to locally reserved position. + * + * We do not drop the slot because the restart_lsn can be ahead of the + * current location when recreating the slot in the next cycle. It may + * take more time to create such a slot. Therefore, we keep this slot + * and attempt the synchronization in the next cycle. + */ + return; + } + + /* First time slot update, the function must return true */ + if (!update_local_synced_slot(remote_slot, remote_dbid)) + elog(ERROR, "failed to update slot"); + + ReplicationSlotPersist(); + + ereport(LOG, + errmsg("newly created slot \"%s\" is sync-ready now", + remote_slot->name)); +} + +/* + * Synchronize a single slot to the given position. + * + * This creates a new slot if there is no existing one and updates the + * metadata of the slot as per the data received from the primary server. + * + * The slot is created as a temporary slot and stays in the same state until the + * the remote_slot catches up with locally reserved position and local slot is + * updated. The slot is then persisted and is considered as sync-ready for + * periodic syncs. + */ +static void +synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot; + XLogRecPtr latestFlushPtr; + + /* + * Make sure that concerned WAL is received and flushed before syncing + * slot to target lsn received from the primary server. + */ + latestFlushPtr = GetStandbyFlushRecPtr(NULL); + if (remote_slot->confirmed_lsn > latestFlushPtr) + elog(ERROR, + "skipping slot synchronization as the received slot sync" + " LSN %X/%X for slot \"%s\" is ahead of the standby position %X/%X", + LSN_FORMAT_ARGS(remote_slot->confirmed_lsn), + remote_slot->name, + LSN_FORMAT_ARGS(latestFlushPtr)); + + /* Search for the named slot */ + if ((slot = SearchNamedReplicationSlot(remote_slot->name, true))) + { + bool synced; + + SpinLockAcquire(&slot->mutex); + synced = slot->data.synced; + SpinLockRelease(&slot->mutex); + + /* User-created slot with the same name exists, raise ERROR. */ + if (!synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("exiting from slot synchronization because same" + " name slot \"%s\" already exists on the standby", + remote_slot->name)); + + /* + * The slot has been synchronized before. + * + * It is important to acquire the slot here before checking + * invalidation. If we don't acquire the slot first, there could be a + * race condition that the local slot could be invalidated just after + * checking the 'invalidated' flag here and we could end up + * overwriting 'invalidated' flag to remote_slot's value. See + * InvalidatePossiblyObsoleteSlot() where it invalidates slot directly + * if the slot is not acquired by other processes. + */ + ReplicationSlotAcquire(remote_slot->name, true); + + Assert(slot == MyReplicationSlot); + + /* + * Copy the invalidation cause from remote only if local slot is not + * invalidated locally, we don't want to overwrite existing one. + */ + if (slot->data.invalidated == RS_INVAL_NONE && + remote_slot->invalidated != RS_INVAL_NONE) + { + SpinLockAcquire(&slot->mutex); + slot->data.invalidated = remote_slot->invalidated; + SpinLockRelease(&slot->mutex); + + /* Make sure the invalidated state persists across server restart */ + ReplicationSlotMarkDirty(); + ReplicationSlotSave(); + } + + /* Skip the sync of an invalidated slot */ + if (slot->data.invalidated != RS_INVAL_NONE) + { + ReplicationSlotRelease(); + return; + } + + /* Slot not ready yet, let's attempt to make it sync-ready now. */ + if (slot->data.persistency == RS_TEMPORARY) + { + update_and_persist_local_synced_slot(remote_slot, remote_dbid); + } + + /* Slot ready for sync, so sync it. */ + else + { + /* + * Sanity check: As long as the invalidations are handled + * appropriately as above, this should never happen. + */ + if (remote_slot->restart_lsn < slot->data.restart_lsn) + elog(ERROR, + "cannot synchronize local slot \"%s\" LSN(%X/%X)" + " to remote slot's LSN(%X/%X) as synchronization" + " would move it backwards", remote_slot->name, + LSN_FORMAT_ARGS(slot->data.restart_lsn), + LSN_FORMAT_ARGS(remote_slot->restart_lsn)); + + /* Make sure the slot changes persist across server restart */ + if (update_local_synced_slot(remote_slot, remote_dbid)) + { + ReplicationSlotMarkDirty(); + ReplicationSlotSave(); + } + } + } + /* Otherwise create the slot first. */ + else + { + NameData plugin_name; + TransactionId xmin_horizon = InvalidTransactionId; + + /* Skip creating the local slot if remote_slot is invalidated already */ + if (remote_slot->invalidated != RS_INVAL_NONE) + return; + + ReplicationSlotCreate(remote_slot->name, true, RS_TEMPORARY, + remote_slot->two_phase, + remote_slot->failover, + true); + + /* For shorter lines. */ + slot = MyReplicationSlot; + + /* Avoid expensive operations while holding a spinlock. */ + namestrcpy(&plugin_name, remote_slot->plugin); + + SpinLockAcquire(&slot->mutex); + slot->data.database = remote_dbid; + slot->data.plugin = plugin_name; + SpinLockRelease(&slot->mutex); + + reserve_wal_for_local_slot(remote_slot->restart_lsn); + + LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE); + xmin_horizon = GetOldestSafeDecodingTransactionId(true); + SpinLockAcquire(&slot->mutex); + slot->effective_catalog_xmin = xmin_horizon; + slot->data.catalog_xmin = xmin_horizon; + SpinLockRelease(&slot->mutex); + ReplicationSlotsComputeRequiredXmin(true); + LWLockRelease(ProcArrayLock); + + update_and_persist_local_synced_slot(remote_slot, remote_dbid); + } + + ReplicationSlotRelease(); +} + +/* + * Synchronize slots. + * + * Gets the failover logical slots info from the primary server and updates + * the slots locally. Creates the slots if not present on the standby. + */ +static void +synchronize_slots(WalReceiverConn *wrconn) +{ +#define SLOTSYNC_COLUMN_COUNT 9 + Oid slotRow[SLOTSYNC_COLUMN_COUNT] = {TEXTOID, TEXTOID, LSNOID, + LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID}; + + WalRcvExecResult *res; + TupleTableSlot *tupslot; + StringInfoData s; + List *remote_slot_list = NIL; + + SpinLockAcquire(&SlotSyncCtx->mutex); + if (SlotSyncCtx->syncing) + { + SpinLockRelease(&SlotSyncCtx->mutex); + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot synchronize replication slots concurrently")); + } + + SlotSyncCtx->syncing = true; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = true; + + initStringInfo(&s); + + /* Construct query to fetch slots with failover enabled. */ + appendStringInfo(&s, + "SELECT slot_name, plugin, confirmed_flush_lsn," + " restart_lsn, catalog_xmin, two_phase, failover," + " database, conflict_reason" + " FROM pg_catalog.pg_replication_slots" + " WHERE failover and NOT temporary"); + + /* Execute the query */ + res = walrcv_exec(wrconn, s.data, SLOTSYNC_COLUMN_COUNT, slotRow); + pfree(s.data); + + if (res->status != WALRCV_OK_TUPLES) + ereport(ERROR, + errmsg("could not fetch failover logical slots info from the primary server: %s", + res->err)); + + /* Construct the remote_slot tuple and synchronize each slot locally */ + tupslot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple); + while (tuplestore_gettupleslot(res->tuplestore, true, false, tupslot)) + { + bool isnull; + RemoteSlot *remote_slot = palloc0(sizeof(RemoteSlot)); + Datum d; + int col = 0; + + remote_slot->name = TextDatumGetCString(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->plugin = TextDatumGetCString(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + /* + * It is possible to get null values for LSN and Xmin if slot is + * invalidated on the primary server, so handle accordingly. + */ + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->confirmed_lsn = isnull ? InvalidXLogRecPtr : + DatumGetLSN(d); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->restart_lsn = isnull ? InvalidXLogRecPtr : DatumGetLSN(d); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->catalog_xmin = isnull ? InvalidTransactionId : + DatumGetTransactionId(d); + + remote_slot->two_phase = DatumGetBool(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->failover = DatumGetBool(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->database = TextDatumGetCString(slot_getattr(tupslot, + ++col, &isnull)); + Assert(!isnull); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->invalidated = isnull ? RS_INVAL_NONE : + GetSlotInvalidationCause(TextDatumGetCString(d)); + + /* Sanity check */ + Assert(col == SLOTSYNC_COLUMN_COUNT); + + /* + * If restart_lsn, confirmed_lsn or catalog_xmin is invalid but the + * slot is valid, that means we have fetched the remote_slot in its + * RS_EPHEMERAL state. In such a case, don't sync it; we can always + * sync it in the next sync cycle when the remote_slot is persisted + * and has valid lsn(s) and xmin values. + * + * XXX: In future, if we plan to expose 'slot->data.persistency' in + * pg_replication_slots view, then we can avoid fetching RS_EPHEMERAL + * slots in the first place. + */ + if ((XLogRecPtrIsInvalid(remote_slot->restart_lsn) || + XLogRecPtrIsInvalid(remote_slot->confirmed_lsn) || + !TransactionIdIsValid(remote_slot->catalog_xmin)) && + remote_slot->invalidated == RS_INVAL_NONE) + pfree(remote_slot); + else + /* Create list of remote slots */ + remote_slot_list = lappend(remote_slot_list, remote_slot); + + ExecClearTuple(tupslot); + } + + /* Drop local slots that no longer need to be synced. */ + drop_local_obsolete_slots(remote_slot_list); + + /* Now sync the slots locally */ + foreach_ptr(RemoteSlot, remote_slot, remote_slot_list) + { + Oid remote_dbid = get_database_oid(remote_slot->database, false); + + /* + * Use shared lock to prevent a conflict with + * ReplicationSlotsDropDBSlots(), trying to drop the same slot during + * a drop-database operation. + */ + LockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); + + synchronize_one_slot(remote_slot, remote_dbid); + + UnlockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); + } + + /* We are done, free remote_slot_list elements */ + list_free_deep(remote_slot_list); + + walrcv_clear_result(res); + + SpinLockAcquire(&SlotSyncCtx->mutex); + SlotSyncCtx->syncing = false; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = false; +} + +/* + * Checks the primary server info. + * + * Using the specified primary server connection, check whether we are a + * cascading standby. It also validates primary_slot_name for non-cascading + * standbys. + */ +static void +check_primary_info(WalReceiverConn *wrconn, int elevel) +{ +#define PRIMARY_INFO_OUTPUT_COL_COUNT 2 + WalRcvExecResult *res; + Oid slotRow[PRIMARY_INFO_OUTPUT_COL_COUNT] = {BOOLOID, BOOLOID}; + StringInfoData cmd; + bool isnull; + TupleTableSlot *tupslot; + bool remote_in_recovery; + bool primary_info_valid; + + initStringInfo(&cmd); + appendStringInfo(&cmd, + "SELECT pg_is_in_recovery(), count(*) = 1" + " FROM pg_catalog.pg_replication_slots" + " WHERE slot_type='physical' AND slot_name=%s", + quote_literal_cstr(PrimarySlotName)); + + res = walrcv_exec(wrconn, cmd.data, PRIMARY_INFO_OUTPUT_COL_COUNT, slotRow); + pfree(cmd.data); + + if (res->status != WALRCV_OK_TUPLES) + ereport(ERROR, + errmsg("could not fetch primary_slot_name \"%s\" info from the primary server: %s", + PrimarySlotName, res->err), + errhint("Check if \"primary_slot_name\" is configured correctly.")); + + tupslot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple); + if (!tuplestore_gettupleslot(res->tuplestore, true, false, tupslot)) + elog(ERROR, + "failed to fetch tuple for the primary server slot specified by \"primary_slot_name\""); + + remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull)); + Assert(!isnull); + + if (remote_in_recovery) + ereport(elevel, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot synchronize replication slots from a standby server")); + + primary_info_valid = DatumGetBool(slot_getattr(tupslot, 2, &isnull)); + Assert(!isnull); + + if (!primary_info_valid) + ereport(elevel, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + /* translator: second %s is a GUC variable name */ + errdetail("The replication slot \"%s\" specified by \"%s\" does not exist on the primary server.", + PrimarySlotName, "primary_slot_name")); + + ExecClearTuple(tupslot); + walrcv_clear_result(res); +} + +/* + * Check all necessary GUCs for slot synchronization are set + * appropriately, otherwise, raise ERROR. + */ +void +ValidateSlotSyncParams(void) +{ + char *dbname; + + /* + * A physical replication slot(primary_slot_name) is required on the + * primary to ensure that the rows needed by the standby are not removed + * after restarting, so that the synchronized slot on the standby will not + * be invalidated. + */ + if (PrimarySlotName == NULL || *PrimarySlotName == '\0') + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be defined.", "primary_slot_name")); + + /* + * hot_standby_feedback must be enabled to cooperate with the physical + * replication slot, which allows informing the primary about the xmin and + * catalog_xmin values on the standby. + */ + if (!hot_standby_feedback) + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be enabled.", "hot_standby_feedback")); + + /* Logical slot sync/creation requires wal_level >= logical. */ + if (wal_level < WAL_LEVEL_LOGICAL) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"wal_level\" must be >= logical.")); + + /* + * The primary_conninfo is required to make connection to primary for + * getting slots information. + */ + if (PrimaryConnInfo == NULL || *PrimaryConnInfo == '\0') + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be defined.", "primary_conninfo")); + + /* + * The slot synchronization needs a database connection for walrcv_exec to + * work. + */ + dbname = walrcv_get_dbname_from_conninfo(PrimaryConnInfo); + if (dbname == NULL) + ereport(ERROR, + + /* + * translator: 'dbname' is a specific option; %s is a GUC variable + * name + */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("'dbname' must be specified in \"%s\".", "primary_conninfo")); +} + +/* + * Is current process syncing replication slots ? + */ +bool +IsSyncingReplicationSlots(void) +{ + return syncing_slots; +} + +/* + * Amount of shared memory required for slot synchronization. + */ +Size +SlotSyncShmemSize(void) +{ + return sizeof(SlotSyncCtxStruct); +} + +/* + * Allocate and initialize the shared memory of slot synchronization. + */ +void +SlotSyncShmemInit(void) +{ + bool found; + + SlotSyncCtx = (SlotSyncCtxStruct *) + ShmemInitStruct("Slot Sync Data", SlotSyncShmemSize(), &found); + + if (!found) + { + SlotSyncCtx->syncing = false; + SpinLockInit(&SlotSyncCtx->mutex); + } +} + +/* + * Error cleanup callback for slot synchronization. + */ +static void +slotsync_failure_callback(int code, Datum arg) +{ + WalReceiverConn *wrconn = (WalReceiverConn *) DatumGetPointer(arg); + + if (syncing_slots) + { + /* + * If syncing_slots is true, it indicates that the process errored out + * without resetting the flag. So, we need to clean up shared memory + * and reset the flag here. + */ + SpinLockAcquire(&SlotSyncCtx->mutex); + SlotSyncCtx->syncing = false; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = false; + } + + walrcv_disconnect(wrconn); +} + +/* + * Synchronize the failover enabled replication slots using the specified + * primary server connection. + */ +void +SyncReplicationSlots(WalReceiverConn *wrconn) +{ + PG_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); + { + check_primary_info(wrconn, ERROR); + + synchronize_slots(wrconn); + } + PG_END_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); + + walrcv_disconnect(wrconn); +} diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index fd4e96c9d6..7df22a0251 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -46,6 +46,7 @@ #include "common/string.h" #include "miscadmin.h" #include "pgstat.h" +#include "replication/slotsync.h" #include "replication/slot.h" #include "storage/fd.h" #include "storage/ipc.h" @@ -103,7 +104,6 @@ int max_replication_slots = 10; /* the maximum number of replication * slots */ static void ReplicationSlotShmemExit(int code, Datum arg); -static void ReplicationSlotDropAcquired(void); static void ReplicationSlotDropPtr(ReplicationSlot *slot); /* internal persistency functions */ @@ -250,11 +250,12 @@ ReplicationSlotValidateName(const char *name, int elevel) * user will only get commit prepared. * failover: If enabled, allows the slot to be synced to standbys so * that logical replication can be resumed after failover. + * synced: True if the slot is synchronized from the primary server. */ void ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotPersistency persistency, - bool two_phase, bool failover) + bool two_phase, bool failover, bool synced) { ReplicationSlot *slot = NULL; int i; @@ -263,6 +264,34 @@ ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotValidateName(name, ERROR); + if (failover) + { + /* + * Do not allow users to create failover enabled temporary slots, + * because temporary slots will not be synced to the standby. + * + * However, failover enabled temporary slots can be created during + * slot synchronization. See the comments atop slotsync.c for details. + */ + if (persistency == RS_TEMPORARY && !IsSyncingReplicationSlots()) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a temporary replication slot")); + + /* + * Do not allow users to create the failover enabled slots on the + * standby as we do not support sync to the cascading standby. + * + * However, failover enabled slots can be created during slot + * synchronization because we need to retain the same values as the + * remote slot. + */ + if (RecoveryInProgress() && !IsSyncingReplicationSlots()) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a replication slot created on the standby")); + } + /* * If some other backend ran this code concurrently with us, we'd likely * both allocate the same slot, and that would be bad. We'd also be at @@ -315,6 +344,7 @@ ReplicationSlotCreate(const char *name, bool db_specific, slot->data.two_phase = two_phase; slot->data.two_phase_at = InvalidXLogRecPtr; slot->data.failover = failover; + slot->data.synced = synced; /* and then data only present in shared memory */ slot->just_dirtied = false; @@ -677,6 +707,16 @@ ReplicationSlotDrop(const char *name, bool nowait) ReplicationSlotAcquire(name, nowait); + /* + * Do not allow users to drop the slots which are currently being synced + * from the primary to the standby. + */ + if (RecoveryInProgress() && MyReplicationSlot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot drop replication slot \"%s\"", name), + errdetail("This slot is being synced from the primary server.")); + ReplicationSlotDropAcquired(); } @@ -696,6 +736,38 @@ ReplicationSlotAlter(const char *name, bool failover) errmsg("cannot use %s with a physical replication slot", "ALTER_REPLICATION_SLOT")); + /* + * Do not allow users to enable failover for temporary slots as we do not + * support syncing temporary slots to the standby. + */ + if (failover && MyReplicationSlot->data.persistency == RS_TEMPORARY) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a temporary replication slot")); + + if (RecoveryInProgress()) + { + /* + * Do not allow users to alter the slots which are currently being + * synced from the primary to the standby. + */ + if (MyReplicationSlot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot alter replication slot \"%s\"", name), + errdetail("This slot is being synced from the primary server.")); + + /* + * Do not allow users to enable failover on the standby as we do not + * support sync to the cascading standby. + */ + if (failover) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a replication slot" + " on the standby")); + } + if (MyReplicationSlot->data.failover != failover) { SpinLockAcquire(&MyReplicationSlot->mutex); @@ -712,7 +784,7 @@ ReplicationSlotAlter(const char *name, bool failover) /* * Permanently drop the currently acquired replication slot. */ -static void +void ReplicationSlotDropAcquired(void) { ReplicationSlot *slot = MyReplicationSlot; @@ -868,8 +940,8 @@ ReplicationSlotMarkDirty(void) } /* - * Convert a slot that's marked as RS_EPHEMERAL to a RS_PERSISTENT slot, - * guaranteeing it will be there after an eventual crash. + * Convert a slot that's marked as RS_EPHEMERAL or RS_TEMPORARY to a + * RS_PERSISTENT slot, guaranteeing it will be there after an eventual crash. */ void ReplicationSlotPersist(void) @@ -2189,3 +2261,25 @@ RestoreSlotFromDisk(const char *name) (errmsg("too many replication slots active before shutdown"), errhint("Increase max_replication_slots and try again."))); } + +/* + * Maps the pg_replication_slots.conflict_reason text value to + * ReplicationSlotInvalidationCause enum value + */ +ReplicationSlotInvalidationCause +GetSlotInvalidationCause(char *conflict_reason) +{ + Assert(conflict_reason); + + if (strcmp(conflict_reason, SLOT_INVAL_WAL_REMOVED_TEXT) == 0) + return RS_INVAL_WAL_REMOVED; + else if (strcmp(conflict_reason, SLOT_INVAL_HORIZON_TEXT) == 0) + return RS_INVAL_HORIZON; + else if (strcmp(conflict_reason, SLOT_INVAL_WAL_LEVEL_TEXT) == 0) + return RS_INVAL_WAL_LEVEL; + else + Assert(0); + + /* Keep compiler quiet */ + return RS_INVAL_NONE; +} diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index eb685089b3..527e0c8ed5 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -21,7 +21,9 @@ #include "replication/decode.h" #include "replication/logical.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "utils/builtins.h" +#include "utils/guc.h" #include "utils/inval.h" #include "utils/pg_lsn.h" #include "utils/resowner.h" @@ -43,7 +45,7 @@ create_physical_replication_slot(char *name, bool immediately_reserve, /* acquire replication slot, this will check for conflicting names */ ReplicationSlotCreate(name, false, temporary ? RS_TEMPORARY : RS_PERSISTENT, false, - false); + false, false); if (immediately_reserve) { @@ -136,7 +138,7 @@ create_logical_replication_slot(char *name, char *plugin, */ ReplicationSlotCreate(name, true, temporary ? RS_TEMPORARY : RS_EPHEMERAL, two_phase, - failover); + failover, false); /* * Create logical decoding context to find start point or, if we don't @@ -237,7 +239,7 @@ pg_drop_replication_slot(PG_FUNCTION_ARGS) Datum pg_get_replication_slots(PG_FUNCTION_ARGS) { -#define PG_GET_REPLICATION_SLOTS_COLS 16 +#define PG_GET_REPLICATION_SLOTS_COLS 17 ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; XLogRecPtr currlsn; int slotno; @@ -418,21 +420,23 @@ pg_get_replication_slots(PG_FUNCTION_ARGS) break; case RS_INVAL_WAL_REMOVED: - values[i++] = CStringGetTextDatum("wal_removed"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_WAL_REMOVED_TEXT); break; case RS_INVAL_HORIZON: - values[i++] = CStringGetTextDatum("rows_removed"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_HORIZON_TEXT); break; case RS_INVAL_WAL_LEVEL: - values[i++] = CStringGetTextDatum("wal_level_insufficient"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_WAL_LEVEL_TEXT); break; } } values[i++] = BoolGetDatum(slot_contents.data.failover); + values[i++] = BoolGetDatum(slot_contents.data.synced); + Assert(i == PG_GET_REPLICATION_SLOTS_COLS); tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, @@ -700,7 +704,6 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) XLogRecPtr src_restart_lsn; bool src_islogical; bool temporary; - bool failover; char *plugin; Datum values[2]; bool nulls[2]; @@ -756,7 +759,6 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) src_islogical = SlotIsLogical(&first_slot_contents); src_restart_lsn = first_slot_contents.data.restart_lsn; temporary = (first_slot_contents.data.persistency == RS_TEMPORARY); - failover = first_slot_contents.data.failover; plugin = logical_slot ? NameStr(first_slot_contents.data.plugin) : NULL; /* Check type of replication slot */ @@ -791,12 +793,20 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) * We must not try to read WAL, since we haven't reserved it yet -- * hence pass find_startpoint false. confirmed_flush will be set * below, by copying from the source slot. + * + * To avoid potential issues with the slot synchronization where the + * restart_lsn of a replication slot can go backward, we set the + * failover option to false here. This situation occurs when a slot + * on the primary server is dropped and immediately replaced with a + * new slot of the same name, created by copying from another existing + * slot. However, the slot synchronization will only observe the + * restart_lsn of the same slot going backward. */ create_logical_replication_slot(NameStr(*dst_name), plugin, temporary, false, - failover, + false, src_restart_lsn, false); } @@ -943,3 +953,47 @@ pg_copy_physical_replication_slot_b(PG_FUNCTION_ARGS) { return copy_replication_slot(fcinfo, false); } + +/* + * Synchronize failover enabled replication slots to a standby server + * from the primary server. + */ +Datum +pg_sync_replication_slots(PG_FUNCTION_ARGS) +{ + WalReceiverConn *wrconn; + char *err; + StringInfoData app_name; + + CheckSlotPermissions(); + + if (!RecoveryInProgress()) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("replication slots can only be synchronized to a standby server")); + + /* Load the libpq-specific functions */ + load_file("libpqwalreceiver", false); + + ValidateSlotSyncParams(); + + initStringInfo(&app_name); + if (cluster_name[0]) + appendStringInfo(&app_name, "%s_slotsync", cluster_name); + else + appendStringInfoString(&app_name, "slotsync"); + + /* Connect to the primary server. */ + wrconn = walrcv_connect(PrimaryConnInfo, false, false, false, + app_name.data, &err); + pfree(app_name.data); + + if (!wrconn) + ereport(ERROR, + errcode(ERRCODE_CONNECTION_FAILURE), + errmsg("could not connect to the primary server: %s", err)); + + SyncReplicationSlots(wrconn); + + PG_RETURN_VOID(); +} diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 146826d5db..4e54779a9e 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -72,6 +72,7 @@ #include "postmaster/interrupt.h" #include "replication/decode.h" #include "replication/logical.h" +#include "replication/slotsync.h" #include "replication/slot.h" #include "replication/snapbuild.h" #include "replication/syncrep.h" @@ -243,7 +244,6 @@ static void WalSndShutdown(void) pg_attribute_noreturn(); static void XLogSendPhysical(void); static void XLogSendLogical(void); static void WalSndDone(WalSndSendDataCallback send_data); -static XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); static void IdentifySystem(void); static void UploadManifest(void); static bool HandleUploadManifestPacket(StringInfo buf, off_t *offset, @@ -1224,7 +1224,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) { ReplicationSlotCreate(cmd->slotname, false, cmd->temporary ? RS_TEMPORARY : RS_PERSISTENT, - false, false); + false, false, false); if (reserve_wal) { @@ -1255,7 +1255,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) */ ReplicationSlotCreate(cmd->slotname, true, cmd->temporary ? RS_TEMPORARY : RS_EPHEMERAL, - two_phase, failover); + two_phase, failover, false); /* * Do options check early so that we can bail before calling the @@ -3385,14 +3385,17 @@ WalSndDone(WalSndSendDataCallback send_data) } /* - * Returns the latest point in WAL that has been safely flushed to disk, and - * can be sent to the standby. This should only be called when in recovery, - * ie. we're streaming to a cascaded standby. + * Returns the latest point in WAL that has been safely flushed to disk. + * This should only be called when in recovery. + * + * This is called either by cascading walsender to find WAL postion to be sent + * to a cascaded standby or by slot synchronization function to validate remote + * slot's lsn before syncing it locally. * * As a side-effect, *tli is updated to the TLI of the last * replayed WAL record. */ -static XLogRecPtr +XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli) { XLogRecPtr replayPtr; @@ -3401,6 +3404,8 @@ GetStandbyFlushRecPtr(TimeLineID *tli) TimeLineID receiveTLI; XLogRecPtr result; + Assert(am_cascading_walsender || IsSyncingReplicationSlots()); + /* * We can safely send what's already been replayed. Also, if walreceiver * is streaming WAL from the same timeline, we can send anything that it diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index 7084e18861..7e7941d625 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -36,6 +36,7 @@ #include "replication/logicallauncher.h" #include "replication/origin.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/walreceiver.h" #include "replication/walsender.h" #include "storage/bufmgr.h" @@ -153,6 +154,7 @@ CalculateShmemSize(int *num_semaphores) size = add_size(size, StatsShmemSize()); size = add_size(size, WaitEventExtensionShmemSize()); size = add_size(size, InjectionPointShmemSize()); + size = add_size(size, SlotSyncShmemSize()); #ifdef EXEC_BACKEND size = add_size(size, ShmemBackendArraySize()); #endif @@ -347,6 +349,7 @@ CreateOrAttachShmemStructs(void) WalSummarizerShmemInit(); PgArchShmemInit(); ApplyLauncherShmemInit(); + SlotSyncShmemInit(); /* * Set up other modules that need some shared memory space diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 29af4ce65d..9c120fc2b7 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11127,9 +11127,9 @@ proname => 'pg_get_replication_slots', prorows => '10', proisstrict => 'f', proretset => 't', provolatile => 's', prorettype => 'record', proargtypes => '', - proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,text,bool}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,conflict_reason,failover}', + proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,text,bool,bool}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,conflict_reason,failover,synced}', prosrc => 'pg_get_replication_slots' }, { oid => '3786', descr => 'set up a logical replication slot', proname => 'pg_create_logical_replication_slot', provolatile => 'v', @@ -11212,6 +11212,10 @@ proname => 'pg_logical_emit_message', provolatile => 'v', proparallel => 'u', prorettype => 'pg_lsn', proargtypes => 'bool text bytea bool', prosrc => 'pg_logical_emit_message_bytea' }, +{ oid => '9929', descr => 'sync replication slots from the primary to the standby', + proname => 'pg_sync_replication_slots', provolatile => 'v', proparallel => 'u', + prorettype => 'void', proargtypes => '', + prosrc => 'pg_sync_replication_slots' }, # event triggers { oid => '3566', descr => 'list objects dropped by the current command', diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index da4c776492..e706ca834c 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -52,6 +52,14 @@ typedef enum ReplicationSlotInvalidationCause RS_INVAL_WAL_LEVEL, } ReplicationSlotInvalidationCause; +/* + * The possible values for 'conflict_reason' returned in + * pg_get_replication_slots. + */ +#define SLOT_INVAL_WAL_REMOVED_TEXT "wal_removed" +#define SLOT_INVAL_HORIZON_TEXT "rows_removed" +#define SLOT_INVAL_WAL_LEVEL_TEXT "wal_level_insufficient" + /* * On-Disk data of a replication slot, preserved across restarts. */ @@ -112,6 +120,11 @@ typedef struct ReplicationSlotPersistentData /* plugin name */ NameData plugin; + /* + * Was this slot synchronized from the primary server? + */ + char synced; + /* * Is this a failover slot (sync candidate for standbys)? Only relevant * for logical slots on the primary server. @@ -224,9 +237,11 @@ extern void ReplicationSlotsShmemInit(void); /* management of individual slots */ extern void ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotPersistency persistency, - bool two_phase, bool failover); + bool two_phase, bool failover, + bool synced); extern void ReplicationSlotPersist(void); extern void ReplicationSlotDrop(const char *name, bool nowait); +extern void ReplicationSlotDropAcquired(void); extern void ReplicationSlotAlter(const char *name, bool failover); extern void ReplicationSlotAcquire(const char *name, bool nowait); @@ -259,5 +274,7 @@ extern void CheckPointReplicationSlots(bool is_shutdown); extern void CheckSlotRequirements(void); extern void CheckSlotPermissions(void); +extern ReplicationSlotInvalidationCause + GetSlotInvalidationCause(char *conflict_reason); #endif /* SLOT_H */ diff --git a/src/include/replication/slotsync.h b/src/include/replication/slotsync.h new file mode 100644 index 0000000000..e86d8a47b8 --- /dev/null +++ b/src/include/replication/slotsync.h @@ -0,0 +1,23 @@ +/*------------------------------------------------------------------------- + * + * slotsync.h + * Exports for slot synchronization. + * + * Portions Copyright (c) 2016-2024, PostgreSQL Global Development Group + * + * src/include/replication/slotsync.h + * + *------------------------------------------------------------------------- + */ +#ifndef SLOTSYNC_H +#define SLOTSYNC_H + +#include "replication/walreceiver.h" + +extern void ValidateSlotSyncParams(void); +extern bool IsSyncingReplicationSlots(void); +extern Size SlotSyncShmemSize(void); +extern void SlotSyncShmemInit(void); +extern void SyncReplicationSlots(WalReceiverConn *wrconn); + +#endif /* SLOTSYNC_H */ diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h index 1b58d50b3b..0c3996e926 100644 --- a/src/include/replication/walsender.h +++ b/src/include/replication/walsender.h @@ -12,6 +12,8 @@ #ifndef _WALSENDER_H #define _WALSENDER_H +#include "access/xlogdefs.h" + /* * What to do with a snapshot in create replication slot command. */ @@ -37,6 +39,7 @@ extern void InitWalSender(void); extern bool exec_replication_command(const char *cmd_string); extern void WalSndErrorCleanup(void); extern void WalSndResourceCleanup(bool isCommit); +extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); extern void WalSndSignals(void); extern Size WalSndShmemSize(void); extern void WalSndShmemInit(void); diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index bc58ff4cab..7536173776 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -97,4 +97,197 @@ my ($result, $stdout, $stderr) = $subscriber1->psql('postgres', ok( $stderr =~ /ERROR: cannot set failover for enabled subscription/, "altering failover is not allowed for enabled subscription"); +################################################## +# Test that pg_sync_replication_slots() cannot be executed on a non-standby server. +################################################## + +($result, $stdout, $stderr) = + $publisher->psql('postgres', "SELECT pg_sync_replication_slots();"); +ok( $stderr =~ + /ERROR: replication slots can only be synchronized to a standby server/, + "cannot sync slots on a non-standby server"); + +################################################## +# Test logical failover slots on the standby +# Configure standby1 to replicate and synchronize logical slots configured +# for failover on the primary +# +# failover slot lsub1_slot ->| ----> subscriber1 (connected via logical replication) +# failover slot lsub2_slot | inactive +# primary ---> | +# physical slot sb1_slot --->| ----> standby1 (connected via streaming replication) +# | lsub1_slot, lsub2_slot (synced_slot) +################################################## + +my $primary = $publisher; +my $backup_name = 'backup'; +$primary->backup($backup_name); + +# Create a standby +my $standby1 = PostgreSQL::Test::Cluster->new('standby1'); +$standby1->init_from_backup( + $primary, $backup_name, + has_streaming => 1, + has_restoring => 1); + +my $connstr_1 = $primary->connstr; +$standby1->append_conf( + 'postgresql.conf', qq( +hot_standby_feedback = on +primary_slot_name = 'sb1_slot' +primary_conninfo = '$connstr_1 dbname=postgres' +)); + +$primary->psql('postgres', + q{SELECT pg_create_logical_replication_slot('lsub2_slot', 'test_decoding', false, false, true);} +); + +$primary->psql('postgres', + q{SELECT pg_create_physical_replication_slot('sb1_slot');}); + +my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; +my $offset = -s $standby1->logfile; + +# Start the standby so that slot syncing can begin +$standby1->start; + +$primary->wait_for_catchup('regress_mysub1'); + +# Do not allow any further advancement of the restart_lsn for the lsub1_slot. +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); + +# Wait for the replication slot to become inactive on the publisher +$primary->poll_query_until( + 'postgres', + "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", + 1); + +# Wait for the standby to catch up so that the standby is not lagging behind +# the subscriber. +$primary->wait_for_replay_catchup($standby1); + +# Synchronize the primary server slots to the standby. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Confirm that the logical failover slots are created on the standby and are +# flagged as 'synced' +is( $standby1->safe_psql( + 'postgres', + q{SELECT count(*) = 2 FROM pg_replication_slots WHERE slot_name IN ('lsub1_slot', 'lsub2_slot') AND synced;} + ), + "t", + 'logical slots have synced as true on standby'); + +################################################## +# Test that the synchronized slot will be dropped if the corresponding remote +# slot on the primary server has been dropped. +################################################## + +$primary->psql('postgres', "SELECT pg_drop_replication_slot('lsub2_slot');"); + +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +is( $standby1->safe_psql( + 'postgres', + q{SELECT count(*) = 0 FROM pg_replication_slots WHERE slot_name = 'lsub2_slot';} + ), + "t", + 'synchronized slot has been dropped'); + +################################################## +# Test that if the synchronized slot is invalidated while the remote slot is +# still valid, the slot will be dropped and re-created on the standby by +# executing pg_sync_replication_slots() again. +################################################## + +# Configure the max_slot_wal_keep_size so that the synced slot can be +# invalidated due to wal removal. +$standby1->append_conf('postgresql.conf', 'max_slot_wal_keep_size = 64kB'); +$standby1->reload; + +# Generate some activity and switch WAL file on the primary +$primary->advance_wal(1); +$primary->psql('postgres', "CHECKPOINT;"); +$primary->wait_for_replay_catchup($standby1); + +# Request a checkpoint on the standby to trigger the WAL file(s) removal +$standby1->safe_psql('postgres', "CHECKPOINT;"); + +# Check if the synced slot is invalidated +is( $standby1->safe_psql( + 'postgres', + q{SELECT conflict_reason = 'wal_removed' FROM pg_replication_slots WHERE slot_name = 'lsub1_slot';} + ), + "t", + 'synchronized slot has been invalidated'); + +# Reset max_slot_wal_keep_size to avoid further wal removal +$standby1->append_conf('postgresql.conf', 'max_slot_wal_keep_size = -1'); +$standby1->reload; + +# Enable the subscription to let it catch up to the latest wal position +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 ENABLE"); + +$primary->wait_for_catchup('regress_mysub1'); + +# Do not allow any further advancement of the restart_lsn for the lsub1_slot. +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); + +# Wait for the replication slot to become inactive on the publisher +$primary->poll_query_until( + 'postgres', + "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", + 1); + +# Wait for the standby to catch up so that the standby is not lagging behind +# the subscriber. +$primary->wait_for_replay_catchup($standby1); + +my $log_offset = -s $standby1->logfile; + +# Synchronize the primary server slots to the standby. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Confirm that the invalidated slot has been dropped. +$standby1->wait_for_log(qr/dropped replication slot "lsub1_slot" of dbid 5/, + $log_offset); + +# Confirm that the logical slot has been re-created on the standby and are +# flagged as 'synced' +is( $standby1->safe_psql( + 'postgres', + q{SELECT conflict_reason IS NULL AND synced FROM pg_replication_slots WHERE slot_name = 'lsub1_slot';} + ), + "t", + 'logical slot is re-synced'); + +################################################## +# Test that a synchronized slot can not be decoded, altered or dropped by the +# user +################################################## + +# Attempting to perform logical decoding on a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql('postgres', + "select * from pg_logical_slot_get_changes('lsub1_slot', NULL, NULL);"); +ok( $stderr =~ + /ERROR: cannot use replication slot "lsub1_slot" for logical decoding/, + "logical decoding is not allowed on synced slot"); + +# Attempting to alter a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql( + 'postgres', + qq[ALTER_REPLICATION_SLOT lsub1_slot (failover);], + replication => 'database'); +ok($stderr =~ /ERROR: cannot alter replication slot "lsub1_slot"/, + "synced slot on standby cannot be altered"); + +# Attempting to drop a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql('postgres', + "SELECT pg_drop_replication_slot('lsub1_slot');"); +ok($stderr =~ /ERROR: cannot drop replication slot "lsub1_slot"/, + "synced slot on standby cannot be dropped"); + done_testing(); diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index abc944e8b8..b7488d760e 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1474,8 +1474,9 @@ pg_replication_slots| SELECT l.slot_name, l.safe_wal_size, l.two_phase, l.conflict_reason, - l.failover - FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, conflict_reason, failover) + l.failover, + l.synced + FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, conflict_reason, failover, synced) LEFT JOIN pg_database d ON ((l.datoid = d.oid))); pg_roles| SELECT pg_authid.rolname, pg_authid.rolsuper, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 91433d439b..d808aad8b0 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2325,6 +2325,7 @@ RelocationBufferInfo RelptrFreePageBtree RelptrFreePageManager RelptrFreePageSpanLeader +RemoteSlot RenameStmt ReopenPtrType ReorderBuffer @@ -2584,6 +2585,7 @@ SlabBlock SlabContext SlabSlot SlotNumber +SlotSyncCtxStruct SlruCtl SlruCtlData SlruErrorCause -- 2.43.0 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-13 06:50 Amit Kapila <[email protected]> parent: Zhijie Hou (Fujitsu) <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: Amit Kapila @ 2024-02-13 06:50 UTC (permalink / raw) To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Tue, Feb 13, 2024 at 9:38 AM Zhijie Hou (Fujitsu) <[email protected]> wrote: > > Here is the V85_2 patch set that added the test and fixed one typo, > there are no other code changes. > Few comments on the latest changes: ============================== 1. +# Confirm that the invalidated slot has been dropped. +$standby1->wait_for_log(qr/dropped replication slot "lsub1_slot" of dbid 5/, + $log_offset); Is it okay to hardcode dbid 5? I am a bit worried that it can lead to instability in the test. 2. +check_primary_info(WalReceiverConn *wrconn, int elevel) +{ .. + bool primary_info_valid; I don't think for 0001, we need an elevel as an argument, so let's remove it. Additionally, can we change the variable name primary_info_valid to primary_slot_valid? Also, can we change the function name to validate_remote_info() as the remote can be both primary or standby? 3. +SyncReplicationSlots(WalReceiverConn *wrconn) +{ + PG_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); + { + check_primary_info(wrconn, ERROR); + + synchronize_slots(wrconn); + } + PG_END_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); + + walrcv_disconnect(wrconn); It is better to disconnect in the caller where we have made the connection. -- With Regards, Amit Kapila. diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index bfa2a13377..a9ff6acd31 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -365,19 +365,18 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU <title>Replication Slot Synchronization</title> <para> The logical replication slots on the primary can be synchronized to - the hot standby by enabling <literal>failover</literal> during slot - creation (e.g., using the <literal>failover</literal> parameter of + the hot standby by using the <literal>failover</literal> parameter of <link linkend="pg-create-logical-replication-slot"> <function>pg_create_logical_replication_slot</function></link>, or using the <link linkend="sql-createsubscription-params-with-failover"> <literal>failover</literal></link> option of - <command>CREATE SUBSCRIPTION</command>), and then calling + <command>CREATE SUBSCRIPTION</command> during slot creation, and then calling <link linkend="pg-sync-replication-slots"> <function>pg_sync_replication_slots</function></link> on the standby. For the synchronization to work, it is mandatory to - have a physical replication slot between the primary and the standby (e.g., + have a physical replication slot between the primary and the standby aka <link linkend="guc-primary-slot-name"><varname>primary_slot_name</varname></link> - should be configured on the standby), and + should be configured on the standby, and <link linkend="guc-hot-standby-feedback"><varname>hot_standby_feedback</varname></link> must be enabled on the standby. It is also necessary to specify a valid <literal>dbname</literal> in the diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index feb04e1451..eb238cef92 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -304,6 +304,12 @@ reserve_wal_for_local_slot(XLogRecPtr restart_lsn) * number. However, if no WAL segment files have been removed by a * checkpoint since startup, we need to search for the oldest segment * file from the current timeline existing in XLOGDIR. + * + * XXX: Currently, we are searching for the oldest segment in the + * current timeline as there is less chance of the slot's restart_lsn + * from being some prior timeline, and even if it happens, in the worst + * case, we will wait to sync till the slot's restart_lsn moved to the + * current timeline. */ oldest_segno = XLogGetLastRemovedSegno() + 1; @@ -685,11 +691,10 @@ synchronize_slots(WalReceiverConn *wrconn) } /* - * Checks the primary server info. + * Checks the remote server info. * - * Using the specified primary server connection, check whether we are a - * cascading standby. It also validates primary_slot_name for non-cascading - * standbys. + * We ensure that the 'primary_slot_name' exists on the remote server and + * the remote server is not a standby node. */ static void check_primary_info(WalReceiverConn *wrconn, int elevel) Attachments: [text/plain] v85_amit_1.patch.txt (3.1K, ../../CAA4eK1JXQRExUk48_M5EjE5k48MSi0P7T6bGhQusYv9bEDkrRQ@mail.gmail.com/2-v85_amit_1.patch.txt) download | inline diff: diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index bfa2a13377..a9ff6acd31 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -365,19 +365,18 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU <title>Replication Slot Synchronization</title> <para> The logical replication slots on the primary can be synchronized to - the hot standby by enabling <literal>failover</literal> during slot - creation (e.g., using the <literal>failover</literal> parameter of + the hot standby by using the <literal>failover</literal> parameter of <link linkend="pg-create-logical-replication-slot"> <function>pg_create_logical_replication_slot</function></link>, or using the <link linkend="sql-createsubscription-params-with-failover"> <literal>failover</literal></link> option of - <command>CREATE SUBSCRIPTION</command>), and then calling + <command>CREATE SUBSCRIPTION</command> during slot creation, and then calling <link linkend="pg-sync-replication-slots"> <function>pg_sync_replication_slots</function></link> on the standby. For the synchronization to work, it is mandatory to - have a physical replication slot between the primary and the standby (e.g., + have a physical replication slot between the primary and the standby aka <link linkend="guc-primary-slot-name"><varname>primary_slot_name</varname></link> - should be configured on the standby), and + should be configured on the standby, and <link linkend="guc-hot-standby-feedback"><varname>hot_standby_feedback</varname></link> must be enabled on the standby. It is also necessary to specify a valid <literal>dbname</literal> in the diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index feb04e1451..eb238cef92 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -304,6 +304,12 @@ reserve_wal_for_local_slot(XLogRecPtr restart_lsn) * number. However, if no WAL segment files have been removed by a * checkpoint since startup, we need to search for the oldest segment * file from the current timeline existing in XLOGDIR. + * + * XXX: Currently, we are searching for the oldest segment in the + * current timeline as there is less chance of the slot's restart_lsn + * from being some prior timeline, and even if it happens, in the worst + * case, we will wait to sync till the slot's restart_lsn moved to the + * current timeline. */ oldest_segno = XLogGetLastRemovedSegno() + 1; @@ -685,11 +691,10 @@ synchronize_slots(WalReceiverConn *wrconn) } /* - * Checks the primary server info. + * Checks the remote server info. * - * Using the specified primary server connection, check whether we are a - * cascading standby. It also validates primary_slot_name for non-cascading - * standbys. + * We ensure that the 'primary_slot_name' exists on the remote server and + * the remote server is not a standby node. */ static void check_primary_info(WalReceiverConn *wrconn, int elevel) ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-13 11:29 Bertrand Drouvot <[email protected]> parent: Zhijie Hou (Fujitsu) <[email protected]> 1 sibling, 2 replies; 46+ messages in thread From: Bertrand Drouvot @ 2024-02-13 11:29 UTC (permalink / raw) To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> Hi, On Tue, Feb 13, 2024 at 04:08:23AM +0000, Zhijie Hou (Fujitsu) wrote: > On Tuesday, February 13, 2024 9:16 AM Zhijie Hou (Fujitsu) <[email protected]> wrote: > > > > Here is the new version patch which addressed above and most of Bertrand's > > comments. > > > > TODO: trying to add one test for the case the slot is valid on primary while the > > synced slots is invalidated on the standby. > > Here is the V85_2 patch set that added the test and fixed one typo, > there are no other code changes. Thanks! Out of curiosity I ran a code coverage and the result for slotsync.c can be found in [1]. It appears that: - only one function is not covered (slotsync_failure_callback()). - 84% of the slotsync.c code is covered, the parts that are not are mainly related to "errors". Worth to try to extend the coverage? (I've in mind 731, 739, 766, 778, 786, 796, 808) [1]: https://htmlpreview.github.io/?https://raw.githubusercontent.com/bdrouvot/pg_code_coverage/main/src/... Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-13 11:50 Amit Kapila <[email protected]> parent: Bertrand Drouvot <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: Amit Kapila @ 2024-02-13 11:50 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Tue, Feb 13, 2024 at 4:59 PM Bertrand Drouvot <[email protected]> wrote: > > On Tue, Feb 13, 2024 at 04:08:23AM +0000, Zhijie Hou (Fujitsu) wrote: > > On Tuesday, February 13, 2024 9:16 AM Zhijie Hou (Fujitsu) <[email protected]> wrote: > > > > > > Here is the new version patch which addressed above and most of Bertrand's > > > comments. > > > > > > TODO: trying to add one test for the case the slot is valid on primary while the > > > synced slots is invalidated on the standby. > > > > Here is the V85_2 patch set that added the test and fixed one typo, > > there are no other code changes. > > Thanks! > > Out of curiosity I ran a code coverage and the result for slotsync.c can be > found in [1]. > > It appears that: > > - only one function is not covered (slotsync_failure_callback()). > - 84% of the slotsync.c code is covered, the parts that are not are mainly > related to "errors". > > Worth to try to extend the coverage? (I've in mind 731, 739, 766, 778, 786, 796, > 808) > All these additional line numbers mentioned by you are ERROR paths. I think if we want we can easily cover most of those but I am not sure if there is a benefit to cover each error path. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 46+ messages in thread
* RE: Synchronizing slots from primary to standby @ 2024-02-13 12:35 Zhijie Hou (Fujitsu) <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Zhijie Hou (Fujitsu) @ 2024-02-13 12:35 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Tuesday, February 13, 2024 2:51 PM Amit Kapila <[email protected]> wrote: > > On Tue, Feb 13, 2024 at 9:38 AM Zhijie Hou (Fujitsu) <[email protected]> > wrote: > > > > Here is the V85_2 patch set that added the test and fixed one typo, > > there are no other code changes. > > > > Few comments on the latest changes: Thanks for the comments. > ============================== > 1. > +# Confirm that the invalidated slot has been dropped. > +$standby1->wait_for_log(qr/dropped replication slot "lsub1_slot" of > +dbid 5/, $log_offset); > > Is it okay to hardcode dbid 5? I am a bit worried that it can lead to instability in > the test. > > 2. > +check_primary_info(WalReceiverConn *wrconn, int elevel) { > .. > + bool primary_info_valid; > > I don't think for 0001, we need an elevel as an argument, so let's remove it. > Additionally, can we change the variable name primary_info_valid to > primary_slot_valid? Also, can we change the function name to > validate_remote_info() as the remote can be both primary or standby? > > 3. > +SyncReplicationSlots(WalReceiverConn *wrconn) { > +PG_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, > +PointerGetDatum(wrconn)); { check_primary_info(wrconn, ERROR); > + > + synchronize_slots(wrconn); > + } > + PG_END_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, > PointerGetDatum(wrconn)); > + > + walrcv_disconnect(wrconn); > > It is better to disconnect in the caller where we have made the connection. All above comments look good to me. Here is the V86 patch that addressed above. This version also includes some other minor changes: 1. Added few comments for the temporary slot creation and XLogGetOldestSegno. 2. Adjusted the doc for the SQL function. 3. Reordered two error messages in slot create function. 4. Fixed few typos. Thanks Shveta for off-list discussions. Best Regards, Hou zj Attachments: [application/octet-stream] v86-0001-Add-a-slot-synchronization-function.patch (76.3K, ../../OS0PR01MB5716E581B4227DDEB4DE6C30944F2@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v86-0001-Add-a-slot-synchronization-function.patch) download | inline diff: From 962134961144ca782afd8bdf178335640da4748e Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Wed, 7 Feb 2024 10:37:31 +0530 Subject: [PATCH v86] Add a slot synchronization function. This commit introduces a new SQL function pg_sync_replication_slots() which is used to synchronize the logical replication slots from the primary server to the physical standby so that logical replication can be resumed after failover. A new 'synced' flag is introduced in pg_replication_slots view, indicating whether the slot has been synchronized from the primary server. On a standby, synced slots cannot be dropped or consumed, and any attempt to perform logical decoding on them will result in an error. The logical replication slots on the primary can be synchronized to the hot standby by enabling failover during slot creation (e.g. using the "failover" parameter of pg_create_logical_replication_slot(), or using the "failover" option of the CREATE SUBSCRIPTION command), and then calling pg_sync_replication_slots() function on the standby. For the synchronization to work, it is mandatory to have a physical replication slot between the primary and the standby, 'hot_standby_feedback' must be enabled on the standby and a valid dbname must be specified in 'primary_conninfo'. If a logical slot is invalidated on the primary, then that slot on the standby is also invalidated. If a logical slot on the primary is valid but is invalidated on the standby, then that slot is dropped but will be recreated on the standby in next pg_sync_replication_slots() call provided the slot still exists on the primary server. It is okay to recreate such slots as long as these are not consumable on the standby (which is the case currently). This situation may occur due to the following reasons: - The 'max_slot_wal_keep_size' on the standby is insufficient to retain WAL records from the restart_lsn of the slot. - 'primary_slot_name' is temporarily reset to null and the physical slot is removed. We may also see the slots invalidated and dropped on the standby if the primary changes 'wal_level' to a level lower than logical. Changing the primary 'wal_level' to a level lower than logical is only possible if the logical slots are removed on the primary server, so it's expected to see the slots being removed on the standby too (and re-created if they are re-created on the primary server). The slots synchronization status on the standby can be monitored using 'synced' column of pg_replication_slots view. --- .../test_decoding/expected/permissions.out | 3 + contrib/test_decoding/expected/slot.out | 2 + contrib/test_decoding/sql/permissions.sql | 1 + contrib/test_decoding/sql/slot.sql | 1 + doc/src/sgml/config.sgml | 9 +- doc/src/sgml/func.sgml | 35 +- doc/src/sgml/logicaldecoding.sgml | 56 ++ doc/src/sgml/protocol.sgml | 6 +- doc/src/sgml/system-views.sgml | 20 +- src/backend/catalog/system_views.sql | 3 +- src/backend/replication/logical/Makefile | 1 + src/backend/replication/logical/logical.c | 12 + src/backend/replication/logical/meson.build | 1 + src/backend/replication/logical/slotsync.c | 909 ++++++++++++++++++ src/backend/replication/slot.c | 104 +- src/backend/replication/slotfuncs.c | 74 +- src/backend/replication/walsender.c | 19 +- src/backend/storage/ipc/ipci.c | 3 + src/include/catalog/pg_proc.dat | 10 +- src/include/replication/slot.h | 19 +- src/include/replication/slotsync.h | 23 + src/include/replication/walsender.h | 3 + .../t/040_standby_failover_slots_sync.pl | 193 ++++ src/test/regress/expected/rules.out | 5 +- src/tools/pgindent/typedefs.list | 2 + 25 files changed, 1479 insertions(+), 35 deletions(-) create mode 100644 src/backend/replication/logical/slotsync.c create mode 100644 src/include/replication/slotsync.h diff --git a/contrib/test_decoding/expected/permissions.out b/contrib/test_decoding/expected/permissions.out index d6eaba8c55..8d100646ce 100644 --- a/contrib/test_decoding/expected/permissions.out +++ b/contrib/test_decoding/expected/permissions.out @@ -64,6 +64,9 @@ DETAIL: Only roles with the REPLICATION attribute may use replication slots. SELECT pg_drop_replication_slot('regression_slot'); ERROR: permission denied to use replication slots DETAIL: Only roles with the REPLICATION attribute may use replication slots. +SELECT pg_sync_replication_slots(); +ERROR: permission denied to use replication slots +DETAIL: Only roles with the REPLICATION attribute may use replication slots. RESET ROLE; -- replication users can drop superuser created slots SET ROLE regress_lr_superuser; diff --git a/contrib/test_decoding/expected/slot.out b/contrib/test_decoding/expected/slot.out index 261d8886d3..349ab2d380 100644 --- a/contrib/test_decoding/expected/slot.out +++ b/contrib/test_decoding/expected/slot.out @@ -425,6 +425,8 @@ SELECT 'init' FROM pg_create_logical_replication_slot('failover_default_slot', ' init (1 row) +SELECT 'init' FROM pg_create_logical_replication_slot('failover_true_temp_slot', 'test_decoding', true, false, true); +ERROR: cannot enable failover for a temporary replication slot SELECT 'init' FROM pg_create_physical_replication_slot('physical_slot'); ?column? ---------- diff --git a/contrib/test_decoding/sql/permissions.sql b/contrib/test_decoding/sql/permissions.sql index 312b514593..94db936aee 100644 --- a/contrib/test_decoding/sql/permissions.sql +++ b/contrib/test_decoding/sql/permissions.sql @@ -29,6 +29,7 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_d INSERT INTO lr_test VALUES('lr_superuser_init'); SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); SELECT pg_drop_replication_slot('regression_slot'); +SELECT pg_sync_replication_slots(); RESET ROLE; -- replication users can drop superuser created slots diff --git a/contrib/test_decoding/sql/slot.sql b/contrib/test_decoding/sql/slot.sql index 45aeae7fd5..580e3ae3be 100644 --- a/contrib/test_decoding/sql/slot.sql +++ b/contrib/test_decoding/sql/slot.sql @@ -181,6 +181,7 @@ SELECT pg_drop_replication_slot('copied_slot2_notemp'); SELECT 'init' FROM pg_create_logical_replication_slot('failover_true_slot', 'test_decoding', false, false, true); SELECT 'init' FROM pg_create_logical_replication_slot('failover_false_slot', 'test_decoding', false, false, false); SELECT 'init' FROM pg_create_logical_replication_slot('failover_default_slot', 'test_decoding', false, false); +SELECT 'init' FROM pg_create_logical_replication_slot('failover_true_temp_slot', 'test_decoding', true, false, true); SELECT 'init' FROM pg_create_physical_replication_slot('physical_slot'); SELECT slot_name, slot_type, failover FROM pg_replication_slots; diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 61038472c5..037a3b8a64 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4612,8 +4612,13 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" <varname>primary_conninfo</varname> string, or in a separate <filename>~/.pgpass</filename> file on the standby server (use <literal>replication</literal> as the database name). - Do not specify a database name in the - <varname>primary_conninfo</varname> string. + </para> + <para> + For replication slot synchronization (see + <xref linkend="logicaldecoding-replication-slots-synchronization"/>), + it is also necessary to specify a valid <literal>dbname</literal> + in the <varname>primary_conninfo</varname> string. This will only be + used for slot synchronization. It is ignored for streaming. </para> <para> This parameter can only be set in the <filename>postgresql.conf</filename> diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 11d537b341..8f147a2417 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -28075,7 +28075,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset </row> <row> - <entry role="func_table_entry"><para role="func_signature"> + <entry id="pg-create-logical-replication-slot" role="func_table_entry"><para role="func_signature"> <indexterm> <primary>pg_create_logical_replication_slot</primary> </indexterm> @@ -28444,6 +28444,39 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset record is flushed along with its transaction. </para></entry> </row> + + <row> + <entry id="pg-sync-replication-slots" role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>pg_sync_replication_slots</primary> + </indexterm> + <function>pg_sync_replication_slots</function> () + <returnvalue>void</returnvalue> + </para> + <para> + Synchronize the logical failover replication slots from the primary + server to the standby server. This function can only be executed on the + standby server. Temporary synced slots, if any, cannot be used for + logical decoding and must be dropped after promotion. See + <xref linkend="logicaldecoding-replication-slots-synchronization"/> for details. + </para> + + <caution> + <para> + If, after executing the function, + <link linkend="guc-hot-standby-feedback"> + <varname>hot_standby_feedback</varname></link> is disabled on + the standby or the physical slot configured in + <link linkend="guc-primary-slot-name"> + <varname>primary_slot_name</varname></link> is + removed, then it is possible that the necessary rows of the + synchronized slot will be removed by the VACUUM process on the primary + server, resulting in the synchronized slot becoming invalidated. + </para> + </caution> + </entry> + </row> + </tbody> </tgroup> </table> diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index cd152d4ced..eceaaaa273 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -358,6 +358,62 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU So if a slot is no longer required it should be dropped. </para> </caution> + + </sect2> + + <sect2 id="logicaldecoding-replication-slots-synchronization"> + <title>Replication Slot Synchronization</title> + <para> + The logical replication slots on the primary can be synchronized to + the hot standby by using the <literal>failover</literal> parameter of + <link linkend="pg-create-logical-replication-slot"> + <function>pg_create_logical_replication_slot</function></link>, or by + using the <link linkend="sql-createsubscription-params-with-failover"> + <literal>failover</literal></link> option of + <command>CREATE SUBSCRIPTION</command> during slot creation, and then calling + <link linkend="pg-sync-replication-slots"> + <function>pg_sync_replication_slots</function></link> + on the standby. For the synchronization to work, it is mandatory to + have a physical replication slot between the primary and the standby aka + <link linkend="guc-primary-slot-name"><varname>primary_slot_name</varname></link> + should be configured on the standby, and + <link linkend="guc-hot-standby-feedback"><varname>hot_standby_feedback</varname></link> + must be enabled on the standby. It is also necessary to specify a valid + <literal>dbname</literal> in the + <link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>. + </para> + + <para> + The ability to resume logical replication after failover depends upon the + <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield> + value for the synchronized slots on the standby at the time of failover. + Only persistent slots that have attained synced state as true on the standby + before failover can be used for logical replication after failover. + Temporary synced slots cannot be used for logical decoding, therefore + logical replication for those slots cannot be resumed. For example, if the + synchronized slot could not become persistent on the standby due to a + disabled subscription, then the subscription cannot be resumed after + failover even when it is enabled. + </para> + + <para> + To resume logical replication after failover from the synced logical + slots, the subscription's 'conninfo' must be altered to point to the + new primary server. This is done using + <link linkend="sql-altersubscription-params-connection"><command>ALTER SUBSCRIPTION ... CONNECTION</command></link>. + It is recommended that subscriptions are first disabled before promoting + the standby and are re-enabled after altering the connection string. + </para> + <caution> + <para> + There is a chance that the old primary is up again during the promotion + and if subscriptions are not disabled, the logical subscribers may + continue to receive data from the old primary server even after promotion + until the connection string is altered. This might result in data + inconsistency issues, preventing the logical subscribers from being + able to continue replication from the new primary server. + </para> + </caution> </sect2> <sect2 id="logicaldecoding-explanation-output-plugins"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index ed1d62f5f8..7ffddfbd82 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -2062,7 +2062,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" <term><literal>FAILOVER [ <replaceable class="parameter">boolean</replaceable> ]</literal></term> <listitem> <para> - If true, the slot is enabled to be synced to the standbys. + If true, the slot is enabled to be synced to the standbys + so that logical replication can be resumed after failover. The default is false. </para> </listitem> @@ -2162,7 +2163,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" <term><literal>FAILOVER [ <replaceable class="parameter">boolean</replaceable> ]</literal></term> <listitem> <para> - If true, the slot is enabled to be synced to the standbys. + If true, the slot is enabled to be synced to the standbys + so that logical replication can be resumed after failover. </para> </listitem> </varlistentry> diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index dd468b31ea..be90edd0e2 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2561,10 +2561,26 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <structfield>failover</structfield> <type>bool</type> </para> <para> - True if this is a logical slot enabled to be synced to the standbys. - Always false for physical slots. + True if this is a logical slot enabled to be synced to the standbys + so that logical replication can be resumed from the new primary + after failover. Always false for physical slots. </para></entry> </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>synced</structfield> <type>bool</type> + </para> + <para> + True if this is a logical slot that was synced from a primary server. + On a hot standby, the slots with the synced column marked as true can + neither be used for logical decoding nor dropped manually. The value + of this column has no meaning on the primary server; the column value on + the primary is default false for all slots but may (if leftover from a + promoted standby) also be true. + </para></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 6791bff9dd..04227a72d1 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1024,7 +1024,8 @@ CREATE VIEW pg_replication_slots AS L.safe_wal_size, L.two_phase, L.conflict_reason, - L.failover + L.failover, + L.synced FROM pg_get_replication_slots() AS L LEFT JOIN pg_database D ON (L.datoid = D.oid); diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index 2dc25e37bb..ba03eeff1c 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -25,6 +25,7 @@ OBJS = \ proto.o \ relation.o \ reorderbuffer.o \ + slotsync.o \ snapbuild.o \ tablesync.o \ worker.o diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index ca09c683f1..a53815f2ed 100644 --- a/src/backend/replication/logical/logical.c +++ b/src/backend/replication/logical/logical.c @@ -524,6 +524,18 @@ CreateDecodingContext(XLogRecPtr start_lsn, errmsg("replication slot \"%s\" was not created in this database", NameStr(slot->data.name)))); + /* + * Do not allow consumption of a "synchronized" slot until the standby + * gets promoted. + */ + if (RecoveryInProgress() && slot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot use replication slot \"%s\" for logical decoding", + NameStr(slot->data.name)), + errdetail("This slot is being synchronized from the primary server."), + errhint("Specify another replication slot.")); + /* * Check if slot has been invalidated due to max_slot_wal_keep_size. Avoid * "cannot get changes" wording in this errmsg because that'd be diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build index 1050eb2c09..3dec36a6de 100644 --- a/src/backend/replication/logical/meson.build +++ b/src/backend/replication/logical/meson.build @@ -11,6 +11,7 @@ backend_sources += files( 'proto.c', 'relation.c', 'reorderbuffer.c', + 'slotsync.c', 'snapbuild.c', 'tablesync.c', 'worker.c', diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c new file mode 100644 index 0000000000..603c75b944 --- /dev/null +++ b/src/backend/replication/logical/slotsync.c @@ -0,0 +1,909 @@ +/*------------------------------------------------------------------------- + * slotsync.c + * Functionality for synchronizing slots to a standby server from the + * primary server. + * + * Copyright (c) 2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/slotsync.c + * + * This file contains the code for slot synchronization on a physical standby + * to fetch logical failover slots information from the primary server, create + * the slots on the standby and synchronize them. This is done by a call to SQL + * function pg_sync_replication_slots. + * + * If on physical standby, the WAL corresponding to the remote's restart_lsn + * is not available or the remote's catalog_xmin precedes the oldest xid for which + * it is guaranteed that rows wouldn't have been removed then we cannot create + * the local standby slot because that would mean moving the local slot + * backward and decoding won't be possible via such a slot. In this case, the + * slot will be marked as RS_TEMPORARY. Once the primary server catches up, + * the slot will be marked as RS_PERSISTENT (which means sync-ready) after + * which we can call pg_sync_replication_slots() periodically to perform + * syncs. + * + * Any standby synchronized slots will be dropped if they no longer need + * to be synchronized. See comment atop drop_local_obsolete_slots() for more + * details. + * + * The SQL function pg_sync_replication_slots currently does not support + * synchronization on the cascading standby. + *--------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/xlog_internal.h" +#include "access/xlogrecovery.h" +#include "catalog/pg_database.h" +#include "commands/dbcommands.h" +#include "replication/logical.h" +#include "replication/slotsync.h" +#include "storage/ipc.h" +#include "storage/lmgr.h" +#include "storage/procarray.h" +#include "utils/builtins.h" +#include "utils/pg_lsn.h" + +/* Struct for sharing information to control slot synchronization. */ +typedef struct SlotSyncCtxStruct +{ + /* prevents concurrent slot syncs to avoid slot overwrites */ + bool syncing; + slock_t mutex; +} SlotSyncCtxStruct; + +SlotSyncCtxStruct *SlotSyncCtx = NULL; + +/* + * Flag to tell if we are syncing replication slots. Unlike the 'syncing' flag + * in SlotSyncCtxStruct, this flag is true only if the current process is + * performing slot synchronization. + */ +static bool syncing_slots = false; + +/* + * Structure to hold information fetched from the primary server about a logical + * replication slot. + */ +typedef struct RemoteSlot +{ + char *name; + char *plugin; + char *database; + bool two_phase; + bool failover; + XLogRecPtr restart_lsn; + XLogRecPtr confirmed_lsn; + TransactionId catalog_xmin; + + /* RS_INVAL_NONE if valid, or the reason of invalidation */ + ReplicationSlotInvalidationCause invalidated; +} RemoteSlot; + +/* + * If necessary, update the local synced slot's metadata based on the data + * from the remote slot. + * + * If no update was needed (the data of the remote slot is the same as the + * local slot) return false, otherwise true. + */ +static bool +update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot = MyReplicationSlot; + bool xmin_changed; + bool restart_lsn_changed; + NameData plugin_name; + + Assert(slot->data.invalidated == RS_INVAL_NONE); + + xmin_changed = (remote_slot->catalog_xmin != slot->data.catalog_xmin); + restart_lsn_changed = (remote_slot->restart_lsn != slot->data.restart_lsn); + + if (!xmin_changed && + !restart_lsn_changed && + remote_dbid == slot->data.database && + remote_slot->two_phase == slot->data.two_phase && + remote_slot->failover == slot->data.failover && + remote_slot->confirmed_lsn == slot->data.confirmed_flush && + strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) + return false; + + /* Avoid expensive operations while holding a spinlock. */ + namestrcpy(&plugin_name, remote_slot->plugin); + + SpinLockAcquire(&slot->mutex); + slot->data.plugin = plugin_name; + slot->data.database = remote_dbid; + slot->data.two_phase = remote_slot->two_phase; + slot->data.failover = remote_slot->failover; + slot->data.restart_lsn = remote_slot->restart_lsn; + slot->data.confirmed_flush = remote_slot->confirmed_lsn; + slot->data.catalog_xmin = remote_slot->catalog_xmin; + slot->effective_catalog_xmin = remote_slot->catalog_xmin; + SpinLockRelease(&slot->mutex); + + if (xmin_changed) + ReplicationSlotsComputeRequiredXmin(false); + + if (restart_lsn_changed) + ReplicationSlotsComputeRequiredLSN(); + + return true; +} + +/* + * Get the list of local logical slots that are synchronized from the + * primary server. + */ +static List * +get_local_synced_slots(void) +{ + List *local_slots = NIL; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + + for (int i = 0; i < max_replication_slots; i++) + { + ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i]; + + /* Check if it is a synchronized slot */ + if (s->in_use && s->data.synced) + { + Assert(SlotIsLogical(s)); + local_slots = lappend(local_slots, s); + } + } + + LWLockRelease(ReplicationSlotControlLock); + + return local_slots; +} + +/* + * Helper function to check if local_slot is required to be retained. + * + * Return false either if local_slot does not exist in the remote_slots list + * or is invalidated while the corresponding remote slot is still valid, + * otherwise true. + */ +static bool +local_sync_slot_required(ReplicationSlot *local_slot, List *remote_slots) +{ + bool remote_exists = false; + bool locally_invalidated = false; + + foreach_ptr(RemoteSlot, remote_slot, remote_slots) + { + if (strcmp(remote_slot->name, NameStr(local_slot->data.name)) == 0) + { + remote_exists = true; + + /* + * If remote slot is not invalidated but local slot is marked as + * invalidated, then set locally_invalidated flag. + */ + SpinLockAcquire(&local_slot->mutex); + locally_invalidated = + (remote_slot->invalidated == RS_INVAL_NONE) && + (local_slot->data.invalidated != RS_INVAL_NONE); + SpinLockRelease(&local_slot->mutex); + + break; + } + } + + return (remote_exists && !locally_invalidated); +} + +/* + * Drop local obsolete slots. + * + * Drop the local slots that no longer need to be synced i.e. these either do + * not exist on the primary or are no longer enabled for failover. + * + * Additionally, drop any slots that are valid on the primary but got + * invalidated on the standby. This situation may occur due to the following + * reasons: + * - The 'max_slot_wal_keep_size' on the standby is insufficient to retain WAL + * records from the restart_lsn of the slot. + * - 'primary_slot_name' is temporarily reset to null and the physical slot is + * removed. + * These dropped slots will get recreated in next sync-cycle and it is okay to + * drop and recreate such slots as long as these are not consumable on the + * standby (which is the case currently). + * + * Note: Change of 'wal_level' on the primary server to a level lower than + * logical may also result in slot invalidation and removal on the standby. + * This is because such 'wal_level' change is only possible if the logical + * slots are removed on the primary server, so it's expected to see the + * slots being invalidated and removed on the standby too (and re-created + * if they are re-created on the primary server). + */ +static void +drop_local_obsolete_slots(List *remote_slot_list) +{ + List *local_slots = get_local_synced_slots(); + + foreach_ptr(ReplicationSlot, local_slot, local_slots) + { + /* Drop the local slot if it is not required to be retained. */ + if (!local_sync_slot_required(local_slot, remote_slot_list)) + { + bool synced_slot; + + /* + * Use shared lock to prevent a conflict with + * ReplicationSlotsDropDBSlots(), trying to drop the same slot + * during a drop-database operation. + */ + LockSharedObject(DatabaseRelationId, local_slot->data.database, + 0, AccessShareLock); + + /* + * In the small window between getting the slot to drop and + * locking the database, there is a possibility of a parallel + * database drop by the startup process and the creation of a new + * slot by the user. This new user-created slot may end up using + * the same shared memory as that of 'local_slot'. Thus check if + * local_slot is still the synced one before performing actual + * drop. + */ + SpinLockAcquire(&local_slot->mutex); + synced_slot = local_slot->in_use && local_slot->data.synced; + SpinLockRelease(&local_slot->mutex); + + if (synced_slot) + { + ReplicationSlotAcquire(NameStr(local_slot->data.name), true); + ReplicationSlotDropAcquired(); + } + + UnlockSharedObject(DatabaseRelationId, local_slot->data.database, + 0, AccessShareLock); + + ereport(LOG, + errmsg("dropped replication slot \"%s\" of dbid %d", + NameStr(local_slot->data.name), + local_slot->data.database)); + } + } +} + +/* + * Reserve WAL for the currently active local slot using the specified WAL + * location (restart_lsn). + * + * If the given WAL location has been removed, reserve WAL using the oldest + * existing WAL segment. + */ +static void +reserve_wal_for_local_slot(XLogRecPtr restart_lsn) +{ + XLogSegNo oldest_segno; + XLogSegNo segno; + ReplicationSlot *slot = MyReplicationSlot; + + Assert(slot != NULL); + Assert(XLogRecPtrIsInvalid(slot->data.restart_lsn)); + + while (true) + { + SpinLockAcquire(&slot->mutex); + slot->data.restart_lsn = restart_lsn; + SpinLockRelease(&slot->mutex); + + /* Prevent WAL removal as fast as possible */ + ReplicationSlotsComputeRequiredLSN(); + + XLByteToSeg(slot->data.restart_lsn, segno, wal_segment_size); + + /* + * Find the oldest existing WAL segment file. + * + * Normally, we can determine it by using the last removed segment + * number. However, if no WAL segment files have been removed by a + * checkpoint since startup, we need to search for the oldest segment + * file from the current timeline existing in XLOGDIR. + * + * XXX: Currently, we are searching for the oldest segment in the + * current timeline as there is less chance of the slot's restart_lsn + * from being some prior timeline, and even if it happens, in the + * worst case, we will wait to sync till the slot's restart_lsn moved + * to the current timeline. + */ + oldest_segno = XLogGetLastRemovedSegno() + 1; + + if (oldest_segno == 1) + { + TimeLineID cur_timeline; + + GetWalRcvFlushRecPtr(NULL, &cur_timeline); + oldest_segno = XLogGetOldestSegno(cur_timeline); + } + + /* + * If all required WAL is still there, great, otherwise retry. The + * slot should prevent further removal of WAL, unless there's a + * concurrent ReplicationSlotsComputeRequiredLSN() after we've written + * the new restart_lsn above, so normally we should never need to loop + * more than twice. + */ + if (segno >= oldest_segno) + break; + + /* Retry using the location of the oldest wal segment */ + XLogSegNoOffsetToRecPtr(oldest_segno, 0, wal_segment_size, restart_lsn); + } +} + +/* + * If the remote restart_lsn and catalog_xmin have caught up with the + * local ones, then update the LSNs and persist the local synced slot for + * future synchronization; otherwise, do nothing. + */ +static void +update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot = MyReplicationSlot; + + /* + * Check if the primary server has caught up. Refer to the comment atop + * the file for details on this check. + */ + if (remote_slot->restart_lsn < slot->data.restart_lsn || + TransactionIdPrecedes(remote_slot->catalog_xmin, + slot->data.catalog_xmin)) + { + /* + * The remote slot didn't catch up to locally reserved position. + * + * We do not drop the slot because the restart_lsn can be ahead of the + * current location when recreating the slot in the next cycle. It may + * take more time to create such a slot. Therefore, we keep this slot + * and attempt the synchronization in the next cycle. + */ + return; + } + + /* First time slot update, the function must return true */ + if (!update_local_synced_slot(remote_slot, remote_dbid)) + elog(ERROR, "failed to update slot"); + + ReplicationSlotPersist(); + + ereport(LOG, + errmsg("newly created slot \"%s\" is sync-ready now", + remote_slot->name)); +} + +/* + * Synchronize a single slot to the given position. + * + * This creates a new slot if there is no existing one and updates the + * metadata of the slot as per the data received from the primary server. + * + * The slot is created as a temporary slot and stays in the same state until the + * the remote_slot catches up with locally reserved position and local slot is + * updated. The slot is then persisted and is considered as sync-ready for + * periodic syncs. + */ +static void +synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot; + XLogRecPtr latestFlushPtr; + + /* + * Make sure that concerned WAL is received and flushed before syncing + * slot to target lsn received from the primary server. + */ + latestFlushPtr = GetStandbyFlushRecPtr(NULL); + if (remote_slot->confirmed_lsn > latestFlushPtr) + elog(ERROR, + "skipping slot synchronization as the received slot sync" + " LSN %X/%X for slot \"%s\" is ahead of the standby position %X/%X", + LSN_FORMAT_ARGS(remote_slot->confirmed_lsn), + remote_slot->name, + LSN_FORMAT_ARGS(latestFlushPtr)); + + /* Search for the named slot */ + if ((slot = SearchNamedReplicationSlot(remote_slot->name, true))) + { + bool synced; + + SpinLockAcquire(&slot->mutex); + synced = slot->data.synced; + SpinLockRelease(&slot->mutex); + + /* User-created slot with the same name exists, raise ERROR. */ + if (!synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("exiting from slot synchronization because same" + " name slot \"%s\" already exists on the standby", + remote_slot->name)); + + /* + * The slot has been synchronized before. + * + * It is important to acquire the slot here before checking + * invalidation. If we don't acquire the slot first, there could be a + * race condition that the local slot could be invalidated just after + * checking the 'invalidated' flag here and we could end up + * overwriting 'invalidated' flag to remote_slot's value. See + * InvalidatePossiblyObsoleteSlot() where it invalidates slot directly + * if the slot is not acquired by other processes. + */ + ReplicationSlotAcquire(remote_slot->name, true); + + Assert(slot == MyReplicationSlot); + + /* + * Copy the invalidation cause from remote only if local slot is not + * invalidated locally, we don't want to overwrite existing one. + */ + if (slot->data.invalidated == RS_INVAL_NONE && + remote_slot->invalidated != RS_INVAL_NONE) + { + SpinLockAcquire(&slot->mutex); + slot->data.invalidated = remote_slot->invalidated; + SpinLockRelease(&slot->mutex); + + /* Make sure the invalidated state persists across server restart */ + ReplicationSlotMarkDirty(); + ReplicationSlotSave(); + } + + /* Skip the sync of an invalidated slot */ + if (slot->data.invalidated != RS_INVAL_NONE) + { + ReplicationSlotRelease(); + return; + } + + /* Slot not ready yet, let's attempt to make it sync-ready now. */ + if (slot->data.persistency == RS_TEMPORARY) + { + update_and_persist_local_synced_slot(remote_slot, remote_dbid); + } + + /* Slot ready for sync, so sync it. */ + else + { + /* + * Sanity check: As long as the invalidations are handled + * appropriately as above, this should never happen. + */ + if (remote_slot->restart_lsn < slot->data.restart_lsn) + elog(ERROR, + "cannot synchronize local slot \"%s\" LSN(%X/%X)" + " to remote slot's LSN(%X/%X) as synchronization" + " would move it backwards", remote_slot->name, + LSN_FORMAT_ARGS(slot->data.restart_lsn), + LSN_FORMAT_ARGS(remote_slot->restart_lsn)); + + /* Make sure the slot changes persist across server restart */ + if (update_local_synced_slot(remote_slot, remote_dbid)) + { + ReplicationSlotMarkDirty(); + ReplicationSlotSave(); + } + } + } + /* Otherwise create the slot first. */ + else + { + NameData plugin_name; + TransactionId xmin_horizon = InvalidTransactionId; + + /* Skip creating the local slot if remote_slot is invalidated already */ + if (remote_slot->invalidated != RS_INVAL_NONE) + return; + + /* + * We create temporary slots instead of ephemeral slots here because + * we want the slots to survive after releasing them. This is done to + * avoid dropping and re-creating the slots in each synchronization + * cycle if the restart_lsn or catalog_xmin of the remote slot has not + * caught up. + */ + ReplicationSlotCreate(remote_slot->name, true, RS_TEMPORARY, + remote_slot->two_phase, + remote_slot->failover, + true); + + /* For shorter lines. */ + slot = MyReplicationSlot; + + /* Avoid expensive operations while holding a spinlock. */ + namestrcpy(&plugin_name, remote_slot->plugin); + + SpinLockAcquire(&slot->mutex); + slot->data.database = remote_dbid; + slot->data.plugin = plugin_name; + SpinLockRelease(&slot->mutex); + + reserve_wal_for_local_slot(remote_slot->restart_lsn); + + LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE); + xmin_horizon = GetOldestSafeDecodingTransactionId(true); + SpinLockAcquire(&slot->mutex); + slot->effective_catalog_xmin = xmin_horizon; + slot->data.catalog_xmin = xmin_horizon; + SpinLockRelease(&slot->mutex); + ReplicationSlotsComputeRequiredXmin(true); + LWLockRelease(ProcArrayLock); + + update_and_persist_local_synced_slot(remote_slot, remote_dbid); + } + + ReplicationSlotRelease(); +} + +/* + * Synchronize slots. + * + * Gets the failover logical slots info from the primary server and updates + * the slots locally. Creates the slots if not present on the standby. + */ +static void +synchronize_slots(WalReceiverConn *wrconn) +{ +#define SLOTSYNC_COLUMN_COUNT 9 + Oid slotRow[SLOTSYNC_COLUMN_COUNT] = {TEXTOID, TEXTOID, LSNOID, + LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID}; + + WalRcvExecResult *res; + TupleTableSlot *tupslot; + StringInfoData s; + List *remote_slot_list = NIL; + + SpinLockAcquire(&SlotSyncCtx->mutex); + if (SlotSyncCtx->syncing) + { + SpinLockRelease(&SlotSyncCtx->mutex); + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot synchronize replication slots concurrently")); + } + + SlotSyncCtx->syncing = true; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = true; + + initStringInfo(&s); + + /* Construct query to fetch slots with failover enabled. */ + appendStringInfo(&s, + "SELECT slot_name, plugin, confirmed_flush_lsn," + " restart_lsn, catalog_xmin, two_phase, failover," + " database, conflict_reason" + " FROM pg_catalog.pg_replication_slots" + " WHERE failover and NOT temporary"); + + /* Execute the query */ + res = walrcv_exec(wrconn, s.data, SLOTSYNC_COLUMN_COUNT, slotRow); + pfree(s.data); + + if (res->status != WALRCV_OK_TUPLES) + ereport(ERROR, + errmsg("could not fetch failover logical slots info from the primary server: %s", + res->err)); + + /* Construct the remote_slot tuple and synchronize each slot locally */ + tupslot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple); + while (tuplestore_gettupleslot(res->tuplestore, true, false, tupslot)) + { + bool isnull; + RemoteSlot *remote_slot = palloc0(sizeof(RemoteSlot)); + Datum d; + int col = 0; + + remote_slot->name = TextDatumGetCString(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->plugin = TextDatumGetCString(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + /* + * It is possible to get null values for LSN and Xmin if slot is + * invalidated on the primary server, so handle accordingly. + */ + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->confirmed_lsn = isnull ? InvalidXLogRecPtr : + DatumGetLSN(d); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->restart_lsn = isnull ? InvalidXLogRecPtr : DatumGetLSN(d); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->catalog_xmin = isnull ? InvalidTransactionId : + DatumGetTransactionId(d); + + remote_slot->two_phase = DatumGetBool(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->failover = DatumGetBool(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->database = TextDatumGetCString(slot_getattr(tupslot, + ++col, &isnull)); + Assert(!isnull); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->invalidated = isnull ? RS_INVAL_NONE : + GetSlotInvalidationCause(TextDatumGetCString(d)); + + /* Sanity check */ + Assert(col == SLOTSYNC_COLUMN_COUNT); + + /* + * If restart_lsn, confirmed_lsn or catalog_xmin is invalid but the + * slot is valid, that means we have fetched the remote_slot in its + * RS_EPHEMERAL state. In such a case, don't sync it; we can always + * sync it in the next sync cycle when the remote_slot is persisted + * and has valid lsn(s) and xmin values. + * + * XXX: In future, if we plan to expose 'slot->data.persistency' in + * pg_replication_slots view, then we can avoid fetching RS_EPHEMERAL + * slots in the first place. + */ + if ((XLogRecPtrIsInvalid(remote_slot->restart_lsn) || + XLogRecPtrIsInvalid(remote_slot->confirmed_lsn) || + !TransactionIdIsValid(remote_slot->catalog_xmin)) && + remote_slot->invalidated == RS_INVAL_NONE) + pfree(remote_slot); + else + /* Create list of remote slots */ + remote_slot_list = lappend(remote_slot_list, remote_slot); + + ExecClearTuple(tupslot); + } + + /* Drop local slots that no longer need to be synced. */ + drop_local_obsolete_slots(remote_slot_list); + + /* Now sync the slots locally */ + foreach_ptr(RemoteSlot, remote_slot, remote_slot_list) + { + Oid remote_dbid = get_database_oid(remote_slot->database, false); + + /* + * Use shared lock to prevent a conflict with + * ReplicationSlotsDropDBSlots(), trying to drop the same slot during + * a drop-database operation. + */ + LockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); + + synchronize_one_slot(remote_slot, remote_dbid); + + UnlockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); + } + + /* We are done, free remote_slot_list elements */ + list_free_deep(remote_slot_list); + + walrcv_clear_result(res); + + SpinLockAcquire(&SlotSyncCtx->mutex); + SlotSyncCtx->syncing = false; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = false; +} + +/* + * Checks the remote server info. + * + * We ensure that the 'primary_slot_name' exists on the remote server and the + * remote server is not a standby node. + */ +static void +validate_remote_info(WalReceiverConn *wrconn) +{ +#define PRIMARY_INFO_OUTPUT_COL_COUNT 2 + WalRcvExecResult *res; + Oid slotRow[PRIMARY_INFO_OUTPUT_COL_COUNT] = {BOOLOID, BOOLOID}; + StringInfoData cmd; + bool isnull; + TupleTableSlot *tupslot; + bool remote_in_recovery; + bool primary_slot_valid; + + initStringInfo(&cmd); + appendStringInfo(&cmd, + "SELECT pg_is_in_recovery(), count(*) = 1" + " FROM pg_catalog.pg_replication_slots" + " WHERE slot_type='physical' AND slot_name=%s", + quote_literal_cstr(PrimarySlotName)); + + res = walrcv_exec(wrconn, cmd.data, PRIMARY_INFO_OUTPUT_COL_COUNT, slotRow); + pfree(cmd.data); + + if (res->status != WALRCV_OK_TUPLES) + ereport(ERROR, + errmsg("could not fetch primary_slot_name \"%s\" info from the primary server: %s", + PrimarySlotName, res->err), + errhint("Check if \"primary_slot_name\" is configured correctly.")); + + tupslot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple); + if (!tuplestore_gettupleslot(res->tuplestore, true, false, tupslot)) + elog(ERROR, + "failed to fetch tuple for the primary server slot specified by \"primary_slot_name\""); + + remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull)); + Assert(!isnull); + + if (remote_in_recovery) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot synchronize replication slots from a standby server")); + + primary_slot_valid = DatumGetBool(slot_getattr(tupslot, 2, &isnull)); + Assert(!isnull); + + if (!primary_slot_valid) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + /* translator: second %s is a GUC variable name */ + errdetail("The replication slot \"%s\" specified by \"%s\" does not exist on the primary server.", + PrimarySlotName, "primary_slot_name")); + + ExecClearTuple(tupslot); + walrcv_clear_result(res); +} + +/* + * Check all necessary GUCs for slot synchronization are set + * appropriately, otherwise, raise ERROR. + */ +void +ValidateSlotSyncParams(void) +{ + char *dbname; + + /* + * A physical replication slot(primary_slot_name) is required on the + * primary to ensure that the rows needed by the standby are not removed + * after restarting, so that the synchronized slot on the standby will not + * be invalidated. + */ + if (PrimarySlotName == NULL || *PrimarySlotName == '\0') + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be defined.", "primary_slot_name")); + + /* + * hot_standby_feedback must be enabled to cooperate with the physical + * replication slot, which allows informing the primary about the xmin and + * catalog_xmin values on the standby. + */ + if (!hot_standby_feedback) + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be enabled.", "hot_standby_feedback")); + + /* Logical slot sync/creation requires wal_level >= logical. */ + if (wal_level < WAL_LEVEL_LOGICAL) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"wal_level\" must be >= logical.")); + + /* + * The primary_conninfo is required to make connection to primary for + * getting slots information. + */ + if (PrimaryConnInfo == NULL || *PrimaryConnInfo == '\0') + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be defined.", "primary_conninfo")); + + /* + * The slot synchronization needs a database connection for walrcv_exec to + * work. + */ + dbname = walrcv_get_dbname_from_conninfo(PrimaryConnInfo); + if (dbname == NULL) + ereport(ERROR, + + /* + * translator: 'dbname' is a specific option; %s is a GUC variable + * name + */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("'dbname' must be specified in \"%s\".", "primary_conninfo")); +} + +/* + * Is current process syncing replication slots ? + */ +bool +IsSyncingReplicationSlots(void) +{ + return syncing_slots; +} + +/* + * Amount of shared memory required for slot synchronization. + */ +Size +SlotSyncShmemSize(void) +{ + return sizeof(SlotSyncCtxStruct); +} + +/* + * Allocate and initialize the shared memory of slot synchronization. + */ +void +SlotSyncShmemInit(void) +{ + bool found; + + SlotSyncCtx = (SlotSyncCtxStruct *) + ShmemInitStruct("Slot Sync Data", SlotSyncShmemSize(), &found); + + if (!found) + { + SlotSyncCtx->syncing = false; + SpinLockInit(&SlotSyncCtx->mutex); + } +} + +/* + * Error cleanup callback for slot synchronization. + */ +static void +slotsync_failure_callback(int code, Datum arg) +{ + WalReceiverConn *wrconn = (WalReceiverConn *) DatumGetPointer(arg); + + if (syncing_slots) + { + /* + * If syncing_slots is true, it indicates that the process errored out + * without resetting the flag. So, we need to clean up shared memory + * and reset the flag here. + */ + SpinLockAcquire(&SlotSyncCtx->mutex); + SlotSyncCtx->syncing = false; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = false; + } + + walrcv_disconnect(wrconn); +} + +/* + * Synchronize the failover enabled replication slots using the specified + * primary server connection. + */ +void +SyncReplicationSlots(WalReceiverConn *wrconn) +{ + PG_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); + { + validate_remote_info(wrconn); + + synchronize_slots(wrconn); + } + PG_END_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); +} diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index fd4e96c9d6..3864dd95af 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -46,6 +46,7 @@ #include "common/string.h" #include "miscadmin.h" #include "pgstat.h" +#include "replication/slotsync.h" #include "replication/slot.h" #include "storage/fd.h" #include "storage/ipc.h" @@ -103,7 +104,6 @@ int max_replication_slots = 10; /* the maximum number of replication * slots */ static void ReplicationSlotShmemExit(int code, Datum arg); -static void ReplicationSlotDropAcquired(void); static void ReplicationSlotDropPtr(ReplicationSlot *slot); /* internal persistency functions */ @@ -250,11 +250,12 @@ ReplicationSlotValidateName(const char *name, int elevel) * user will only get commit prepared. * failover: If enabled, allows the slot to be synced to standbys so * that logical replication can be resumed after failover. + * synced: True if the slot is synchronized from the primary server. */ void ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotPersistency persistency, - bool two_phase, bool failover) + bool two_phase, bool failover, bool synced) { ReplicationSlot *slot = NULL; int i; @@ -263,6 +264,34 @@ ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotValidateName(name, ERROR); + if (failover) + { + /* + * Do not allow users to create the failover enabled slots on the + * standby as we do not support sync to the cascading standby. + * + * However, failover enabled slots can be created during slot + * synchronization because we need to retain the same values as the + * remote slot. + */ + if (RecoveryInProgress() && !IsSyncingReplicationSlots()) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a replication slot created on the standby")); + + /* + * Do not allow users to create failover enabled temporary slots, + * because temporary slots will not be synced to the standby. + * + * However, failover enabled temporary slots can be created during + * slot synchronization. See the comments atop slotsync.c for details. + */ + if (persistency == RS_TEMPORARY && !IsSyncingReplicationSlots()) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a temporary replication slot")); + } + /* * If some other backend ran this code concurrently with us, we'd likely * both allocate the same slot, and that would be bad. We'd also be at @@ -315,6 +344,7 @@ ReplicationSlotCreate(const char *name, bool db_specific, slot->data.two_phase = two_phase; slot->data.two_phase_at = InvalidXLogRecPtr; slot->data.failover = failover; + slot->data.synced = synced; /* and then data only present in shared memory */ slot->just_dirtied = false; @@ -677,6 +707,16 @@ ReplicationSlotDrop(const char *name, bool nowait) ReplicationSlotAcquire(name, nowait); + /* + * Do not allow users to drop the slots which are currently being synced + * from the primary to the standby. + */ + if (RecoveryInProgress() && MyReplicationSlot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot drop replication slot \"%s\"", name), + errdetail("This slot is being synced from the primary server.")); + ReplicationSlotDropAcquired(); } @@ -696,6 +736,38 @@ ReplicationSlotAlter(const char *name, bool failover) errmsg("cannot use %s with a physical replication slot", "ALTER_REPLICATION_SLOT")); + if (RecoveryInProgress()) + { + /* + * Do not allow users to alter the slots which are currently being + * synced from the primary to the standby. + */ + if (MyReplicationSlot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot alter replication slot \"%s\"", name), + errdetail("This slot is being synced from the primary server.")); + + /* + * Do not allow users to enable failover on the standby as we do not + * support sync to the cascading standby. + */ + if (failover) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a replication slot" + " on the standby")); + } + + /* + * Do not allow users to enable failover for temporary slots as we do not + * support syncing temporary slots to the standby. + */ + if (failover && MyReplicationSlot->data.persistency == RS_TEMPORARY) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a temporary replication slot")); + if (MyReplicationSlot->data.failover != failover) { SpinLockAcquire(&MyReplicationSlot->mutex); @@ -712,7 +784,7 @@ ReplicationSlotAlter(const char *name, bool failover) /* * Permanently drop the currently acquired replication slot. */ -static void +void ReplicationSlotDropAcquired(void) { ReplicationSlot *slot = MyReplicationSlot; @@ -868,8 +940,8 @@ ReplicationSlotMarkDirty(void) } /* - * Convert a slot that's marked as RS_EPHEMERAL to a RS_PERSISTENT slot, - * guaranteeing it will be there after an eventual crash. + * Convert a slot that's marked as RS_EPHEMERAL or RS_TEMPORARY to a + * RS_PERSISTENT slot, guaranteeing it will be there after an eventual crash. */ void ReplicationSlotPersist(void) @@ -2189,3 +2261,25 @@ RestoreSlotFromDisk(const char *name) (errmsg("too many replication slots active before shutdown"), errhint("Increase max_replication_slots and try again."))); } + +/* + * Maps the pg_replication_slots.conflict_reason text value to + * ReplicationSlotInvalidationCause enum value + */ +ReplicationSlotInvalidationCause +GetSlotInvalidationCause(char *conflict_reason) +{ + Assert(conflict_reason); + + if (strcmp(conflict_reason, SLOT_INVAL_WAL_REMOVED_TEXT) == 0) + return RS_INVAL_WAL_REMOVED; + else if (strcmp(conflict_reason, SLOT_INVAL_HORIZON_TEXT) == 0) + return RS_INVAL_HORIZON; + else if (strcmp(conflict_reason, SLOT_INVAL_WAL_LEVEL_TEXT) == 0) + return RS_INVAL_WAL_LEVEL; + else + Assert(0); + + /* Keep compiler quiet */ + return RS_INVAL_NONE; +} diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index eb685089b3..d2fa5e669a 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -21,7 +21,9 @@ #include "replication/decode.h" #include "replication/logical.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "utils/builtins.h" +#include "utils/guc.h" #include "utils/inval.h" #include "utils/pg_lsn.h" #include "utils/resowner.h" @@ -43,7 +45,7 @@ create_physical_replication_slot(char *name, bool immediately_reserve, /* acquire replication slot, this will check for conflicting names */ ReplicationSlotCreate(name, false, temporary ? RS_TEMPORARY : RS_PERSISTENT, false, - false); + false, false); if (immediately_reserve) { @@ -136,7 +138,7 @@ create_logical_replication_slot(char *name, char *plugin, */ ReplicationSlotCreate(name, true, temporary ? RS_TEMPORARY : RS_EPHEMERAL, two_phase, - failover); + failover, false); /* * Create logical decoding context to find start point or, if we don't @@ -237,7 +239,7 @@ pg_drop_replication_slot(PG_FUNCTION_ARGS) Datum pg_get_replication_slots(PG_FUNCTION_ARGS) { -#define PG_GET_REPLICATION_SLOTS_COLS 16 +#define PG_GET_REPLICATION_SLOTS_COLS 17 ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; XLogRecPtr currlsn; int slotno; @@ -418,21 +420,23 @@ pg_get_replication_slots(PG_FUNCTION_ARGS) break; case RS_INVAL_WAL_REMOVED: - values[i++] = CStringGetTextDatum("wal_removed"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_WAL_REMOVED_TEXT); break; case RS_INVAL_HORIZON: - values[i++] = CStringGetTextDatum("rows_removed"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_HORIZON_TEXT); break; case RS_INVAL_WAL_LEVEL: - values[i++] = CStringGetTextDatum("wal_level_insufficient"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_WAL_LEVEL_TEXT); break; } } values[i++] = BoolGetDatum(slot_contents.data.failover); + values[i++] = BoolGetDatum(slot_contents.data.synced); + Assert(i == PG_GET_REPLICATION_SLOTS_COLS); tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, @@ -700,7 +704,6 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) XLogRecPtr src_restart_lsn; bool src_islogical; bool temporary; - bool failover; char *plugin; Datum values[2]; bool nulls[2]; @@ -756,7 +759,6 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) src_islogical = SlotIsLogical(&first_slot_contents); src_restart_lsn = first_slot_contents.data.restart_lsn; temporary = (first_slot_contents.data.persistency == RS_TEMPORARY); - failover = first_slot_contents.data.failover; plugin = logical_slot ? NameStr(first_slot_contents.data.plugin) : NULL; /* Check type of replication slot */ @@ -791,12 +793,20 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) * We must not try to read WAL, since we haven't reserved it yet -- * hence pass find_startpoint false. confirmed_flush will be set * below, by copying from the source slot. + * + * To avoid potential issues with the slot synchronization where the + * restart_lsn of a replication slot can go backward, we set the + * failover option to false here. This situation occurs when a slot + * on the primary server is dropped and immediately replaced with a + * new slot of the same name, created by copying from another existing + * slot. However, the slot synchronization will only observe the + * restart_lsn of the same slot going backward. */ create_logical_replication_slot(NameStr(*dst_name), plugin, temporary, false, - failover, + false, src_restart_lsn, false); } @@ -943,3 +953,49 @@ pg_copy_physical_replication_slot_b(PG_FUNCTION_ARGS) { return copy_replication_slot(fcinfo, false); } + +/* + * Synchronize failover enabled replication slots to a standby server + * from the primary server. + */ +Datum +pg_sync_replication_slots(PG_FUNCTION_ARGS) +{ + WalReceiverConn *wrconn; + char *err; + StringInfoData app_name; + + CheckSlotPermissions(); + + if (!RecoveryInProgress()) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("replication slots can only be synchronized to a standby server")); + + /* Load the libpq-specific functions */ + load_file("libpqwalreceiver", false); + + ValidateSlotSyncParams(); + + initStringInfo(&app_name); + if (cluster_name[0]) + appendStringInfo(&app_name, "%s_slotsync", cluster_name); + else + appendStringInfoString(&app_name, "slotsync"); + + /* Connect to the primary server. */ + wrconn = walrcv_connect(PrimaryConnInfo, false, false, false, + app_name.data, &err); + pfree(app_name.data); + + if (!wrconn) + ereport(ERROR, + errcode(ERRCODE_CONNECTION_FAILURE), + errmsg("could not connect to the primary server: %s", err)); + + SyncReplicationSlots(wrconn); + + walrcv_disconnect(wrconn); + + PG_RETURN_VOID(); +} diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 146826d5db..4e54779a9e 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -72,6 +72,7 @@ #include "postmaster/interrupt.h" #include "replication/decode.h" #include "replication/logical.h" +#include "replication/slotsync.h" #include "replication/slot.h" #include "replication/snapbuild.h" #include "replication/syncrep.h" @@ -243,7 +244,6 @@ static void WalSndShutdown(void) pg_attribute_noreturn(); static void XLogSendPhysical(void); static void XLogSendLogical(void); static void WalSndDone(WalSndSendDataCallback send_data); -static XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); static void IdentifySystem(void); static void UploadManifest(void); static bool HandleUploadManifestPacket(StringInfo buf, off_t *offset, @@ -1224,7 +1224,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) { ReplicationSlotCreate(cmd->slotname, false, cmd->temporary ? RS_TEMPORARY : RS_PERSISTENT, - false, false); + false, false, false); if (reserve_wal) { @@ -1255,7 +1255,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) */ ReplicationSlotCreate(cmd->slotname, true, cmd->temporary ? RS_TEMPORARY : RS_EPHEMERAL, - two_phase, failover); + two_phase, failover, false); /* * Do options check early so that we can bail before calling the @@ -3385,14 +3385,17 @@ WalSndDone(WalSndSendDataCallback send_data) } /* - * Returns the latest point in WAL that has been safely flushed to disk, and - * can be sent to the standby. This should only be called when in recovery, - * ie. we're streaming to a cascaded standby. + * Returns the latest point in WAL that has been safely flushed to disk. + * This should only be called when in recovery. + * + * This is called either by cascading walsender to find WAL postion to be sent + * to a cascaded standby or by slot synchronization function to validate remote + * slot's lsn before syncing it locally. * * As a side-effect, *tli is updated to the TLI of the last * replayed WAL record. */ -static XLogRecPtr +XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli) { XLogRecPtr replayPtr; @@ -3401,6 +3404,8 @@ GetStandbyFlushRecPtr(TimeLineID *tli) TimeLineID receiveTLI; XLogRecPtr result; + Assert(am_cascading_walsender || IsSyncingReplicationSlots()); + /* * We can safely send what's already been replayed. Also, if walreceiver * is streaming WAL from the same timeline, we can send anything that it diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index 7084e18861..7e7941d625 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -36,6 +36,7 @@ #include "replication/logicallauncher.h" #include "replication/origin.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/walreceiver.h" #include "replication/walsender.h" #include "storage/bufmgr.h" @@ -153,6 +154,7 @@ CalculateShmemSize(int *num_semaphores) size = add_size(size, StatsShmemSize()); size = add_size(size, WaitEventExtensionShmemSize()); size = add_size(size, InjectionPointShmemSize()); + size = add_size(size, SlotSyncShmemSize()); #ifdef EXEC_BACKEND size = add_size(size, ShmemBackendArraySize()); #endif @@ -347,6 +349,7 @@ CreateOrAttachShmemStructs(void) WalSummarizerShmemInit(); PgArchShmemInit(); ApplyLauncherShmemInit(); + SlotSyncShmemInit(); /* * Set up other modules that need some shared memory space diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 29af4ce65d..9c120fc2b7 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11127,9 +11127,9 @@ proname => 'pg_get_replication_slots', prorows => '10', proisstrict => 'f', proretset => 't', provolatile => 's', prorettype => 'record', proargtypes => '', - proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,text,bool}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,conflict_reason,failover}', + proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,text,bool,bool}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,conflict_reason,failover,synced}', prosrc => 'pg_get_replication_slots' }, { oid => '3786', descr => 'set up a logical replication slot', proname => 'pg_create_logical_replication_slot', provolatile => 'v', @@ -11212,6 +11212,10 @@ proname => 'pg_logical_emit_message', provolatile => 'v', proparallel => 'u', prorettype => 'pg_lsn', proargtypes => 'bool text bytea bool', prosrc => 'pg_logical_emit_message_bytea' }, +{ oid => '9929', descr => 'sync replication slots from the primary to the standby', + proname => 'pg_sync_replication_slots', provolatile => 'v', proparallel => 'u', + prorettype => 'void', proargtypes => '', + prosrc => 'pg_sync_replication_slots' }, # event triggers { oid => '3566', descr => 'list objects dropped by the current command', diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index da4c776492..e706ca834c 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -52,6 +52,14 @@ typedef enum ReplicationSlotInvalidationCause RS_INVAL_WAL_LEVEL, } ReplicationSlotInvalidationCause; +/* + * The possible values for 'conflict_reason' returned in + * pg_get_replication_slots. + */ +#define SLOT_INVAL_WAL_REMOVED_TEXT "wal_removed" +#define SLOT_INVAL_HORIZON_TEXT "rows_removed" +#define SLOT_INVAL_WAL_LEVEL_TEXT "wal_level_insufficient" + /* * On-Disk data of a replication slot, preserved across restarts. */ @@ -112,6 +120,11 @@ typedef struct ReplicationSlotPersistentData /* plugin name */ NameData plugin; + /* + * Was this slot synchronized from the primary server? + */ + char synced; + /* * Is this a failover slot (sync candidate for standbys)? Only relevant * for logical slots on the primary server. @@ -224,9 +237,11 @@ extern void ReplicationSlotsShmemInit(void); /* management of individual slots */ extern void ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotPersistency persistency, - bool two_phase, bool failover); + bool two_phase, bool failover, + bool synced); extern void ReplicationSlotPersist(void); extern void ReplicationSlotDrop(const char *name, bool nowait); +extern void ReplicationSlotDropAcquired(void); extern void ReplicationSlotAlter(const char *name, bool failover); extern void ReplicationSlotAcquire(const char *name, bool nowait); @@ -259,5 +274,7 @@ extern void CheckPointReplicationSlots(bool is_shutdown); extern void CheckSlotRequirements(void); extern void CheckSlotPermissions(void); +extern ReplicationSlotInvalidationCause + GetSlotInvalidationCause(char *conflict_reason); #endif /* SLOT_H */ diff --git a/src/include/replication/slotsync.h b/src/include/replication/slotsync.h new file mode 100644 index 0000000000..e86d8a47b8 --- /dev/null +++ b/src/include/replication/slotsync.h @@ -0,0 +1,23 @@ +/*------------------------------------------------------------------------- + * + * slotsync.h + * Exports for slot synchronization. + * + * Portions Copyright (c) 2016-2024, PostgreSQL Global Development Group + * + * src/include/replication/slotsync.h + * + *------------------------------------------------------------------------- + */ +#ifndef SLOTSYNC_H +#define SLOTSYNC_H + +#include "replication/walreceiver.h" + +extern void ValidateSlotSyncParams(void); +extern bool IsSyncingReplicationSlots(void); +extern Size SlotSyncShmemSize(void); +extern void SlotSyncShmemInit(void); +extern void SyncReplicationSlots(WalReceiverConn *wrconn); + +#endif /* SLOTSYNC_H */ diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h index 1b58d50b3b..0c3996e926 100644 --- a/src/include/replication/walsender.h +++ b/src/include/replication/walsender.h @@ -12,6 +12,8 @@ #ifndef _WALSENDER_H #define _WALSENDER_H +#include "access/xlogdefs.h" + /* * What to do with a snapshot in create replication slot command. */ @@ -37,6 +39,7 @@ extern void InitWalSender(void); extern bool exec_replication_command(const char *cmd_string); extern void WalSndErrorCleanup(void); extern void WalSndResourceCleanup(bool isCommit); +extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); extern void WalSndSignals(void); extern Size WalSndShmemSize(void); extern void WalSndShmemInit(void); diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index bc58ff4cab..1735290b4b 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -97,4 +97,197 @@ my ($result, $stdout, $stderr) = $subscriber1->psql('postgres', ok( $stderr =~ /ERROR: cannot set failover for enabled subscription/, "altering failover is not allowed for enabled subscription"); +################################################## +# Test that pg_sync_replication_slots() cannot be executed on a non-standby server. +################################################## + +($result, $stdout, $stderr) = + $publisher->psql('postgres', "SELECT pg_sync_replication_slots();"); +ok( $stderr =~ + /ERROR: replication slots can only be synchronized to a standby server/, + "cannot sync slots on a non-standby server"); + +################################################## +# Test logical failover slots on the standby +# Configure standby1 to replicate and synchronize logical slots configured +# for failover on the primary +# +# failover slot lsub1_slot ->| ----> subscriber1 (connected via logical replication) +# failover slot lsub2_slot | inactive +# primary ---> | +# physical slot sb1_slot --->| ----> standby1 (connected via streaming replication) +# | lsub1_slot, lsub2_slot (synced_slot) +################################################## + +my $primary = $publisher; +my $backup_name = 'backup'; +$primary->backup($backup_name); + +# Create a standby +my $standby1 = PostgreSQL::Test::Cluster->new('standby1'); +$standby1->init_from_backup( + $primary, $backup_name, + has_streaming => 1, + has_restoring => 1); + +my $connstr_1 = $primary->connstr; +$standby1->append_conf( + 'postgresql.conf', qq( +hot_standby_feedback = on +primary_slot_name = 'sb1_slot' +primary_conninfo = '$connstr_1 dbname=postgres' +)); + +$primary->psql('postgres', + q{SELECT pg_create_logical_replication_slot('lsub2_slot', 'test_decoding', false, false, true);} +); + +$primary->psql('postgres', + q{SELECT pg_create_physical_replication_slot('sb1_slot');}); + +my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; +my $offset = -s $standby1->logfile; + +# Start the standby so that slot syncing can begin +$standby1->start; + +$primary->wait_for_catchup('regress_mysub1'); + +# Do not allow any further advancement of the restart_lsn for the lsub1_slot. +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); + +# Wait for the replication slot to become inactive on the publisher +$primary->poll_query_until( + 'postgres', + "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", + 1); + +# Wait for the standby to catch up so that the standby is not lagging behind +# the subscriber. +$primary->wait_for_replay_catchup($standby1); + +# Synchronize the primary server slots to the standby. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Confirm that the logical failover slots are created on the standby and are +# flagged as 'synced' +is( $standby1->safe_psql( + 'postgres', + q{SELECT count(*) = 2 FROM pg_replication_slots WHERE slot_name IN ('lsub1_slot', 'lsub2_slot') AND synced;} + ), + "t", + 'logical slots have synced as true on standby'); + +################################################## +# Test that the synchronized slot will be dropped if the corresponding remote +# slot on the primary server has been dropped. +################################################## + +$primary->psql('postgres', "SELECT pg_drop_replication_slot('lsub2_slot');"); + +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +is( $standby1->safe_psql( + 'postgres', + q{SELECT count(*) = 0 FROM pg_replication_slots WHERE slot_name = 'lsub2_slot';} + ), + "t", + 'synchronized slot has been dropped'); + +################################################## +# Test that if the synchronized slot is invalidated while the remote slot is +# still valid, the slot will be dropped and re-created on the standby by +# executing pg_sync_replication_slots() again. +################################################## + +# Configure the max_slot_wal_keep_size so that the synced slot can be +# invalidated due to wal removal. +$standby1->append_conf('postgresql.conf', 'max_slot_wal_keep_size = 64kB'); +$standby1->reload; + +# Generate some activity and switch WAL file on the primary +$primary->advance_wal(1); +$primary->psql('postgres', "CHECKPOINT"); +$primary->wait_for_replay_catchup($standby1); + +# Request a checkpoint on the standby to trigger the WAL file(s) removal +$standby1->safe_psql('postgres', "CHECKPOINT"); + +# Check if the synced slot is invalidated +is( $standby1->safe_psql( + 'postgres', + q{SELECT conflict_reason = 'wal_removed' FROM pg_replication_slots WHERE slot_name = 'lsub1_slot';} + ), + "t", + 'synchronized slot has been invalidated'); + +# Reset max_slot_wal_keep_size to avoid further wal removal +$standby1->append_conf('postgresql.conf', 'max_slot_wal_keep_size = -1'); +$standby1->reload; + +# Enable the subscription to let it catch up to the latest wal position +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 ENABLE"); + +$primary->wait_for_catchup('regress_mysub1'); + +# Do not allow any further advancement of the restart_lsn for the lsub1_slot. +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); + +# Wait for the replication slot to become inactive on the publisher +$primary->poll_query_until( + 'postgres', + "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", + 1); + +# Wait for the standby to catch up so that the standby is not lagging behind +# the subscriber. +$primary->wait_for_replay_catchup($standby1); + +my $log_offset = -s $standby1->logfile; + +# Synchronize the primary server slots to the standby. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Confirm that the invalidated slot has been dropped. +$standby1->wait_for_log(qr/dropped replication slot "lsub1_slot" of dbid [0-9]+/, + $log_offset); + +# Confirm that the logical slot has been re-created on the standby and is +# flagged as 'synced' +is( $standby1->safe_psql( + 'postgres', + q{SELECT conflict_reason IS NULL AND synced FROM pg_replication_slots WHERE slot_name = 'lsub1_slot';} + ), + "t", + 'logical slot is re-synced'); + +################################################## +# Test that a synchronized slot can not be decoded, altered or dropped by the +# user +################################################## + +# Attempting to perform logical decoding on a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql('postgres', + "select * from pg_logical_slot_get_changes('lsub1_slot', NULL, NULL);"); +ok( $stderr =~ + /ERROR: cannot use replication slot "lsub1_slot" for logical decoding/, + "logical decoding is not allowed on synced slot"); + +# Attempting to alter a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql( + 'postgres', + qq[ALTER_REPLICATION_SLOT lsub1_slot (failover);], + replication => 'database'); +ok($stderr =~ /ERROR: cannot alter replication slot "lsub1_slot"/, + "synced slot on standby cannot be altered"); + +# Attempting to drop a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql('postgres', + "SELECT pg_drop_replication_slot('lsub1_slot');"); +ok($stderr =~ /ERROR: cannot drop replication slot "lsub1_slot"/, + "synced slot on standby cannot be dropped"); + done_testing(); diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index abc944e8b8..b7488d760e 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1474,8 +1474,9 @@ pg_replication_slots| SELECT l.slot_name, l.safe_wal_size, l.two_phase, l.conflict_reason, - l.failover - FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, conflict_reason, failover) + l.failover, + l.synced + FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, conflict_reason, failover, synced) LEFT JOIN pg_database d ON ((l.datoid = d.oid))); pg_roles| SELECT pg_authid.rolname, pg_authid.rolsuper, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 91433d439b..d808aad8b0 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2325,6 +2325,7 @@ RelocationBufferInfo RelptrFreePageBtree RelptrFreePageManager RelptrFreePageSpanLeader +RemoteSlot RenameStmt ReopenPtrType ReorderBuffer @@ -2584,6 +2585,7 @@ SlabBlock SlabContext SlabSlot SlotNumber +SlotSyncCtxStruct SlruCtl SlruCtlData SlruErrorCause -- 2.30.0.windows.2 ^ permalink raw reply [nested|flat] 46+ messages in thread
* RE: Synchronizing slots from primary to standby @ 2024-02-13 12:39 Zhijie Hou (Fujitsu) <[email protected]> parent: Bertrand Drouvot <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: Zhijie Hou (Fujitsu) @ 2024-02-13 12:39 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Tuesday, February 13, 2024 7:30 PM Bertrand Drouvot <[email protected]> wrote: > > On Tue, Feb 13, 2024 at 04:08:23AM +0000, Zhijie Hou (Fujitsu) wrote: > > On Tuesday, February 13, 2024 9:16 AM Zhijie Hou (Fujitsu) > <[email protected]> wrote: > > > > > > Here is the new version patch which addressed above and most of > > > Bertrand's comments. > > > > > > TODO: trying to add one test for the case the slot is valid on > > > primary while the synced slots is invalidated on the standby. > > > > Here is the V85_2 patch set that added the test and fixed one typo, > > there are no other code changes. > > Thanks! > > Out of curiosity I ran a code coverage and the result for slotsync.c can be found > in [1]. > > It appears that: > > - only one function is not covered (slotsync_failure_callback()). Thanks for the test ! I think slotsync_failure_callback can be covered easier in the next slotsync worker patch on worker exit, I will post that after rebasing. Best Regards, Hou zj ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-13 15:55 Bertrand Drouvot <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Bertrand Drouvot @ 2024-02-13 15:55 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> Hi, On Tue, Feb 13, 2024 at 05:20:35PM +0530, Amit Kapila wrote: > On Tue, Feb 13, 2024 at 4:59 PM Bertrand Drouvot > <[email protected]> wrote: > > - 84% of the slotsync.c code is covered, the parts that are not are mainly > > related to "errors". > > > > Worth to try to extend the coverage? (I've in mind 731, 739, 766, 778, 786, 796, > > 808) > > > > All these additional line numbers mentioned by you are ERROR paths. I > think if we want we can easily cover most of those but I am not sure > if there is a benefit to cover each error path. Yeah, I think 731, 739 and one among the remaining ones mentioned up-thread should be enough, thoughts? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-14 02:39 Amit Kapila <[email protected]> parent: Bertrand Drouvot <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Amit Kapila @ 2024-02-14 02:39 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Tue, Feb 13, 2024 at 9:25 PM Bertrand Drouvot <[email protected]> wrote: > > On Tue, Feb 13, 2024 at 05:20:35PM +0530, Amit Kapila wrote: > > On Tue, Feb 13, 2024 at 4:59 PM Bertrand Drouvot > > <[email protected]> wrote: > > > - 84% of the slotsync.c code is covered, the parts that are not are mainly > > > related to "errors". > > > > > > Worth to try to extend the coverage? (I've in mind 731, 739, 766, 778, 786, 796, > > > 808) > > > > > > > All these additional line numbers mentioned by you are ERROR paths. I > > think if we want we can easily cover most of those but I am not sure > > if there is a benefit to cover each error path. > > Yeah, I think 731, 739 and one among the remaining ones mentioned up-thread should > be enough, thoughts? > I don't know how beneficial those selective ones would be but if I have to pick a few among those then I would pick the ones at 731 and 808. The reason is that 731 is related to cascading standby restriction which we may uplift in the future and at that time one needs to be careful about the behavior, for 808 as well, in the future, we may have a separate GUC for slot_db_name. These may not be good enough reasons as to why we add tests for these ERROR cases but not for others, however, if we have to randomly pick a few among all ERROR paths, these seem better to me than others. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 46+ messages in thread
* RE: Synchronizing slots from primary to standby @ 2024-02-14 04:03 Zhijie Hou (Fujitsu) <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Zhijie Hou (Fujitsu) @ 2024-02-14 04:03 UTC (permalink / raw) To: Amit Kapila <[email protected]>; Bertrand Drouvot <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Wednesday, February 14, 2024 10:40 AM Amit Kapila <[email protected]> wrote: > > On Tue, Feb 13, 2024 at 9:25 PM Bertrand Drouvot > <[email protected]> wrote: > > > > On Tue, Feb 13, 2024 at 05:20:35PM +0530, Amit Kapila wrote: > > > On Tue, Feb 13, 2024 at 4:59 PM Bertrand Drouvot > > > <[email protected]> wrote: > > > > - 84% of the slotsync.c code is covered, the parts that are not > > > > are mainly related to "errors". > > > > > > > > Worth to try to extend the coverage? (I've in mind 731, 739, 766, > > > > 778, 786, 796, > > > > 808) > > > > > > > > > > All these additional line numbers mentioned by you are ERROR paths. > > > I think if we want we can easily cover most of those but I am not > > > sure if there is a benefit to cover each error path. > > > > Yeah, I think 731, 739 and one among the remaining ones mentioned > > up-thread should be enough, thoughts? > > > > I don't know how beneficial those selective ones would be but if I have to pick a > few among those then I would pick the ones at 731 and 808. The reason is that > 731 is related to cascading standby restriction which we may uplift in the future > and at that time one needs to be careful about the behavior, for 808 as well, in > the future, we may have a separate GUC for slot_db_name. These may not be > good enough reasons as to why we add tests for these ERROR cases but not for > others, however, if we have to randomly pick a few among all ERROR paths, > these seem better to me than others. Here is V87 patch that adds test for the suggested cases. Best Regards, Hou zj Attachments: [application/octet-stream] v87-0001-Add-a-slot-synchronization-function.patch (77.8K, ../../OS0PR01MB57162E9C2935E47A33D2A566944E2@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v87-0001-Add-a-slot-synchronization-function.patch) download | inline diff: From b46bb868d808a6238b91b8ad508d7058b3f62393 Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Wed, 7 Feb 2024 10:37:31 +0530 Subject: [PATCH v872] Add a slot synchronization function. This commit introduces a new SQL function pg_sync_replication_slots() which is used to synchronize the logical replication slots from the primary server to the physical standby so that logical replication can be resumed after failover. A new 'synced' flag is introduced in pg_replication_slots view, indicating whether the slot has been synchronized from the primary server. On a standby, synced slots cannot be dropped or consumed, and any attempt to perform logical decoding on them will result in an error. The logical replication slots on the primary can be synchronized to the hot standby by enabling failover during slot creation (e.g. using the "failover" parameter of pg_create_logical_replication_slot(), or using the "failover" option of the CREATE SUBSCRIPTION command), and then calling pg_sync_replication_slots() function on the standby. For the synchronization to work, it is mandatory to have a physical replication slot between the primary and the standby, 'hot_standby_feedback' must be enabled on the standby and a valid dbname must be specified in 'primary_conninfo'. If a logical slot is invalidated on the primary, then that slot on the standby is also invalidated. If a logical slot on the primary is valid but is invalidated on the standby, then that slot is dropped but will be recreated on the standby in next pg_sync_replication_slots() call provided the slot still exists on the primary server. It is okay to recreate such slots as long as these are not consumable on the standby (which is the case currently). This situation may occur due to the following reasons: - The 'max_slot_wal_keep_size' on the standby is insufficient to retain WAL records from the restart_lsn of the slot. - 'primary_slot_name' is temporarily reset to null and the physical slot is removed. We may also see the slots invalidated and dropped on the standby if the primary changes 'wal_level' to a level lower than logical. Changing the primary 'wal_level' to a level lower than logical is only possible if the logical slots are removed on the primary server, so it's expected to see the slots being removed on the standby too (and re-created if they are re-created on the primary server). The slots synchronization status on the standby can be monitored using 'synced' column of pg_replication_slots view. --- .../test_decoding/expected/permissions.out | 3 + contrib/test_decoding/expected/slot.out | 2 + contrib/test_decoding/sql/permissions.sql | 1 + contrib/test_decoding/sql/slot.sql | 1 + doc/src/sgml/config.sgml | 9 +- doc/src/sgml/func.sgml | 35 +- doc/src/sgml/logicaldecoding.sgml | 56 ++ doc/src/sgml/protocol.sgml | 6 +- doc/src/sgml/system-views.sgml | 20 +- src/backend/catalog/system_views.sql | 3 +- src/backend/replication/logical/Makefile | 1 + src/backend/replication/logical/logical.c | 12 + src/backend/replication/logical/meson.build | 1 + src/backend/replication/logical/slotsync.c | 909 ++++++++++++++++++ src/backend/replication/slot.c | 104 +- src/backend/replication/slotfuncs.c | 74 +- src/backend/replication/walsender.c | 19 +- src/backend/storage/ipc/ipci.c | 3 + src/include/catalog/pg_proc.dat | 10 +- src/include/replication/slot.h | 19 +- src/include/replication/slotsync.h | 23 + src/include/replication/walsender.h | 3 + .../t/040_standby_failover_slots_sync.pl | 237 +++++ src/test/regress/expected/rules.out | 5 +- src/tools/pgindent/typedefs.list | 2 + 25 files changed, 1523 insertions(+), 35 deletions(-) create mode 100644 src/backend/replication/logical/slotsync.c create mode 100644 src/include/replication/slotsync.h diff --git a/contrib/test_decoding/expected/permissions.out b/contrib/test_decoding/expected/permissions.out index d6eaba8c55..8d100646ce 100644 --- a/contrib/test_decoding/expected/permissions.out +++ b/contrib/test_decoding/expected/permissions.out @@ -64,6 +64,9 @@ DETAIL: Only roles with the REPLICATION attribute may use replication slots. SELECT pg_drop_replication_slot('regression_slot'); ERROR: permission denied to use replication slots DETAIL: Only roles with the REPLICATION attribute may use replication slots. +SELECT pg_sync_replication_slots(); +ERROR: permission denied to use replication slots +DETAIL: Only roles with the REPLICATION attribute may use replication slots. RESET ROLE; -- replication users can drop superuser created slots SET ROLE regress_lr_superuser; diff --git a/contrib/test_decoding/expected/slot.out b/contrib/test_decoding/expected/slot.out index 261d8886d3..349ab2d380 100644 --- a/contrib/test_decoding/expected/slot.out +++ b/contrib/test_decoding/expected/slot.out @@ -425,6 +425,8 @@ SELECT 'init' FROM pg_create_logical_replication_slot('failover_default_slot', ' init (1 row) +SELECT 'init' FROM pg_create_logical_replication_slot('failover_true_temp_slot', 'test_decoding', true, false, true); +ERROR: cannot enable failover for a temporary replication slot SELECT 'init' FROM pg_create_physical_replication_slot('physical_slot'); ?column? ---------- diff --git a/contrib/test_decoding/sql/permissions.sql b/contrib/test_decoding/sql/permissions.sql index 312b514593..94db936aee 100644 --- a/contrib/test_decoding/sql/permissions.sql +++ b/contrib/test_decoding/sql/permissions.sql @@ -29,6 +29,7 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_d INSERT INTO lr_test VALUES('lr_superuser_init'); SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); SELECT pg_drop_replication_slot('regression_slot'); +SELECT pg_sync_replication_slots(); RESET ROLE; -- replication users can drop superuser created slots diff --git a/contrib/test_decoding/sql/slot.sql b/contrib/test_decoding/sql/slot.sql index 45aeae7fd5..580e3ae3be 100644 --- a/contrib/test_decoding/sql/slot.sql +++ b/contrib/test_decoding/sql/slot.sql @@ -181,6 +181,7 @@ SELECT pg_drop_replication_slot('copied_slot2_notemp'); SELECT 'init' FROM pg_create_logical_replication_slot('failover_true_slot', 'test_decoding', false, false, true); SELECT 'init' FROM pg_create_logical_replication_slot('failover_false_slot', 'test_decoding', false, false, false); SELECT 'init' FROM pg_create_logical_replication_slot('failover_default_slot', 'test_decoding', false, false); +SELECT 'init' FROM pg_create_logical_replication_slot('failover_true_temp_slot', 'test_decoding', true, false, true); SELECT 'init' FROM pg_create_physical_replication_slot('physical_slot'); SELECT slot_name, slot_type, failover FROM pg_replication_slots; diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 61038472c5..037a3b8a64 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4612,8 +4612,13 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" <varname>primary_conninfo</varname> string, or in a separate <filename>~/.pgpass</filename> file on the standby server (use <literal>replication</literal> as the database name). - Do not specify a database name in the - <varname>primary_conninfo</varname> string. + </para> + <para> + For replication slot synchronization (see + <xref linkend="logicaldecoding-replication-slots-synchronization"/>), + it is also necessary to specify a valid <literal>dbname</literal> + in the <varname>primary_conninfo</varname> string. This will only be + used for slot synchronization. It is ignored for streaming. </para> <para> This parameter can only be set in the <filename>postgresql.conf</filename> diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 11d537b341..8f147a2417 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -28075,7 +28075,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset </row> <row> - <entry role="func_table_entry"><para role="func_signature"> + <entry id="pg-create-logical-replication-slot" role="func_table_entry"><para role="func_signature"> <indexterm> <primary>pg_create_logical_replication_slot</primary> </indexterm> @@ -28444,6 +28444,39 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset record is flushed along with its transaction. </para></entry> </row> + + <row> + <entry id="pg-sync-replication-slots" role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>pg_sync_replication_slots</primary> + </indexterm> + <function>pg_sync_replication_slots</function> () + <returnvalue>void</returnvalue> + </para> + <para> + Synchronize the logical failover replication slots from the primary + server to the standby server. This function can only be executed on the + standby server. Temporary synced slots, if any, cannot be used for + logical decoding and must be dropped after promotion. See + <xref linkend="logicaldecoding-replication-slots-synchronization"/> for details. + </para> + + <caution> + <para> + If, after executing the function, + <link linkend="guc-hot-standby-feedback"> + <varname>hot_standby_feedback</varname></link> is disabled on + the standby or the physical slot configured in + <link linkend="guc-primary-slot-name"> + <varname>primary_slot_name</varname></link> is + removed, then it is possible that the necessary rows of the + synchronized slot will be removed by the VACUUM process on the primary + server, resulting in the synchronized slot becoming invalidated. + </para> + </caution> + </entry> + </row> + </tbody> </tgroup> </table> diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index cd152d4ced..eceaaaa273 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -358,6 +358,62 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU So if a slot is no longer required it should be dropped. </para> </caution> + + </sect2> + + <sect2 id="logicaldecoding-replication-slots-synchronization"> + <title>Replication Slot Synchronization</title> + <para> + The logical replication slots on the primary can be synchronized to + the hot standby by using the <literal>failover</literal> parameter of + <link linkend="pg-create-logical-replication-slot"> + <function>pg_create_logical_replication_slot</function></link>, or by + using the <link linkend="sql-createsubscription-params-with-failover"> + <literal>failover</literal></link> option of + <command>CREATE SUBSCRIPTION</command> during slot creation, and then calling + <link linkend="pg-sync-replication-slots"> + <function>pg_sync_replication_slots</function></link> + on the standby. For the synchronization to work, it is mandatory to + have a physical replication slot between the primary and the standby aka + <link linkend="guc-primary-slot-name"><varname>primary_slot_name</varname></link> + should be configured on the standby, and + <link linkend="guc-hot-standby-feedback"><varname>hot_standby_feedback</varname></link> + must be enabled on the standby. It is also necessary to specify a valid + <literal>dbname</literal> in the + <link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>. + </para> + + <para> + The ability to resume logical replication after failover depends upon the + <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield> + value for the synchronized slots on the standby at the time of failover. + Only persistent slots that have attained synced state as true on the standby + before failover can be used for logical replication after failover. + Temporary synced slots cannot be used for logical decoding, therefore + logical replication for those slots cannot be resumed. For example, if the + synchronized slot could not become persistent on the standby due to a + disabled subscription, then the subscription cannot be resumed after + failover even when it is enabled. + </para> + + <para> + To resume logical replication after failover from the synced logical + slots, the subscription's 'conninfo' must be altered to point to the + new primary server. This is done using + <link linkend="sql-altersubscription-params-connection"><command>ALTER SUBSCRIPTION ... CONNECTION</command></link>. + It is recommended that subscriptions are first disabled before promoting + the standby and are re-enabled after altering the connection string. + </para> + <caution> + <para> + There is a chance that the old primary is up again during the promotion + and if subscriptions are not disabled, the logical subscribers may + continue to receive data from the old primary server even after promotion + until the connection string is altered. This might result in data + inconsistency issues, preventing the logical subscribers from being + able to continue replication from the new primary server. + </para> + </caution> </sect2> <sect2 id="logicaldecoding-explanation-output-plugins"> diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index 05d6cc42da..a5cb19357f 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -2062,7 +2062,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" <term><literal>FAILOVER [ <replaceable class="parameter">boolean</replaceable> ]</literal></term> <listitem> <para> - If true, the slot is enabled to be synced to the standbys. + If true, the slot is enabled to be synced to the standbys + so that logical replication can be resumed after failover. The default is false. </para> </listitem> @@ -2162,7 +2163,8 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" <term><literal>FAILOVER [ <replaceable class="parameter">boolean</replaceable> ]</literal></term> <listitem> <para> - If true, the slot is enabled to be synced to the standbys. + If true, the slot is enabled to be synced to the standbys + so that logical replication can be resumed after failover. </para> </listitem> </varlistentry> diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index dd468b31ea..be90edd0e2 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2561,10 +2561,26 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <structfield>failover</structfield> <type>bool</type> </para> <para> - True if this is a logical slot enabled to be synced to the standbys. - Always false for physical slots. + True if this is a logical slot enabled to be synced to the standbys + so that logical replication can be resumed from the new primary + after failover. Always false for physical slots. </para></entry> </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>synced</structfield> <type>bool</type> + </para> + <para> + True if this is a logical slot that was synced from a primary server. + On a hot standby, the slots with the synced column marked as true can + neither be used for logical decoding nor dropped manually. The value + of this column has no meaning on the primary server; the column value on + the primary is default false for all slots but may (if leftover from a + promoted standby) also be true. + </para></entry> + </row> + </tbody> </tgroup> </table> diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 6791bff9dd..04227a72d1 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1024,7 +1024,8 @@ CREATE VIEW pg_replication_slots AS L.safe_wal_size, L.two_phase, L.conflict_reason, - L.failover + L.failover, + L.synced FROM pg_get_replication_slots() AS L LEFT JOIN pg_database D ON (L.datoid = D.oid); diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile index 2dc25e37bb..ba03eeff1c 100644 --- a/src/backend/replication/logical/Makefile +++ b/src/backend/replication/logical/Makefile @@ -25,6 +25,7 @@ OBJS = \ proto.o \ relation.o \ reorderbuffer.o \ + slotsync.o \ snapbuild.o \ tablesync.o \ worker.o diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index ca09c683f1..a53815f2ed 100644 --- a/src/backend/replication/logical/logical.c +++ b/src/backend/replication/logical/logical.c @@ -524,6 +524,18 @@ CreateDecodingContext(XLogRecPtr start_lsn, errmsg("replication slot \"%s\" was not created in this database", NameStr(slot->data.name)))); + /* + * Do not allow consumption of a "synchronized" slot until the standby + * gets promoted. + */ + if (RecoveryInProgress() && slot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot use replication slot \"%s\" for logical decoding", + NameStr(slot->data.name)), + errdetail("This slot is being synchronized from the primary server."), + errhint("Specify another replication slot.")); + /* * Check if slot has been invalidated due to max_slot_wal_keep_size. Avoid * "cannot get changes" wording in this errmsg because that'd be diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build index 1050eb2c09..3dec36a6de 100644 --- a/src/backend/replication/logical/meson.build +++ b/src/backend/replication/logical/meson.build @@ -11,6 +11,7 @@ backend_sources += files( 'proto.c', 'relation.c', 'reorderbuffer.c', + 'slotsync.c', 'snapbuild.c', 'tablesync.c', 'worker.c', diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c new file mode 100644 index 0000000000..603c75b944 --- /dev/null +++ b/src/backend/replication/logical/slotsync.c @@ -0,0 +1,909 @@ +/*------------------------------------------------------------------------- + * slotsync.c + * Functionality for synchronizing slots to a standby server from the + * primary server. + * + * Copyright (c) 2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/backend/replication/logical/slotsync.c + * + * This file contains the code for slot synchronization on a physical standby + * to fetch logical failover slots information from the primary server, create + * the slots on the standby and synchronize them. This is done by a call to SQL + * function pg_sync_replication_slots. + * + * If on physical standby, the WAL corresponding to the remote's restart_lsn + * is not available or the remote's catalog_xmin precedes the oldest xid for which + * it is guaranteed that rows wouldn't have been removed then we cannot create + * the local standby slot because that would mean moving the local slot + * backward and decoding won't be possible via such a slot. In this case, the + * slot will be marked as RS_TEMPORARY. Once the primary server catches up, + * the slot will be marked as RS_PERSISTENT (which means sync-ready) after + * which we can call pg_sync_replication_slots() periodically to perform + * syncs. + * + * Any standby synchronized slots will be dropped if they no longer need + * to be synchronized. See comment atop drop_local_obsolete_slots() for more + * details. + * + * The SQL function pg_sync_replication_slots currently does not support + * synchronization on the cascading standby. + *--------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "access/xlog_internal.h" +#include "access/xlogrecovery.h" +#include "catalog/pg_database.h" +#include "commands/dbcommands.h" +#include "replication/logical.h" +#include "replication/slotsync.h" +#include "storage/ipc.h" +#include "storage/lmgr.h" +#include "storage/procarray.h" +#include "utils/builtins.h" +#include "utils/pg_lsn.h" + +/* Struct for sharing information to control slot synchronization. */ +typedef struct SlotSyncCtxStruct +{ + /* prevents concurrent slot syncs to avoid slot overwrites */ + bool syncing; + slock_t mutex; +} SlotSyncCtxStruct; + +SlotSyncCtxStruct *SlotSyncCtx = NULL; + +/* + * Flag to tell if we are syncing replication slots. Unlike the 'syncing' flag + * in SlotSyncCtxStruct, this flag is true only if the current process is + * performing slot synchronization. + */ +static bool syncing_slots = false; + +/* + * Structure to hold information fetched from the primary server about a logical + * replication slot. + */ +typedef struct RemoteSlot +{ + char *name; + char *plugin; + char *database; + bool two_phase; + bool failover; + XLogRecPtr restart_lsn; + XLogRecPtr confirmed_lsn; + TransactionId catalog_xmin; + + /* RS_INVAL_NONE if valid, or the reason of invalidation */ + ReplicationSlotInvalidationCause invalidated; +} RemoteSlot; + +/* + * If necessary, update the local synced slot's metadata based on the data + * from the remote slot. + * + * If no update was needed (the data of the remote slot is the same as the + * local slot) return false, otherwise true. + */ +static bool +update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot = MyReplicationSlot; + bool xmin_changed; + bool restart_lsn_changed; + NameData plugin_name; + + Assert(slot->data.invalidated == RS_INVAL_NONE); + + xmin_changed = (remote_slot->catalog_xmin != slot->data.catalog_xmin); + restart_lsn_changed = (remote_slot->restart_lsn != slot->data.restart_lsn); + + if (!xmin_changed && + !restart_lsn_changed && + remote_dbid == slot->data.database && + remote_slot->two_phase == slot->data.two_phase && + remote_slot->failover == slot->data.failover && + remote_slot->confirmed_lsn == slot->data.confirmed_flush && + strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) + return false; + + /* Avoid expensive operations while holding a spinlock. */ + namestrcpy(&plugin_name, remote_slot->plugin); + + SpinLockAcquire(&slot->mutex); + slot->data.plugin = plugin_name; + slot->data.database = remote_dbid; + slot->data.two_phase = remote_slot->two_phase; + slot->data.failover = remote_slot->failover; + slot->data.restart_lsn = remote_slot->restart_lsn; + slot->data.confirmed_flush = remote_slot->confirmed_lsn; + slot->data.catalog_xmin = remote_slot->catalog_xmin; + slot->effective_catalog_xmin = remote_slot->catalog_xmin; + SpinLockRelease(&slot->mutex); + + if (xmin_changed) + ReplicationSlotsComputeRequiredXmin(false); + + if (restart_lsn_changed) + ReplicationSlotsComputeRequiredLSN(); + + return true; +} + +/* + * Get the list of local logical slots that are synchronized from the + * primary server. + */ +static List * +get_local_synced_slots(void) +{ + List *local_slots = NIL; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + + for (int i = 0; i < max_replication_slots; i++) + { + ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i]; + + /* Check if it is a synchronized slot */ + if (s->in_use && s->data.synced) + { + Assert(SlotIsLogical(s)); + local_slots = lappend(local_slots, s); + } + } + + LWLockRelease(ReplicationSlotControlLock); + + return local_slots; +} + +/* + * Helper function to check if local_slot is required to be retained. + * + * Return false either if local_slot does not exist in the remote_slots list + * or is invalidated while the corresponding remote slot is still valid, + * otherwise true. + */ +static bool +local_sync_slot_required(ReplicationSlot *local_slot, List *remote_slots) +{ + bool remote_exists = false; + bool locally_invalidated = false; + + foreach_ptr(RemoteSlot, remote_slot, remote_slots) + { + if (strcmp(remote_slot->name, NameStr(local_slot->data.name)) == 0) + { + remote_exists = true; + + /* + * If remote slot is not invalidated but local slot is marked as + * invalidated, then set locally_invalidated flag. + */ + SpinLockAcquire(&local_slot->mutex); + locally_invalidated = + (remote_slot->invalidated == RS_INVAL_NONE) && + (local_slot->data.invalidated != RS_INVAL_NONE); + SpinLockRelease(&local_slot->mutex); + + break; + } + } + + return (remote_exists && !locally_invalidated); +} + +/* + * Drop local obsolete slots. + * + * Drop the local slots that no longer need to be synced i.e. these either do + * not exist on the primary or are no longer enabled for failover. + * + * Additionally, drop any slots that are valid on the primary but got + * invalidated on the standby. This situation may occur due to the following + * reasons: + * - The 'max_slot_wal_keep_size' on the standby is insufficient to retain WAL + * records from the restart_lsn of the slot. + * - 'primary_slot_name' is temporarily reset to null and the physical slot is + * removed. + * These dropped slots will get recreated in next sync-cycle and it is okay to + * drop and recreate such slots as long as these are not consumable on the + * standby (which is the case currently). + * + * Note: Change of 'wal_level' on the primary server to a level lower than + * logical may also result in slot invalidation and removal on the standby. + * This is because such 'wal_level' change is only possible if the logical + * slots are removed on the primary server, so it's expected to see the + * slots being invalidated and removed on the standby too (and re-created + * if they are re-created on the primary server). + */ +static void +drop_local_obsolete_slots(List *remote_slot_list) +{ + List *local_slots = get_local_synced_slots(); + + foreach_ptr(ReplicationSlot, local_slot, local_slots) + { + /* Drop the local slot if it is not required to be retained. */ + if (!local_sync_slot_required(local_slot, remote_slot_list)) + { + bool synced_slot; + + /* + * Use shared lock to prevent a conflict with + * ReplicationSlotsDropDBSlots(), trying to drop the same slot + * during a drop-database operation. + */ + LockSharedObject(DatabaseRelationId, local_slot->data.database, + 0, AccessShareLock); + + /* + * In the small window between getting the slot to drop and + * locking the database, there is a possibility of a parallel + * database drop by the startup process and the creation of a new + * slot by the user. This new user-created slot may end up using + * the same shared memory as that of 'local_slot'. Thus check if + * local_slot is still the synced one before performing actual + * drop. + */ + SpinLockAcquire(&local_slot->mutex); + synced_slot = local_slot->in_use && local_slot->data.synced; + SpinLockRelease(&local_slot->mutex); + + if (synced_slot) + { + ReplicationSlotAcquire(NameStr(local_slot->data.name), true); + ReplicationSlotDropAcquired(); + } + + UnlockSharedObject(DatabaseRelationId, local_slot->data.database, + 0, AccessShareLock); + + ereport(LOG, + errmsg("dropped replication slot \"%s\" of dbid %d", + NameStr(local_slot->data.name), + local_slot->data.database)); + } + } +} + +/* + * Reserve WAL for the currently active local slot using the specified WAL + * location (restart_lsn). + * + * If the given WAL location has been removed, reserve WAL using the oldest + * existing WAL segment. + */ +static void +reserve_wal_for_local_slot(XLogRecPtr restart_lsn) +{ + XLogSegNo oldest_segno; + XLogSegNo segno; + ReplicationSlot *slot = MyReplicationSlot; + + Assert(slot != NULL); + Assert(XLogRecPtrIsInvalid(slot->data.restart_lsn)); + + while (true) + { + SpinLockAcquire(&slot->mutex); + slot->data.restart_lsn = restart_lsn; + SpinLockRelease(&slot->mutex); + + /* Prevent WAL removal as fast as possible */ + ReplicationSlotsComputeRequiredLSN(); + + XLByteToSeg(slot->data.restart_lsn, segno, wal_segment_size); + + /* + * Find the oldest existing WAL segment file. + * + * Normally, we can determine it by using the last removed segment + * number. However, if no WAL segment files have been removed by a + * checkpoint since startup, we need to search for the oldest segment + * file from the current timeline existing in XLOGDIR. + * + * XXX: Currently, we are searching for the oldest segment in the + * current timeline as there is less chance of the slot's restart_lsn + * from being some prior timeline, and even if it happens, in the + * worst case, we will wait to sync till the slot's restart_lsn moved + * to the current timeline. + */ + oldest_segno = XLogGetLastRemovedSegno() + 1; + + if (oldest_segno == 1) + { + TimeLineID cur_timeline; + + GetWalRcvFlushRecPtr(NULL, &cur_timeline); + oldest_segno = XLogGetOldestSegno(cur_timeline); + } + + /* + * If all required WAL is still there, great, otherwise retry. The + * slot should prevent further removal of WAL, unless there's a + * concurrent ReplicationSlotsComputeRequiredLSN() after we've written + * the new restart_lsn above, so normally we should never need to loop + * more than twice. + */ + if (segno >= oldest_segno) + break; + + /* Retry using the location of the oldest wal segment */ + XLogSegNoOffsetToRecPtr(oldest_segno, 0, wal_segment_size, restart_lsn); + } +} + +/* + * If the remote restart_lsn and catalog_xmin have caught up with the + * local ones, then update the LSNs and persist the local synced slot for + * future synchronization; otherwise, do nothing. + */ +static void +update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot = MyReplicationSlot; + + /* + * Check if the primary server has caught up. Refer to the comment atop + * the file for details on this check. + */ + if (remote_slot->restart_lsn < slot->data.restart_lsn || + TransactionIdPrecedes(remote_slot->catalog_xmin, + slot->data.catalog_xmin)) + { + /* + * The remote slot didn't catch up to locally reserved position. + * + * We do not drop the slot because the restart_lsn can be ahead of the + * current location when recreating the slot in the next cycle. It may + * take more time to create such a slot. Therefore, we keep this slot + * and attempt the synchronization in the next cycle. + */ + return; + } + + /* First time slot update, the function must return true */ + if (!update_local_synced_slot(remote_slot, remote_dbid)) + elog(ERROR, "failed to update slot"); + + ReplicationSlotPersist(); + + ereport(LOG, + errmsg("newly created slot \"%s\" is sync-ready now", + remote_slot->name)); +} + +/* + * Synchronize a single slot to the given position. + * + * This creates a new slot if there is no existing one and updates the + * metadata of the slot as per the data received from the primary server. + * + * The slot is created as a temporary slot and stays in the same state until the + * the remote_slot catches up with locally reserved position and local slot is + * updated. The slot is then persisted and is considered as sync-ready for + * periodic syncs. + */ +static void +synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) +{ + ReplicationSlot *slot; + XLogRecPtr latestFlushPtr; + + /* + * Make sure that concerned WAL is received and flushed before syncing + * slot to target lsn received from the primary server. + */ + latestFlushPtr = GetStandbyFlushRecPtr(NULL); + if (remote_slot->confirmed_lsn > latestFlushPtr) + elog(ERROR, + "skipping slot synchronization as the received slot sync" + " LSN %X/%X for slot \"%s\" is ahead of the standby position %X/%X", + LSN_FORMAT_ARGS(remote_slot->confirmed_lsn), + remote_slot->name, + LSN_FORMAT_ARGS(latestFlushPtr)); + + /* Search for the named slot */ + if ((slot = SearchNamedReplicationSlot(remote_slot->name, true))) + { + bool synced; + + SpinLockAcquire(&slot->mutex); + synced = slot->data.synced; + SpinLockRelease(&slot->mutex); + + /* User-created slot with the same name exists, raise ERROR. */ + if (!synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("exiting from slot synchronization because same" + " name slot \"%s\" already exists on the standby", + remote_slot->name)); + + /* + * The slot has been synchronized before. + * + * It is important to acquire the slot here before checking + * invalidation. If we don't acquire the slot first, there could be a + * race condition that the local slot could be invalidated just after + * checking the 'invalidated' flag here and we could end up + * overwriting 'invalidated' flag to remote_slot's value. See + * InvalidatePossiblyObsoleteSlot() where it invalidates slot directly + * if the slot is not acquired by other processes. + */ + ReplicationSlotAcquire(remote_slot->name, true); + + Assert(slot == MyReplicationSlot); + + /* + * Copy the invalidation cause from remote only if local slot is not + * invalidated locally, we don't want to overwrite existing one. + */ + if (slot->data.invalidated == RS_INVAL_NONE && + remote_slot->invalidated != RS_INVAL_NONE) + { + SpinLockAcquire(&slot->mutex); + slot->data.invalidated = remote_slot->invalidated; + SpinLockRelease(&slot->mutex); + + /* Make sure the invalidated state persists across server restart */ + ReplicationSlotMarkDirty(); + ReplicationSlotSave(); + } + + /* Skip the sync of an invalidated slot */ + if (slot->data.invalidated != RS_INVAL_NONE) + { + ReplicationSlotRelease(); + return; + } + + /* Slot not ready yet, let's attempt to make it sync-ready now. */ + if (slot->data.persistency == RS_TEMPORARY) + { + update_and_persist_local_synced_slot(remote_slot, remote_dbid); + } + + /* Slot ready for sync, so sync it. */ + else + { + /* + * Sanity check: As long as the invalidations are handled + * appropriately as above, this should never happen. + */ + if (remote_slot->restart_lsn < slot->data.restart_lsn) + elog(ERROR, + "cannot synchronize local slot \"%s\" LSN(%X/%X)" + " to remote slot's LSN(%X/%X) as synchronization" + " would move it backwards", remote_slot->name, + LSN_FORMAT_ARGS(slot->data.restart_lsn), + LSN_FORMAT_ARGS(remote_slot->restart_lsn)); + + /* Make sure the slot changes persist across server restart */ + if (update_local_synced_slot(remote_slot, remote_dbid)) + { + ReplicationSlotMarkDirty(); + ReplicationSlotSave(); + } + } + } + /* Otherwise create the slot first. */ + else + { + NameData plugin_name; + TransactionId xmin_horizon = InvalidTransactionId; + + /* Skip creating the local slot if remote_slot is invalidated already */ + if (remote_slot->invalidated != RS_INVAL_NONE) + return; + + /* + * We create temporary slots instead of ephemeral slots here because + * we want the slots to survive after releasing them. This is done to + * avoid dropping and re-creating the slots in each synchronization + * cycle if the restart_lsn or catalog_xmin of the remote slot has not + * caught up. + */ + ReplicationSlotCreate(remote_slot->name, true, RS_TEMPORARY, + remote_slot->two_phase, + remote_slot->failover, + true); + + /* For shorter lines. */ + slot = MyReplicationSlot; + + /* Avoid expensive operations while holding a spinlock. */ + namestrcpy(&plugin_name, remote_slot->plugin); + + SpinLockAcquire(&slot->mutex); + slot->data.database = remote_dbid; + slot->data.plugin = plugin_name; + SpinLockRelease(&slot->mutex); + + reserve_wal_for_local_slot(remote_slot->restart_lsn); + + LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE); + xmin_horizon = GetOldestSafeDecodingTransactionId(true); + SpinLockAcquire(&slot->mutex); + slot->effective_catalog_xmin = xmin_horizon; + slot->data.catalog_xmin = xmin_horizon; + SpinLockRelease(&slot->mutex); + ReplicationSlotsComputeRequiredXmin(true); + LWLockRelease(ProcArrayLock); + + update_and_persist_local_synced_slot(remote_slot, remote_dbid); + } + + ReplicationSlotRelease(); +} + +/* + * Synchronize slots. + * + * Gets the failover logical slots info from the primary server and updates + * the slots locally. Creates the slots if not present on the standby. + */ +static void +synchronize_slots(WalReceiverConn *wrconn) +{ +#define SLOTSYNC_COLUMN_COUNT 9 + Oid slotRow[SLOTSYNC_COLUMN_COUNT] = {TEXTOID, TEXTOID, LSNOID, + LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID}; + + WalRcvExecResult *res; + TupleTableSlot *tupslot; + StringInfoData s; + List *remote_slot_list = NIL; + + SpinLockAcquire(&SlotSyncCtx->mutex); + if (SlotSyncCtx->syncing) + { + SpinLockRelease(&SlotSyncCtx->mutex); + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot synchronize replication slots concurrently")); + } + + SlotSyncCtx->syncing = true; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = true; + + initStringInfo(&s); + + /* Construct query to fetch slots with failover enabled. */ + appendStringInfo(&s, + "SELECT slot_name, plugin, confirmed_flush_lsn," + " restart_lsn, catalog_xmin, two_phase, failover," + " database, conflict_reason" + " FROM pg_catalog.pg_replication_slots" + " WHERE failover and NOT temporary"); + + /* Execute the query */ + res = walrcv_exec(wrconn, s.data, SLOTSYNC_COLUMN_COUNT, slotRow); + pfree(s.data); + + if (res->status != WALRCV_OK_TUPLES) + ereport(ERROR, + errmsg("could not fetch failover logical slots info from the primary server: %s", + res->err)); + + /* Construct the remote_slot tuple and synchronize each slot locally */ + tupslot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple); + while (tuplestore_gettupleslot(res->tuplestore, true, false, tupslot)) + { + bool isnull; + RemoteSlot *remote_slot = palloc0(sizeof(RemoteSlot)); + Datum d; + int col = 0; + + remote_slot->name = TextDatumGetCString(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->plugin = TextDatumGetCString(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + /* + * It is possible to get null values for LSN and Xmin if slot is + * invalidated on the primary server, so handle accordingly. + */ + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->confirmed_lsn = isnull ? InvalidXLogRecPtr : + DatumGetLSN(d); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->restart_lsn = isnull ? InvalidXLogRecPtr : DatumGetLSN(d); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->catalog_xmin = isnull ? InvalidTransactionId : + DatumGetTransactionId(d); + + remote_slot->two_phase = DatumGetBool(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->failover = DatumGetBool(slot_getattr(tupslot, ++col, + &isnull)); + Assert(!isnull); + + remote_slot->database = TextDatumGetCString(slot_getattr(tupslot, + ++col, &isnull)); + Assert(!isnull); + + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->invalidated = isnull ? RS_INVAL_NONE : + GetSlotInvalidationCause(TextDatumGetCString(d)); + + /* Sanity check */ + Assert(col == SLOTSYNC_COLUMN_COUNT); + + /* + * If restart_lsn, confirmed_lsn or catalog_xmin is invalid but the + * slot is valid, that means we have fetched the remote_slot in its + * RS_EPHEMERAL state. In such a case, don't sync it; we can always + * sync it in the next sync cycle when the remote_slot is persisted + * and has valid lsn(s) and xmin values. + * + * XXX: In future, if we plan to expose 'slot->data.persistency' in + * pg_replication_slots view, then we can avoid fetching RS_EPHEMERAL + * slots in the first place. + */ + if ((XLogRecPtrIsInvalid(remote_slot->restart_lsn) || + XLogRecPtrIsInvalid(remote_slot->confirmed_lsn) || + !TransactionIdIsValid(remote_slot->catalog_xmin)) && + remote_slot->invalidated == RS_INVAL_NONE) + pfree(remote_slot); + else + /* Create list of remote slots */ + remote_slot_list = lappend(remote_slot_list, remote_slot); + + ExecClearTuple(tupslot); + } + + /* Drop local slots that no longer need to be synced. */ + drop_local_obsolete_slots(remote_slot_list); + + /* Now sync the slots locally */ + foreach_ptr(RemoteSlot, remote_slot, remote_slot_list) + { + Oid remote_dbid = get_database_oid(remote_slot->database, false); + + /* + * Use shared lock to prevent a conflict with + * ReplicationSlotsDropDBSlots(), trying to drop the same slot during + * a drop-database operation. + */ + LockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); + + synchronize_one_slot(remote_slot, remote_dbid); + + UnlockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); + } + + /* We are done, free remote_slot_list elements */ + list_free_deep(remote_slot_list); + + walrcv_clear_result(res); + + SpinLockAcquire(&SlotSyncCtx->mutex); + SlotSyncCtx->syncing = false; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = false; +} + +/* + * Checks the remote server info. + * + * We ensure that the 'primary_slot_name' exists on the remote server and the + * remote server is not a standby node. + */ +static void +validate_remote_info(WalReceiverConn *wrconn) +{ +#define PRIMARY_INFO_OUTPUT_COL_COUNT 2 + WalRcvExecResult *res; + Oid slotRow[PRIMARY_INFO_OUTPUT_COL_COUNT] = {BOOLOID, BOOLOID}; + StringInfoData cmd; + bool isnull; + TupleTableSlot *tupslot; + bool remote_in_recovery; + bool primary_slot_valid; + + initStringInfo(&cmd); + appendStringInfo(&cmd, + "SELECT pg_is_in_recovery(), count(*) = 1" + " FROM pg_catalog.pg_replication_slots" + " WHERE slot_type='physical' AND slot_name=%s", + quote_literal_cstr(PrimarySlotName)); + + res = walrcv_exec(wrconn, cmd.data, PRIMARY_INFO_OUTPUT_COL_COUNT, slotRow); + pfree(cmd.data); + + if (res->status != WALRCV_OK_TUPLES) + ereport(ERROR, + errmsg("could not fetch primary_slot_name \"%s\" info from the primary server: %s", + PrimarySlotName, res->err), + errhint("Check if \"primary_slot_name\" is configured correctly.")); + + tupslot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple); + if (!tuplestore_gettupleslot(res->tuplestore, true, false, tupslot)) + elog(ERROR, + "failed to fetch tuple for the primary server slot specified by \"primary_slot_name\""); + + remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull)); + Assert(!isnull); + + if (remote_in_recovery) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot synchronize replication slots from a standby server")); + + primary_slot_valid = DatumGetBool(slot_getattr(tupslot, 2, &isnull)); + Assert(!isnull); + + if (!primary_slot_valid) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + /* translator: second %s is a GUC variable name */ + errdetail("The replication slot \"%s\" specified by \"%s\" does not exist on the primary server.", + PrimarySlotName, "primary_slot_name")); + + ExecClearTuple(tupslot); + walrcv_clear_result(res); +} + +/* + * Check all necessary GUCs for slot synchronization are set + * appropriately, otherwise, raise ERROR. + */ +void +ValidateSlotSyncParams(void) +{ + char *dbname; + + /* + * A physical replication slot(primary_slot_name) is required on the + * primary to ensure that the rows needed by the standby are not removed + * after restarting, so that the synchronized slot on the standby will not + * be invalidated. + */ + if (PrimarySlotName == NULL || *PrimarySlotName == '\0') + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be defined.", "primary_slot_name")); + + /* + * hot_standby_feedback must be enabled to cooperate with the physical + * replication slot, which allows informing the primary about the xmin and + * catalog_xmin values on the standby. + */ + if (!hot_standby_feedback) + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be enabled.", "hot_standby_feedback")); + + /* Logical slot sync/creation requires wal_level >= logical. */ + if (wal_level < WAL_LEVEL_LOGICAL) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"wal_level\" must be >= logical.")); + + /* + * The primary_conninfo is required to make connection to primary for + * getting slots information. + */ + if (PrimaryConnInfo == NULL || *PrimaryConnInfo == '\0') + ereport(ERROR, + /* translator: %s is a GUC variable name */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("\"%s\" must be defined.", "primary_conninfo")); + + /* + * The slot synchronization needs a database connection for walrcv_exec to + * work. + */ + dbname = walrcv_get_dbname_from_conninfo(PrimaryConnInfo); + if (dbname == NULL) + ereport(ERROR, + + /* + * translator: 'dbname' is a specific option; %s is a GUC variable + * name + */ + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + errhint("'dbname' must be specified in \"%s\".", "primary_conninfo")); +} + +/* + * Is current process syncing replication slots ? + */ +bool +IsSyncingReplicationSlots(void) +{ + return syncing_slots; +} + +/* + * Amount of shared memory required for slot synchronization. + */ +Size +SlotSyncShmemSize(void) +{ + return sizeof(SlotSyncCtxStruct); +} + +/* + * Allocate and initialize the shared memory of slot synchronization. + */ +void +SlotSyncShmemInit(void) +{ + bool found; + + SlotSyncCtx = (SlotSyncCtxStruct *) + ShmemInitStruct("Slot Sync Data", SlotSyncShmemSize(), &found); + + if (!found) + { + SlotSyncCtx->syncing = false; + SpinLockInit(&SlotSyncCtx->mutex); + } +} + +/* + * Error cleanup callback for slot synchronization. + */ +static void +slotsync_failure_callback(int code, Datum arg) +{ + WalReceiverConn *wrconn = (WalReceiverConn *) DatumGetPointer(arg); + + if (syncing_slots) + { + /* + * If syncing_slots is true, it indicates that the process errored out + * without resetting the flag. So, we need to clean up shared memory + * and reset the flag here. + */ + SpinLockAcquire(&SlotSyncCtx->mutex); + SlotSyncCtx->syncing = false; + SpinLockRelease(&SlotSyncCtx->mutex); + + syncing_slots = false; + } + + walrcv_disconnect(wrconn); +} + +/* + * Synchronize the failover enabled replication slots using the specified + * primary server connection. + */ +void +SyncReplicationSlots(WalReceiverConn *wrconn) +{ + PG_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); + { + validate_remote_info(wrconn); + + synchronize_slots(wrconn); + } + PG_END_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); +} diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index fd4e96c9d6..3864dd95af 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -46,6 +46,7 @@ #include "common/string.h" #include "miscadmin.h" #include "pgstat.h" +#include "replication/slotsync.h" #include "replication/slot.h" #include "storage/fd.h" #include "storage/ipc.h" @@ -103,7 +104,6 @@ int max_replication_slots = 10; /* the maximum number of replication * slots */ static void ReplicationSlotShmemExit(int code, Datum arg); -static void ReplicationSlotDropAcquired(void); static void ReplicationSlotDropPtr(ReplicationSlot *slot); /* internal persistency functions */ @@ -250,11 +250,12 @@ ReplicationSlotValidateName(const char *name, int elevel) * user will only get commit prepared. * failover: If enabled, allows the slot to be synced to standbys so * that logical replication can be resumed after failover. + * synced: True if the slot is synchronized from the primary server. */ void ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotPersistency persistency, - bool two_phase, bool failover) + bool two_phase, bool failover, bool synced) { ReplicationSlot *slot = NULL; int i; @@ -263,6 +264,34 @@ ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotValidateName(name, ERROR); + if (failover) + { + /* + * Do not allow users to create the failover enabled slots on the + * standby as we do not support sync to the cascading standby. + * + * However, failover enabled slots can be created during slot + * synchronization because we need to retain the same values as the + * remote slot. + */ + if (RecoveryInProgress() && !IsSyncingReplicationSlots()) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a replication slot created on the standby")); + + /* + * Do not allow users to create failover enabled temporary slots, + * because temporary slots will not be synced to the standby. + * + * However, failover enabled temporary slots can be created during + * slot synchronization. See the comments atop slotsync.c for details. + */ + if (persistency == RS_TEMPORARY && !IsSyncingReplicationSlots()) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a temporary replication slot")); + } + /* * If some other backend ran this code concurrently with us, we'd likely * both allocate the same slot, and that would be bad. We'd also be at @@ -315,6 +344,7 @@ ReplicationSlotCreate(const char *name, bool db_specific, slot->data.two_phase = two_phase; slot->data.two_phase_at = InvalidXLogRecPtr; slot->data.failover = failover; + slot->data.synced = synced; /* and then data only present in shared memory */ slot->just_dirtied = false; @@ -677,6 +707,16 @@ ReplicationSlotDrop(const char *name, bool nowait) ReplicationSlotAcquire(name, nowait); + /* + * Do not allow users to drop the slots which are currently being synced + * from the primary to the standby. + */ + if (RecoveryInProgress() && MyReplicationSlot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot drop replication slot \"%s\"", name), + errdetail("This slot is being synced from the primary server.")); + ReplicationSlotDropAcquired(); } @@ -696,6 +736,38 @@ ReplicationSlotAlter(const char *name, bool failover) errmsg("cannot use %s with a physical replication slot", "ALTER_REPLICATION_SLOT")); + if (RecoveryInProgress()) + { + /* + * Do not allow users to alter the slots which are currently being + * synced from the primary to the standby. + */ + if (MyReplicationSlot->data.synced) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot alter replication slot \"%s\"", name), + errdetail("This slot is being synced from the primary server.")); + + /* + * Do not allow users to enable failover on the standby as we do not + * support sync to the cascading standby. + */ + if (failover) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a replication slot" + " on the standby")); + } + + /* + * Do not allow users to enable failover for temporary slots as we do not + * support syncing temporary slots to the standby. + */ + if (failover && MyReplicationSlot->data.persistency == RS_TEMPORARY) + ereport(ERROR, + errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot enable failover for a temporary replication slot")); + if (MyReplicationSlot->data.failover != failover) { SpinLockAcquire(&MyReplicationSlot->mutex); @@ -712,7 +784,7 @@ ReplicationSlotAlter(const char *name, bool failover) /* * Permanently drop the currently acquired replication slot. */ -static void +void ReplicationSlotDropAcquired(void) { ReplicationSlot *slot = MyReplicationSlot; @@ -868,8 +940,8 @@ ReplicationSlotMarkDirty(void) } /* - * Convert a slot that's marked as RS_EPHEMERAL to a RS_PERSISTENT slot, - * guaranteeing it will be there after an eventual crash. + * Convert a slot that's marked as RS_EPHEMERAL or RS_TEMPORARY to a + * RS_PERSISTENT slot, guaranteeing it will be there after an eventual crash. */ void ReplicationSlotPersist(void) @@ -2189,3 +2261,25 @@ RestoreSlotFromDisk(const char *name) (errmsg("too many replication slots active before shutdown"), errhint("Increase max_replication_slots and try again."))); } + +/* + * Maps the pg_replication_slots.conflict_reason text value to + * ReplicationSlotInvalidationCause enum value + */ +ReplicationSlotInvalidationCause +GetSlotInvalidationCause(char *conflict_reason) +{ + Assert(conflict_reason); + + if (strcmp(conflict_reason, SLOT_INVAL_WAL_REMOVED_TEXT) == 0) + return RS_INVAL_WAL_REMOVED; + else if (strcmp(conflict_reason, SLOT_INVAL_HORIZON_TEXT) == 0) + return RS_INVAL_HORIZON; + else if (strcmp(conflict_reason, SLOT_INVAL_WAL_LEVEL_TEXT) == 0) + return RS_INVAL_WAL_LEVEL; + else + Assert(0); + + /* Keep compiler quiet */ + return RS_INVAL_NONE; +} diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index eb685089b3..d2fa5e669a 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -21,7 +21,9 @@ #include "replication/decode.h" #include "replication/logical.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "utils/builtins.h" +#include "utils/guc.h" #include "utils/inval.h" #include "utils/pg_lsn.h" #include "utils/resowner.h" @@ -43,7 +45,7 @@ create_physical_replication_slot(char *name, bool immediately_reserve, /* acquire replication slot, this will check for conflicting names */ ReplicationSlotCreate(name, false, temporary ? RS_TEMPORARY : RS_PERSISTENT, false, - false); + false, false); if (immediately_reserve) { @@ -136,7 +138,7 @@ create_logical_replication_slot(char *name, char *plugin, */ ReplicationSlotCreate(name, true, temporary ? RS_TEMPORARY : RS_EPHEMERAL, two_phase, - failover); + failover, false); /* * Create logical decoding context to find start point or, if we don't @@ -237,7 +239,7 @@ pg_drop_replication_slot(PG_FUNCTION_ARGS) Datum pg_get_replication_slots(PG_FUNCTION_ARGS) { -#define PG_GET_REPLICATION_SLOTS_COLS 16 +#define PG_GET_REPLICATION_SLOTS_COLS 17 ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; XLogRecPtr currlsn; int slotno; @@ -418,21 +420,23 @@ pg_get_replication_slots(PG_FUNCTION_ARGS) break; case RS_INVAL_WAL_REMOVED: - values[i++] = CStringGetTextDatum("wal_removed"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_WAL_REMOVED_TEXT); break; case RS_INVAL_HORIZON: - values[i++] = CStringGetTextDatum("rows_removed"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_HORIZON_TEXT); break; case RS_INVAL_WAL_LEVEL: - values[i++] = CStringGetTextDatum("wal_level_insufficient"); + values[i++] = CStringGetTextDatum(SLOT_INVAL_WAL_LEVEL_TEXT); break; } } values[i++] = BoolGetDatum(slot_contents.data.failover); + values[i++] = BoolGetDatum(slot_contents.data.synced); + Assert(i == PG_GET_REPLICATION_SLOTS_COLS); tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, @@ -700,7 +704,6 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) XLogRecPtr src_restart_lsn; bool src_islogical; bool temporary; - bool failover; char *plugin; Datum values[2]; bool nulls[2]; @@ -756,7 +759,6 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) src_islogical = SlotIsLogical(&first_slot_contents); src_restart_lsn = first_slot_contents.data.restart_lsn; temporary = (first_slot_contents.data.persistency == RS_TEMPORARY); - failover = first_slot_contents.data.failover; plugin = logical_slot ? NameStr(first_slot_contents.data.plugin) : NULL; /* Check type of replication slot */ @@ -791,12 +793,20 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) * We must not try to read WAL, since we haven't reserved it yet -- * hence pass find_startpoint false. confirmed_flush will be set * below, by copying from the source slot. + * + * To avoid potential issues with the slot synchronization where the + * restart_lsn of a replication slot can go backward, we set the + * failover option to false here. This situation occurs when a slot + * on the primary server is dropped and immediately replaced with a + * new slot of the same name, created by copying from another existing + * slot. However, the slot synchronization will only observe the + * restart_lsn of the same slot going backward. */ create_logical_replication_slot(NameStr(*dst_name), plugin, temporary, false, - failover, + false, src_restart_lsn, false); } @@ -943,3 +953,49 @@ pg_copy_physical_replication_slot_b(PG_FUNCTION_ARGS) { return copy_replication_slot(fcinfo, false); } + +/* + * Synchronize failover enabled replication slots to a standby server + * from the primary server. + */ +Datum +pg_sync_replication_slots(PG_FUNCTION_ARGS) +{ + WalReceiverConn *wrconn; + char *err; + StringInfoData app_name; + + CheckSlotPermissions(); + + if (!RecoveryInProgress()) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("replication slots can only be synchronized to a standby server")); + + /* Load the libpq-specific functions */ + load_file("libpqwalreceiver", false); + + ValidateSlotSyncParams(); + + initStringInfo(&app_name); + if (cluster_name[0]) + appendStringInfo(&app_name, "%s_slotsync", cluster_name); + else + appendStringInfoString(&app_name, "slotsync"); + + /* Connect to the primary server. */ + wrconn = walrcv_connect(PrimaryConnInfo, false, false, false, + app_name.data, &err); + pfree(app_name.data); + + if (!wrconn) + ereport(ERROR, + errcode(ERRCODE_CONNECTION_FAILURE), + errmsg("could not connect to the primary server: %s", err)); + + SyncReplicationSlots(wrconn); + + walrcv_disconnect(wrconn); + + PG_RETURN_VOID(); +} diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 146826d5db..4e54779a9e 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -72,6 +72,7 @@ #include "postmaster/interrupt.h" #include "replication/decode.h" #include "replication/logical.h" +#include "replication/slotsync.h" #include "replication/slot.h" #include "replication/snapbuild.h" #include "replication/syncrep.h" @@ -243,7 +244,6 @@ static void WalSndShutdown(void) pg_attribute_noreturn(); static void XLogSendPhysical(void); static void XLogSendLogical(void); static void WalSndDone(WalSndSendDataCallback send_data); -static XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); static void IdentifySystem(void); static void UploadManifest(void); static bool HandleUploadManifestPacket(StringInfo buf, off_t *offset, @@ -1224,7 +1224,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) { ReplicationSlotCreate(cmd->slotname, false, cmd->temporary ? RS_TEMPORARY : RS_PERSISTENT, - false, false); + false, false, false); if (reserve_wal) { @@ -1255,7 +1255,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) */ ReplicationSlotCreate(cmd->slotname, true, cmd->temporary ? RS_TEMPORARY : RS_EPHEMERAL, - two_phase, failover); + two_phase, failover, false); /* * Do options check early so that we can bail before calling the @@ -3385,14 +3385,17 @@ WalSndDone(WalSndSendDataCallback send_data) } /* - * Returns the latest point in WAL that has been safely flushed to disk, and - * can be sent to the standby. This should only be called when in recovery, - * ie. we're streaming to a cascaded standby. + * Returns the latest point in WAL that has been safely flushed to disk. + * This should only be called when in recovery. + * + * This is called either by cascading walsender to find WAL postion to be sent + * to a cascaded standby or by slot synchronization function to validate remote + * slot's lsn before syncing it locally. * * As a side-effect, *tli is updated to the TLI of the last * replayed WAL record. */ -static XLogRecPtr +XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli) { XLogRecPtr replayPtr; @@ -3401,6 +3404,8 @@ GetStandbyFlushRecPtr(TimeLineID *tli) TimeLineID receiveTLI; XLogRecPtr result; + Assert(am_cascading_walsender || IsSyncingReplicationSlots()); + /* * We can safely send what's already been replayed. Also, if walreceiver * is streaming WAL from the same timeline, we can send anything that it diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index 7084e18861..7e7941d625 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -36,6 +36,7 @@ #include "replication/logicallauncher.h" #include "replication/origin.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/walreceiver.h" #include "replication/walsender.h" #include "storage/bufmgr.h" @@ -153,6 +154,7 @@ CalculateShmemSize(int *num_semaphores) size = add_size(size, StatsShmemSize()); size = add_size(size, WaitEventExtensionShmemSize()); size = add_size(size, InjectionPointShmemSize()); + size = add_size(size, SlotSyncShmemSize()); #ifdef EXEC_BACKEND size = add_size(size, ShmemBackendArraySize()); #endif @@ -347,6 +349,7 @@ CreateOrAttachShmemStructs(void) WalSummarizerShmemInit(); PgArchShmemInit(); ApplyLauncherShmemInit(); + SlotSyncShmemInit(); /* * Set up other modules that need some shared memory space diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 29af4ce65d..9c120fc2b7 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11127,9 +11127,9 @@ proname => 'pg_get_replication_slots', prorows => '10', proisstrict => 'f', proretset => 't', provolatile => 's', prorettype => 'record', proargtypes => '', - proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,text,bool}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,conflict_reason,failover}', + proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,text,bool,bool}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,conflict_reason,failover,synced}', prosrc => 'pg_get_replication_slots' }, { oid => '3786', descr => 'set up a logical replication slot', proname => 'pg_create_logical_replication_slot', provolatile => 'v', @@ -11212,6 +11212,10 @@ proname => 'pg_logical_emit_message', provolatile => 'v', proparallel => 'u', prorettype => 'pg_lsn', proargtypes => 'bool text bytea bool', prosrc => 'pg_logical_emit_message_bytea' }, +{ oid => '9929', descr => 'sync replication slots from the primary to the standby', + proname => 'pg_sync_replication_slots', provolatile => 'v', proparallel => 'u', + prorettype => 'void', proargtypes => '', + prosrc => 'pg_sync_replication_slots' }, # event triggers { oid => '3566', descr => 'list objects dropped by the current command', diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index da4c776492..e706ca834c 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -52,6 +52,14 @@ typedef enum ReplicationSlotInvalidationCause RS_INVAL_WAL_LEVEL, } ReplicationSlotInvalidationCause; +/* + * The possible values for 'conflict_reason' returned in + * pg_get_replication_slots. + */ +#define SLOT_INVAL_WAL_REMOVED_TEXT "wal_removed" +#define SLOT_INVAL_HORIZON_TEXT "rows_removed" +#define SLOT_INVAL_WAL_LEVEL_TEXT "wal_level_insufficient" + /* * On-Disk data of a replication slot, preserved across restarts. */ @@ -112,6 +120,11 @@ typedef struct ReplicationSlotPersistentData /* plugin name */ NameData plugin; + /* + * Was this slot synchronized from the primary server? + */ + char synced; + /* * Is this a failover slot (sync candidate for standbys)? Only relevant * for logical slots on the primary server. @@ -224,9 +237,11 @@ extern void ReplicationSlotsShmemInit(void); /* management of individual slots */ extern void ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotPersistency persistency, - bool two_phase, bool failover); + bool two_phase, bool failover, + bool synced); extern void ReplicationSlotPersist(void); extern void ReplicationSlotDrop(const char *name, bool nowait); +extern void ReplicationSlotDropAcquired(void); extern void ReplicationSlotAlter(const char *name, bool failover); extern void ReplicationSlotAcquire(const char *name, bool nowait); @@ -259,5 +274,7 @@ extern void CheckPointReplicationSlots(bool is_shutdown); extern void CheckSlotRequirements(void); extern void CheckSlotPermissions(void); +extern ReplicationSlotInvalidationCause + GetSlotInvalidationCause(char *conflict_reason); #endif /* SLOT_H */ diff --git a/src/include/replication/slotsync.h b/src/include/replication/slotsync.h new file mode 100644 index 0000000000..e86d8a47b8 --- /dev/null +++ b/src/include/replication/slotsync.h @@ -0,0 +1,23 @@ +/*------------------------------------------------------------------------- + * + * slotsync.h + * Exports for slot synchronization. + * + * Portions Copyright (c) 2016-2024, PostgreSQL Global Development Group + * + * src/include/replication/slotsync.h + * + *------------------------------------------------------------------------- + */ +#ifndef SLOTSYNC_H +#define SLOTSYNC_H + +#include "replication/walreceiver.h" + +extern void ValidateSlotSyncParams(void); +extern bool IsSyncingReplicationSlots(void); +extern Size SlotSyncShmemSize(void); +extern void SlotSyncShmemInit(void); +extern void SyncReplicationSlots(WalReceiverConn *wrconn); + +#endif /* SLOTSYNC_H */ diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h index 1b58d50b3b..0c3996e926 100644 --- a/src/include/replication/walsender.h +++ b/src/include/replication/walsender.h @@ -12,6 +12,8 @@ #ifndef _WALSENDER_H #define _WALSENDER_H +#include "access/xlogdefs.h" + /* * What to do with a snapshot in create replication slot command. */ @@ -37,6 +39,7 @@ extern void InitWalSender(void); extern bool exec_replication_command(const char *cmd_string); extern void WalSndErrorCleanup(void); extern void WalSndResourceCleanup(bool isCommit); +extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); extern void WalSndSignals(void); extern Size WalSndShmemSize(void); extern void WalSndShmemInit(void); diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index bc58ff4cab..c96515d178 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -97,4 +97,241 @@ my ($result, $stdout, $stderr) = $subscriber1->psql('postgres', ok( $stderr =~ /ERROR: cannot set failover for enabled subscription/, "altering failover is not allowed for enabled subscription"); +################################################## +# Test that pg_sync_replication_slots() cannot be executed on a non-standby server. +################################################## + +($result, $stdout, $stderr) = + $publisher->psql('postgres', "SELECT pg_sync_replication_slots();"); +ok( $stderr =~ + /ERROR: replication slots can only be synchronized to a standby server/, + "cannot sync slots on a non-standby server"); + +################################################## +# Test logical failover slots on the standby +# Configure standby1 to replicate and synchronize logical slots configured +# for failover on the primary +# +# failover slot lsub1_slot ->| ----> subscriber1 (connected via logical replication) +# failover slot lsub2_slot | inactive +# primary ---> | +# physical slot sb1_slot --->| ----> standby1 (connected via streaming replication) +# | lsub1_slot, lsub2_slot (synced_slot) +################################################## + +my $primary = $publisher; +my $backup_name = 'backup'; +$primary->backup($backup_name); + +# Create a standby +my $standby1 = PostgreSQL::Test::Cluster->new('standby1'); +$standby1->init_from_backup( + $primary, $backup_name, + has_streaming => 1, + has_restoring => 1); + +my $connstr_1 = $primary->connstr; +$standby1->append_conf( + 'postgresql.conf', qq( +hot_standby_feedback = on +primary_slot_name = 'sb1_slot' +primary_conninfo = '$connstr_1 dbname=postgres' +)); + +$primary->psql('postgres', + q{SELECT pg_create_logical_replication_slot('lsub2_slot', 'test_decoding', false, false, true);} +); + +$primary->psql('postgres', + q{SELECT pg_create_physical_replication_slot('sb1_slot');}); + +# Start the standby so that slot syncing can begin +$standby1->start; + +$primary->wait_for_catchup('regress_mysub1'); + +# Do not allow any further advancement of the restart_lsn for the lsub1_slot. +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); + +# Wait for the replication slot to become inactive on the publisher +$primary->poll_query_until( + 'postgres', + "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", + 1); + +# Wait for the standby to catch up so that the standby is not lagging behind +# the subscriber. +$primary->wait_for_replay_catchup($standby1); + +# Synchronize the primary server slots to the standby. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Confirm that the logical failover slots are created on the standby and are +# flagged as 'synced' +is( $standby1->safe_psql( + 'postgres', + q{SELECT count(*) = 2 FROM pg_replication_slots WHERE slot_name IN ('lsub1_slot', 'lsub2_slot') AND synced;} + ), + "t", + 'logical slots have synced as true on standby'); + +################################################## +# Test that the synchronized slot will be dropped if the corresponding remote +# slot on the primary server has been dropped. +################################################## + +$primary->psql('postgres', "SELECT pg_drop_replication_slot('lsub2_slot');"); + +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +is( $standby1->safe_psql( + 'postgres', + q{SELECT count(*) = 0 FROM pg_replication_slots WHERE slot_name = 'lsub2_slot';} + ), + "t", + 'synchronized slot has been dropped'); + +################################################## +# Test that if the synchronized slot is invalidated while the remote slot is +# still valid, the slot will be dropped and re-created on the standby by +# executing pg_sync_replication_slots() again. +################################################## + +# Configure the max_slot_wal_keep_size so that the synced slot can be +# invalidated due to wal removal. +$standby1->append_conf('postgresql.conf', 'max_slot_wal_keep_size = 64kB'); +$standby1->reload; + +# Generate some activity and switch WAL file on the primary +$primary->advance_wal(1); +$primary->psql('postgres', "CHECKPOINT"); +$primary->wait_for_replay_catchup($standby1); + +# Request a checkpoint on the standby to trigger the WAL file(s) removal +$standby1->safe_psql('postgres', "CHECKPOINT"); + +# Check if the synced slot is invalidated +is( $standby1->safe_psql( + 'postgres', + q{SELECT conflict_reason = 'wal_removed' FROM pg_replication_slots WHERE slot_name = 'lsub1_slot';} + ), + "t", + 'synchronized slot has been invalidated'); + +# Reset max_slot_wal_keep_size to avoid further wal removal +$standby1->append_conf('postgresql.conf', 'max_slot_wal_keep_size = -1'); +$standby1->reload; + +# Enable the subscription to let it catch up to the latest wal position +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 ENABLE"); + +$primary->wait_for_catchup('regress_mysub1'); + +# Do not allow any further advancement of the restart_lsn for the lsub1_slot. +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); + +# Wait for the replication slot to become inactive on the publisher +$primary->poll_query_until( + 'postgres', + "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", + 1); + +# Wait for the standby to catch up so that the standby is not lagging behind +# the subscriber. +$primary->wait_for_replay_catchup($standby1); + +my $log_offset = -s $standby1->logfile; + +# Synchronize the primary server slots to the standby. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Confirm that the invalidated slot has been dropped. +$standby1->wait_for_log(qr/dropped replication slot "lsub1_slot" of dbid [0-9]+/, + $log_offset); + +# Confirm that the logical slot has been re-created on the standby and is +# flagged as 'synced' +is( $standby1->safe_psql( + 'postgres', + q{SELECT conflict_reason IS NULL AND synced FROM pg_replication_slots WHERE slot_name = 'lsub1_slot';} + ), + "t", + 'logical slot is re-synced'); + +################################################## +# Test that a synchronized slot can not be decoded, altered or dropped by the +# user +################################################## + +# Attempting to perform logical decoding on a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql('postgres', + "select * from pg_logical_slot_get_changes('lsub1_slot', NULL, NULL);"); +ok( $stderr =~ + /ERROR: cannot use replication slot "lsub1_slot" for logical decoding/, + "logical decoding is not allowed on synced slot"); + +# Attempting to alter a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql( + 'postgres', + qq[ALTER_REPLICATION_SLOT lsub1_slot (failover);], + replication => 'database'); +ok($stderr =~ /ERROR: cannot alter replication slot "lsub1_slot"/, + "synced slot on standby cannot be altered"); + +# Attempting to drop a synced slot should result in an error +($result, $stdout, $stderr) = $standby1->psql('postgres', + "SELECT pg_drop_replication_slot('lsub1_slot');"); +ok($stderr =~ /ERROR: cannot drop replication slot "lsub1_slot"/, + "synced slot on standby cannot be dropped"); + +################################################## +# Test that we cannot synchronize slots if dbname is not specified in the +# primary_conninfo. +################################################## + +$standby1->append_conf('postgresql.conf', "primary_conninfo = '$connstr_1'"); +$standby1->reload; + +($result, $stdout, $stderr) = + $standby1->psql('postgres', "SELECT pg_sync_replication_slots();"); +ok( $stderr =~ + /HINT: 'dbname' must be specified in "primary_conninfo"/, + "cannot sync slots if dbname is not specified in primary_conninfo"); + +################################################## +# Test that we cannot synchronize slots to a cascading standby server. +################################################## + +# Create a cascading standby +$backup_name = 'backup2'; +$standby1->backup($backup_name); + +my $cascading_standby = PostgreSQL::Test::Cluster->new('cascading_standby'); +$cascading_standby->init_from_backup( + $standby1, $backup_name, + has_streaming => 1, + has_restoring => 1); + +my $cascading_connstr = $standby1->connstr; +$cascading_standby->append_conf( + 'postgresql.conf', qq( +hot_standby_feedback = on +primary_slot_name = 'cascading_sb_slot' +primary_conninfo = '$cascading_connstr dbname=postgres' +)); + +$standby1->psql('postgres', + q{SELECT pg_create_physical_replication_slot('cascading_sb_slot');}); + +$cascading_standby->start; + +($result, $stdout, $stderr) = + $cascading_standby->psql('postgres', "SELECT pg_sync_replication_slots();"); +ok( $stderr =~ + /ERROR: cannot synchronize replication slots from a standby server/, + "cannot sync slots to a cascading standby server"); + done_testing(); diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index abc944e8b8..b7488d760e 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1474,8 +1474,9 @@ pg_replication_slots| SELECT l.slot_name, l.safe_wal_size, l.two_phase, l.conflict_reason, - l.failover - FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, conflict_reason, failover) + l.failover, + l.synced + FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, conflict_reason, failover, synced) LEFT JOIN pg_database d ON ((l.datoid = d.oid))); pg_roles| SELECT pg_authid.rolname, pg_authid.rolsuper, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 91433d439b..d808aad8b0 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2325,6 +2325,7 @@ RelocationBufferInfo RelptrFreePageBtree RelptrFreePageManager RelptrFreePageSpanLeader +RemoteSlot RenameStmt ReopenPtrType ReorderBuffer @@ -2584,6 +2585,7 @@ SlabBlock SlabContext SlabSlot SlotNumber +SlotSyncCtxStruct SlruCtl SlruCtlData SlruErrorCause -- 2.30.0.windows.2 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-14 08:44 Amit Kapila <[email protected]> parent: Zhijie Hou (Fujitsu) <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Amit Kapila @ 2024-02-14 08:44 UTC (permalink / raw) To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Wed, Feb 14, 2024 at 9:34 AM Zhijie Hou (Fujitsu) <[email protected]> wrote: > > Here is V87 patch that adds test for the suggested cases. > I have pushed this patch and it leads to a BF failure: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2024-02-14%2004%3A43%3A3... The test failures are: # Failed test 'logical decoding is not allowed on synced slot' # at /home/bf/bf-build/flaviventris/HEAD/pgsql/src/test/recovery/t/040_standby_failover_slots_sync.pl line 272. # Failed test 'synced slot on standby cannot be altered' # at /home/bf/bf-build/flaviventris/HEAD/pgsql/src/test/recovery/t/040_standby_failover_slots_sync.pl line 281. # Failed test 'synced slot on standby cannot be dropped' # at /home/bf/bf-build/flaviventris/HEAD/pgsql/src/test/recovery/t/040_standby_failover_slots_sync.pl line 287. The reason is that in LOGs, we see a different ERROR message than what is expected: 2024-02-14 04:52:32.916 UTC [1767765][client backend][3/4:0] ERROR: replication slot "lsub1_slot" is active for PID 1760871 Now, we see the slot still active because a test before these tests (# Test that if the synchronized slot is invalidated while the remote slot is still valid, ....) is not able to successfully persist the slot and the synced temporary slot remains active. The reason is clear by referring to below standby LOGS: LOG: connection authorized: user=bf database=postgres application_name=040_standby_failover_slots_sync.pl LOG: statement: SELECT pg_sync_replication_slots(); LOG: dropped replication slot "lsub1_slot" of dbid 5 STATEMENT: SELECT pg_sync_replication_slots(); ... SELECT conflict_reason IS NULL AND synced FROM pg_replication_slots WHERE slot_name = 'lsub1_slot'; In the above LOGs, we should ideally see: "newly created slot "lsub1_slot" is sync-ready now" after the "LOG: dropped replication slot "lsub1_slot" of dbid 5" but lack of that means the test didn't accomplish what it was supposed to. Ideally, the same test should have failed but the pass criteria for the test failed to check whether the slot is persisted or not. The probable reason for failure is that remote_slot's restart_lsn lags behind the oldest WAL segment on standby. Now, in the test, we do ensure that the publisher and subscriber are caught up by following steps: # Enable the subscription to let it catch up to the latest wal position $subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE"); $primary->wait_for_catchup('regress_mysub1'); However, this doesn't guarantee that restart_lsn is moved to a position new enough that standby has a WAL corresponding to it. One easy fix is to re-create the subscription with the same slot_name after we have ensured that the slot has been invalidated on standby so that a new restart_lsn is assigned to the slot but it is better to analyze some more why the slot's restart_lsn hasn't moved enough only sometimes. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-14 10:05 Amit Kapila <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Amit Kapila @ 2024-02-14 10:05 UTC (permalink / raw) To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Wed, Feb 14, 2024 at 2:14 PM Amit Kapila <[email protected]> wrote: > > On Wed, Feb 14, 2024 at 9:34 AM Zhijie Hou (Fujitsu) > <[email protected]> wrote: > > > > Here is V87 patch that adds test for the suggested cases. > > > > I have pushed this patch and it leads to a BF failure: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2024-02-14%2004%3A43%3A3... > > The test failures are: > # Failed test 'logical decoding is not allowed on synced slot' > # at /home/bf/bf-build/flaviventris/HEAD/pgsql/src/test/recovery/t/040_standby_failover_slots_sync.pl > line 272. > # Failed test 'synced slot on standby cannot be altered' > # at /home/bf/bf-build/flaviventris/HEAD/pgsql/src/test/recovery/t/040_standby_failover_slots_sync.pl > line 281. > # Failed test 'synced slot on standby cannot be dropped' > # at /home/bf/bf-build/flaviventris/HEAD/pgsql/src/test/recovery/t/040_standby_failover_slots_sync.pl > line 287. > > The reason is that in LOGs, we see a different ERROR message than what > is expected: > 2024-02-14 04:52:32.916 UTC [1767765][client backend][3/4:0] ERROR: > replication slot "lsub1_slot" is active for PID 1760871 > > Now, we see the slot still active because a test before these tests (# > Test that if the synchronized slot is invalidated while the remote > slot is still valid, ....) is not able to successfully persist the > slot and the synced temporary slot remains active. > > The reason is clear by referring to below standby LOGS: > > LOG: connection authorized: user=bf database=postgres > application_name=040_standby_failover_slots_sync.pl > LOG: statement: SELECT pg_sync_replication_slots(); > LOG: dropped replication slot "lsub1_slot" of dbid 5 > STATEMENT: SELECT pg_sync_replication_slots(); > ... > SELECT conflict_reason IS NULL AND synced FROM pg_replication_slots > WHERE slot_name = 'lsub1_slot'; > > In the above LOGs, we should ideally see: "newly created slot > "lsub1_slot" is sync-ready now" after the "LOG: dropped replication > slot "lsub1_slot" of dbid 5" but lack of that means the test didn't > accomplish what it was supposed to. Ideally, the same test should have > failed but the pass criteria for the test failed to check whether the > slot is persisted or not. > > The probable reason for failure is that remote_slot's restart_lsn lags > behind the oldest WAL segment on standby. Now, in the test, we do > ensure that the publisher and subscriber are caught up by following > steps: > # Enable the subscription to let it catch up to the latest wal position > $subscriber1->safe_psql('postgres', > "ALTER SUBSCRIPTION regress_mysub1 ENABLE"); > > $primary->wait_for_catchup('regress_mysub1'); > > However, this doesn't guarantee that restart_lsn is moved to a > position new enough that standby has a WAL corresponding to it. > To ensure that restart_lsn has been moved to a recent position, we need to log XLOG_RUNNING_XACTS and make sure the same is processed as well by walsender. The attached patch does the required change. Hou-San can reproduce this problem by adding additional checkpoints in the test and after applying the attached it fixes the problem. Now, this patch is mostly based on the theory we formed based on LOGs on BF and a reproducer by Hou-San, so still, there is some chance that this doesn't fix the BF failures in which case I'll again look into those. -- With Regards, Amit Kapila. Attachments: [application/octet-stream] fix_040_standby_failover_slots_sync.1.patch (860B, ../../CAA4eK1+Jn=W6_XVxq2gG+fWX9a8iHa0DU0pcPcb41UAejUZ0rQ@mail.gmail.com/2-fix_040_standby_failover_slots_sync.1.patch) download | inline diff: diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index c96515d178..117546674f 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -227,6 +227,13 @@ $standby1->reload; $subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE"); +# This wait ensures that confirmed_flush_lsn has been moved to latest +# position. +$primary->wait_for_catchup('regress_mysub1'); + +# To ensure that restart_lsn has moved to a recent WAL position, we need +# to log XLOG_RUNNING_XACTS and make sure the same is processed as well +$primary->psql('postgres', "CHECKPOINT"); $primary->wait_for_catchup('regress_mysub1'); # Do not allow any further advancement of the restart_lsn for the lsub1_slot. ^ permalink raw reply [nested|flat] 46+ messages in thread
* RE: Synchronizing slots from primary to standby @ 2024-02-15 06:36 Zhijie Hou (Fujitsu) <[email protected]> parent: Zhijie Hou (Fujitsu) <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Zhijie Hou (Fujitsu) @ 2024-02-15 06:36 UTC (permalink / raw) To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Bertrand Drouvot <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> Hi, Since the slotsync function is committed, I rebased remaining patches. And here is the V88 patch set. Best Regards, Hou zj Attachments: [application/octet-stream] v88-0003-Document-the-steps-to-check-if-the-standby-is-re.patch (7.0K, ../../OS0PR01MB571618061B7774189B3493D7944D2@OS0PR01MB5716.jpnprd01.prod.outlook.com/2-v88-0003-Document-the-steps-to-check-if-the-standby-is-re.patch) download | inline diff: From 795588c0da620aea9ee0472341e98ff32070b5f7 Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Fri, 19 Jan 2024 11:04:16 +0530 Subject: [PATCH v88 3/3] Document the steps to check if the standby is ready for failover --- doc/src/sgml/high-availability.sgml | 9 ++ doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++ 2 files changed, 145 insertions(+) diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml index 236c0af65f..36215aa68c 100644 --- a/doc/src/sgml/high-availability.sgml +++ b/doc/src/sgml/high-availability.sgml @@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)' Written administration procedures are advised. </para> + <para> + If you have opted for synchronization of logical slots (see + <xref linkend="logicaldecoding-replication-slots-synchronization"/>), + then before switching to the standby server, it is recommended to check + if the logical slots synchronized on the standby server are ready + for failover. This can be done by following the steps described in + <xref linkend="logical-replication-failover"/>. + </para> + <para> To trigger failover of a log-shipping standby server, run <command>pg_ctl promote</command> or call <function>pg_promote()</function>. diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index ec2130669e..be59d306a1 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -687,6 +687,142 @@ ALTER SUBSCRIPTION </sect1> + <sect1 id="logical-replication-failover"> + <title>Logical Replication Failover</title> + + <para> + When the publisher server is the primary server of a streaming replication, + the logical slots on that primary server can be synchronized to the standby + server by specifying <literal>failover = true</literal> when creating + subscriptions for those publications. Enabling failover ensures a seamless + transition of those subscriptions after the standby is promoted. They can + continue subscribing to publications now on the new primary server without + any data loss. + </para> + + <para> + Because the slot synchronization logic copies asynchronously, it is + necessary to confirm that replication slots have been synced to the standby + server before the failover happens. Furthermore, to ensure a successful + failover, the standby server must not be lagging behind the subscriber. It + is highly recommended to use + <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link> + to prevent the subscriber from consuming changes faster than the hot standby. + To confirm that the standby server is indeed ready for failover, follow + these 2 steps: + </para> + + <procedure> + <step performance="required"> + <para> + Confirm that all the necessary logical replication slots have been synced to + the standby server. + </para> + <substeps> + <step performance="required"> + <para> + Firstly, on the subscriber node, use the following SQL to identify + which slots should be synced to the standby that we plan to promote. +<programlisting> +test_sub=# SELECT + array_agg(slotname) AS slots + FROM + (( + SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname + FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s + WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover + ) UNION ( + SELECT s.oid AS subid, s.subslotname as slotname + FROM pg_subscription s + WHERE s.subfailover + )); + slots +------- + {sub1,sub2,sub3} +(1 row) +</programlisting></para> + </step> + <step performance="required"> + <para> + Next, check that the logical replication slots identified above exist on + the standby server and are ready for failover. +<programlisting> +test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready + FROM pg_replication_slots + WHERE slot_name IN ('sub1','sub2','sub3'); + slot_name | failover_ready +-------------+---------------- + sub1 | t + sub2 | t + sub3 | t +(3 rows) +</programlisting></para> + </step> + </substeps> + </step> + + <step performance="required"> + <para> + Confirm that the standby server is not lagging behind the subscribers. + This step can be skipped if + <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link> + has been correctly configured. If standby_slot_names is not configured + correctly, it is highly recommended to run this step after the primary + server is down, otherwise the results of the query may vary at different + points of time due to the ongoing replication on the logical subscribers + from the primary server. + </para> + <substeps> + <step performance="required"> + <para> + Firstly, on the subscriber node check the last replayed WAL. + This step needs to be run on the database(s) that includes the failover + enabled subscription(s), to find the last replayed WAL on each database. +<programlisting> +test_sub=# SELECT + MAX(remote_lsn) AS remote_lsn_on_subscriber + FROM + (( + SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false) + WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn + FROM pg_subscription_rel r, pg_subscription s + WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover + ) UNION ( + SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn + FROM pg_subscription s + WHERE s.subfailover + )); + remote_lsn_on_subscriber +-------------------------- + 0/3000388 +</programlisting></para> + </step> + <step performance="required"> + <para> + Next, on the standby server check that the last-received WAL location + is ahead of the replayed WAL location(s) on the subscriber identified + above. If the above SQL result was NULL, it means the subscriber has not + yet replayed any WAL, so the standby server must be ahead of the + subscriber, and this step can be skipped. +<programlisting> +test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready; + failover_ready +---------------- + t +(1 row) +</programlisting></para> + </step> + </substeps> + </step> + </procedure> + + <para> + If the result (<literal>failover_ready</literal>) of both above steps is + true, existing subscriptions will be able to continue without data loss. + </para> + + </sect1> + <sect1 id="logical-replication-row-filter"> <title>Row Filters</title> -- 2.30.0.windows.2 [application/octet-stream] v88-0001-Add-a-new-slotsync-worker.patch (60.2K, ../../OS0PR01MB571618061B7774189B3493D7944D2@OS0PR01MB5716.jpnprd01.prod.outlook.com/3-v88-0001-Add-a-new-slotsync-worker.patch) download | inline diff: From 8956ca258371d2b78635cc9930a0d8e8b5782342 Mon Sep 17 00:00:00 2001 From: Hou Zhijie <[email protected]> Date: Thu, 15 Feb 2024 12:09:55 +0800 Subject: [PATCH v88 1/3] Add a new slotsync worker Be enabling slot synchronization, all the failover logical replication slots on the primary (assuming configurations are appropriate) are automatically created on the physical standbys and are synced periodically. Slot-sync worker on the standby server ping the primary server at regular intervals to get the necessary failover logical slots information and create/update the slots locally. The slots that no longer require synchronization are automatically dropped by the worker. The nap time of the worker is tuned according to the activity on the primary. The worker waits for a period of time before the next synchronization, with the duration varying based on whether any slots were updated during the last cycle. A new parameter sync_replication_slots enables or disables this new process. --- doc/src/sgml/config.sgml | 18 + doc/src/sgml/logicaldecoding.sgml | 5 +- src/backend/access/transam/xlogrecovery.c | 15 + src/backend/postmaster/postmaster.c | 81 +- .../libpqwalreceiver/libpqwalreceiver.c | 3 + src/backend/replication/logical/slotsync.c | 749 ++++++++++++++++-- src/backend/replication/slot.c | 18 +- src/backend/replication/slotfuncs.c | 2 +- src/backend/replication/walsender.c | 2 +- src/backend/storage/ipc/ipci.c | 4 +- src/backend/storage/lmgr/proc.c | 13 +- src/backend/utils/activity/pgstat_io.c | 1 + .../utils/activity/wait_event_names.txt | 2 + src/backend/utils/init/miscinit.c | 9 +- src/backend/utils/init/postinit.c | 8 +- src/backend/utils/misc/guc_tables.c | 10 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/miscadmin.h | 1 + src/include/replication/slotsync.h | 25 +- .../t/040_standby_failover_slots_sync.pl | 88 +- src/tools/pgindent/typedefs.list | 2 +- 21 files changed, 971 insertions(+), 86 deletions(-) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 037a3b8a64..02d28a0be9 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4943,6 +4943,24 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-sync-replication-slots" xreflabel="sync_replication_slots"> + <term><varname>sync_replication_slots</varname> (<type>boolean</type>) + <indexterm> + <primary><varname>sync_replication_slots</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + It enables a physical standby to synchronize logical failover slots + from the primary server so that logical subscribers are not blocked + after failover. + </para> + <para> + It is disabled by default. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command line. + </para> + </listitem> + </varlistentry> </variablelist> </sect2> diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index eceaaaa273..930c0fa8a6 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -373,7 +373,10 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU <command>CREATE SUBSCRIPTION</command> during slot creation, and then calling <link linkend="pg-sync-replication-slots"> <function>pg_sync_replication_slots</function></link> - on the standby. For the synchronization to work, it is mandatory to + on the standby. By setting <link linkend="guc-sync-replication-slots"> + <varname>sync_replication_slots</varname></link> + on the standby, the failover slots can be synchronized periodically in + the slotsync worker. For the synchronization to work, it is mandatory to have a physical replication slot between the primary and the standby aka <link linkend="guc-primary-slot-name"><varname>primary_slot_name</varname></link> should be configured on the standby, and diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 0bb472da27..68f48c052c 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -49,6 +49,7 @@ #include "postmaster/bgwriter.h" #include "postmaster/startup.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/walreceiver.h" #include "storage/fd.h" #include "storage/ipc.h" @@ -1467,6 +1468,20 @@ FinishWalRecovery(void) */ XLogShutdownWalRcv(); + /* + * Shutdown the slot sync workers to prevent potential conflicts between + * user processes and slotsync workers after a promotion. + * + * We do not update the 'synced' column from true to false here, as any + * failed update could leave 'synced' column false for some slots. This + * could cause issues during slot sync after restarting the server as a + * standby. While updating after switching to the new timeline is an + * option, it does not simplify the handling for 'synced' column. + * Therefore, we retain the 'synced' column as true after promotion as it + * may provide useful information about the slot origin. + */ + ShutDownSlotSync(); + /* * We are now done reading the xlog from stream. Turn off streaming * recovery to force fetching the files (which would be required at end of diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index df945a5ac4..b0334ce449 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -115,6 +115,7 @@ #include "postmaster/syslogger.h" #include "postmaster/walsummarizer.h" #include "replication/logicallauncher.h" +#include "replication/slotsync.h" #include "replication/walsender.h" #include "storage/fd.h" #include "storage/ipc.h" @@ -167,11 +168,11 @@ * they will never become live backends. dead_end children are not assigned a * PMChildSlot. dead_end children have bkend_type NORMAL. * - * "Special" children such as the startup, bgwriter and autovacuum launcher - * tasks are not in this list. They are tracked via StartupPID and other - * pid_t variables below. (Thus, there can't be more than one of any given - * "special" child process type. We use BackendList entries for any child - * process there can be more than one of.) + * "Special" children such as the startup, bgwriter, autovacuum launcher and + * slot sync worker tasks are not in this list. They are tracked via StartupPID + * and other pid_t variables below. (Thus, there can't be more than one of any + * given "special" child process type. We use BackendList entries for any + * child process there can be more than one of.) */ typedef struct bkend { @@ -254,7 +255,8 @@ static pid_t StartupPID = 0, WalSummarizerPID = 0, AutoVacPID = 0, PgArchPID = 0, - SysLoggerPID = 0; + SysLoggerPID = 0, + SlotSyncWorkerPID = 0; /* Startup process's status */ typedef enum @@ -458,6 +460,10 @@ static void InitPostmasterDeathWatchHandle(void); (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \ PgArchCanRestart()) +#define SlotSyncWorkerAllowed() \ + (sync_replication_slots && pmState == PM_HOT_STANDBY && \ + SlotSyncWorkerCanRestart()) + #ifdef EXEC_BACKEND #ifdef WIN32 @@ -1822,6 +1828,10 @@ ServerLoop(void) if (PgArchPID == 0 && PgArchStartupAllowed()) PgArchPID = StartChildProcess(ArchiverProcess); + /* If we need to start a slot sync worker, try to do that now */ + if (SlotSyncWorkerPID == 0 && SlotSyncWorkerAllowed()) + SlotSyncWorkerPID = StartSlotSyncWorker(); + /* If we need to signal the autovacuum launcher, do so now */ if (avlauncher_needs_signal) { @@ -2661,6 +2671,8 @@ process_pm_reload_request(void) signal_child(PgArchPID, SIGHUP); if (SysLoggerPID != 0) signal_child(SysLoggerPID, SIGHUP); + if (SlotSyncWorkerPID != 0) + signal_child(SlotSyncWorkerPID, SIGHUP); /* Reload authentication config files too */ if (!load_hba()) @@ -3011,6 +3023,9 @@ process_pm_child_exit(void) if (PgArchStartupAllowed() && PgArchPID == 0) PgArchPID = StartChildProcess(ArchiverProcess); + if (SlotSyncWorkerAllowed() && SlotSyncWorkerPID == 0) + SlotSyncWorkerPID = StartSlotSyncWorker(); + /* workers may be scheduled to start now */ maybe_start_bgworkers(); @@ -3180,6 +3195,22 @@ process_pm_child_exit(void) continue; } + /* + * Was it the slot sync worker? Normal exit or FATAL exit can be + * ignored (FATAL can be caused by libpqwalreceiver on receiving + * shutdown request by the startup process during promotion); we'll + * start a new one at the next iteration of the postmaster's main + * loop, if necessary. Any other exit condition is treated as a crash. + */ + if (pid == SlotSyncWorkerPID) + { + SlotSyncWorkerPID = 0; + if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus)) + HandleChildCrash(pid, exitstatus, + _("slot sync worker process")); + continue; + } + /* Was it one of our background workers? */ if (CleanupBackgroundWorker(pid, exitstatus)) { @@ -3384,7 +3415,7 @@ CleanupBackend(int pid, /* * HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer, - * walwriter, autovacuum, archiver or background worker. + * walwriter, autovacuum, archiver, slot sync worker or background worker. * * The objectives here are to clean up our local state about the child * process, and to signal all other remaining children to quickdie. @@ -3546,6 +3577,12 @@ HandleChildCrash(int pid, int exitstatus, const char *procname) else if (PgArchPID != 0 && take_action) sigquit_child(PgArchPID); + /* Take care of the slot sync worker too */ + if (pid == SlotSyncWorkerPID) + SlotSyncWorkerPID = 0; + else if (SlotSyncWorkerPID != 0 && take_action) + sigquit_child(SlotSyncWorkerPID); + /* We do NOT restart the syslogger */ if (Shutdown != ImmediateShutdown) @@ -3686,6 +3723,8 @@ PostmasterStateMachine(void) signal_child(WalReceiverPID, SIGTERM); if (WalSummarizerPID != 0) signal_child(WalSummarizerPID, SIGTERM); + if (SlotSyncWorkerPID != 0) + signal_child(SlotSyncWorkerPID, SIGTERM); /* checkpointer, archiver, stats, and syslogger may continue for now */ /* Now transition to PM_WAIT_BACKENDS state to wait for them to die */ @@ -3701,13 +3740,13 @@ PostmasterStateMachine(void) /* * PM_WAIT_BACKENDS state ends when we have no regular backends * (including autovac workers), no bgworkers (including unconnected - * ones), and no walwriter, autovac launcher or bgwriter. If we are - * doing crash recovery or an immediate shutdown then we expect the - * checkpointer to exit as well, otherwise not. The stats and - * syslogger processes are disregarded since they are not connected to - * shared memory; we also disregard dead_end children here. Walsenders - * and archiver are also disregarded, they will be terminated later - * after writing the checkpoint record. + * ones), and no walwriter, autovac launcher, bgwriter or slot sync + * worker. If we are doing crash recovery or an immediate shutdown + * then we expect the checkpointer to exit as well, otherwise not. The + * stats and syslogger processes are disregarded since they are not + * connected to shared memory; we also disregard dead_end children + * here. Walsenders and archiver are also disregarded, they will be + * terminated later after writing the checkpoint record. */ if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 && StartupPID == 0 && @@ -3717,7 +3756,8 @@ PostmasterStateMachine(void) (CheckpointerPID == 0 || (!FatalError && Shutdown < ImmediateShutdown)) && WalWriterPID == 0 && - AutoVacPID == 0) + AutoVacPID == 0 && + SlotSyncWorkerPID == 0) { if (Shutdown >= ImmediateShutdown || FatalError) { @@ -3815,6 +3855,7 @@ PostmasterStateMachine(void) Assert(CheckpointerPID == 0); Assert(WalWriterPID == 0); Assert(AutoVacPID == 0); + Assert(SlotSyncWorkerPID == 0); /* syslogger is not considered here */ pmState = PM_NO_CHILDREN; } @@ -4038,6 +4079,8 @@ TerminateChildren(int signal) signal_child(AutoVacPID, signal); if (PgArchPID != 0) signal_child(PgArchPID, signal); + if (SlotSyncWorkerPID != 0) + signal_child(SlotSyncWorkerPID, signal); } /* @@ -4850,6 +4893,7 @@ SubPostmasterMain(int argc, char *argv[]) */ if (strcmp(argv[1], "--forkbackend") == 0 || strcmp(argv[1], "--forkavlauncher") == 0 || + strcmp(argv[1], "--forkssworker") == 0 || strcmp(argv[1], "--forkavworker") == 0 || strcmp(argv[1], "--forkaux") == 0 || strcmp(argv[1], "--forkbgworker") == 0) @@ -4953,6 +4997,13 @@ SubPostmasterMain(int argc, char *argv[]) AutoVacWorkerMain(argc - 2, argv + 2); /* does not return */ } + if (strcmp(argv[1], "--forkssworker") == 0) + { + /* Restore basic shared memory pointers */ + InitShmemAccess(UsedShmemSegAddr); + + ReplSlotSyncWorkerMain(argc - 2, argv + 2); /* does not return */ + } if (strcmp(argv[1], "--forkbgworker") == 0) { /* do this as early as possible; in particular, before InitProcess() */ diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index 9270d7b855..e40cdbe4d9 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -6,6 +6,9 @@ * loaded as a dynamic module to avoid linking the main server binary with * libpq. * + * Apart from walreceiver, the libpq-specific routines here are now being used + * by logical replication workers and slot sync worker as well. + * * Portions Copyright (c) 2010-2024, PostgreSQL Global Development Group * * diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 0aa1bf1ad2..6492582156 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -10,8 +10,7 @@ * * This file contains the code for slot synchronization on a physical standby * to fetch logical failover slots information from the primary server, create - * the slots on the standby and synchronize them. This is done by a call to SQL - * function pg_sync_replication_slots. + * the slots on the standby and synchronize them periodically. * * If on physical standby, the WAL corresponding to the remote's restart_lsn * is not available or the remote's catalog_xmin precedes the oldest xid for which @@ -23,6 +22,11 @@ * which we can call pg_sync_replication_slots() periodically to perform * syncs. * + * The slot sync worker worker waits for a period of time before the next + * synchronization, with the duration varying based on whether any slots were + * updated during the last cycle. Refer to the comments above + * wait_for_slot_activity() for more details. + * * Any standby synchronized slots will be dropped if they no longer need * to be synchronized. See comment atop drop_local_obsolete_slots() for more * details. @@ -31,31 +35,88 @@ #include "postgres.h" +#include <time.h> + +#include "access/genam.h" +#include "access/table.h" #include "access/xlog_internal.h" #include "access/xlogrecovery.h" #include "catalog/pg_database.h" #include "commands/dbcommands.h" +#include "libpq/pqsignal.h" +#include "pgstat.h" +#include "postmaster/bgworker.h" +#include "postmaster/fork_process.h" +#include "postmaster/interrupt.h" +#include "postmaster/postmaster.h" #include "replication/logical.h" +#include "replication/logicallauncher.h" +#include "replication/walreceiver.h" #include "replication/slotsync.h" #include "storage/ipc.h" #include "storage/lmgr.h" +#include "storage/proc.h" #include "storage/procarray.h" +#include "tcop/tcopprot.h" #include "utils/builtins.h" +#include "utils/fmgroids.h" +#include "utils/guc_hooks.h" #include "utils/pg_lsn.h" +#include "utils/ps_status.h" +#include "utils/timeout.h" +#include "utils/varlena.h" -/* Struct for sharing information to control slot synchronization. */ -typedef struct SlotSyncCtxStruct +/* + * Struct for sharing information to control slot synchronization. + * + * Slot sync worker's pid is needed by the startup process in order to shut it + * down during promotion. Startup process shuts down the slot sync worker and + * also sets stopSignaled=true to handle the race condition when postmaster has + * not noticed the promotion yet and thus may end up restarting slot sync + * worker. If stopSignaled is set, the worker will exit in such a case. + * + * The last_start_time is needed by postmaster to start the slot sync worker + * once per SLOTSYNC_RESTART_INTERVAL_SEC. In cases where a immediate restart + * is expected (e.g., slot sync GUCs change), slot sync worker will reset + * last_start_time before exiting, so that postmaster can start the worker + * without waiting for SLOTSYNC_RESTART_INTERVAL_SEC. + */ +typedef struct SlotSyncWorkerCtxStruct { + pid_t pid; + /* prevents concurrent slot syncs to avoid slot overwrites */ bool syncing; + + bool stopSignaled; + time_t last_start_time; slock_t mutex; -} SlotSyncCtxStruct; +} SlotSyncWorkerCtxStruct; -SlotSyncCtxStruct *SlotSyncCtx = NULL; +SlotSyncWorkerCtxStruct *SlotSyncWorker = NULL; + +/* GUC variable */ +bool sync_replication_slots = false; + +/* + * The sleep time (ms) between slot-sync cycles varies dynamically + * (within a MIN/MAX range) according to slot activity. See + * wait_for_slot_activity() for details. + */ +#define MIN_WORKER_NAPTIME_MS 200 +#define MAX_WORKER_NAPTIME_MS 30000 /* 30s */ + +static long sleep_ms = MIN_WORKER_NAPTIME_MS; + +/* The restart interval for slot sync work used by postmaster */ +#define SLOTSYNC_RESTART_INTERVAL_SEC 10 + +/* Flag to tell if we are in a slot sync worker process */ +static bool am_slotsync_worker = false; /* * Flag to tell if we are syncing replication slots. Unlike the 'syncing' flag - * in SlotSyncCtxStruct, this flag is true only if the current process is + * in SlotSyncWorkerCtxStruct, this flag is true only if the current process is * performing slot synchronization. */ static bool syncing_slots = false; @@ -79,6 +140,15 @@ typedef struct RemoteSlot ReplicationSlotInvalidationCause invalidated; } RemoteSlot; +static void ProcessSlotSyncInterrupts(WalReceiverConn *wrconn, bool restart); + +#ifdef EXEC_BACKEND +static pid_t slotsyncworker_forkexec(void); +#endif +NON_EXEC_STATIC void ReplSlotSyncWorkerMain(int argc, char *argv[]) pg_attribute_noreturn(); + +static void slotsync_failure_callback(int code, Datum arg); + /* * If necessary, update the local synced slot's metadata based on the data * from the remote slot. @@ -340,8 +410,11 @@ reserve_wal_for_local_slot(XLogRecPtr restart_lsn) * If the remote restart_lsn and catalog_xmin have caught up with the * local ones, then update the LSNs and persist the local synced slot for * future synchronization; otherwise, do nothing. + * + * Return true if the slot is marked as RS_PERSISTENT (sync-ready), otherwise + * false. */ -static void +static bool update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) { ReplicationSlot *slot = MyReplicationSlot; @@ -362,7 +435,7 @@ update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) * take more time to create such a slot. Therefore, we keep this slot * and attempt the synchronization in the next cycle. */ - return; + return false; } /* First time slot update, the function must return true */ @@ -374,6 +447,8 @@ update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) ereport(LOG, errmsg("newly created slot \"%s\" is sync-ready now", remote_slot->name)); + + return true; } /* @@ -386,12 +461,15 @@ update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) * the remote_slot catches up with locally reserved position and local slot is * updated. The slot is then persisted and is considered as sync-ready for * periodic syncs. + * + * Returns TRUE if the local slot is updated. */ -static void +static bool synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) { ReplicationSlot *slot; XLogRecPtr latestFlushPtr; + bool slot_updated = false; /* * Make sure that concerned WAL is received and flushed before syncing @@ -399,13 +477,17 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) */ latestFlushPtr = GetStandbyFlushRecPtr(NULL); if (remote_slot->confirmed_lsn > latestFlushPtr) - elog(ERROR, + { + elog(am_slotsync_worker ? LOG : ERROR, "skipping slot synchronization as the received slot sync" " LSN %X/%X for slot \"%s\" is ahead of the standby position %X/%X", LSN_FORMAT_ARGS(remote_slot->confirmed_lsn), remote_slot->name, LSN_FORMAT_ARGS(latestFlushPtr)); + return false; + } + /* Search for the named slot */ if ((slot = SearchNamedReplicationSlot(remote_slot->name, true))) { @@ -452,19 +534,22 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) /* Make sure the invalidated state persists across server restart */ ReplicationSlotMarkDirty(); ReplicationSlotSave(); + + slot_updated = true; } /* Skip the sync of an invalidated slot */ if (slot->data.invalidated != RS_INVAL_NONE) { ReplicationSlotRelease(); - return; + return slot_updated; } /* Slot not ready yet, let's attempt to make it sync-ready now. */ if (slot->data.persistency == RS_TEMPORARY) { - update_and_persist_local_synced_slot(remote_slot, remote_dbid); + slot_updated = update_and_persist_local_synced_slot(remote_slot, + remote_dbid); } /* Slot ready for sync, so sync it. */ @@ -487,6 +572,8 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) { ReplicationSlotMarkDirty(); ReplicationSlotSave(); + + slot_updated = true; } } } @@ -498,7 +585,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) /* Skip creating the local slot if remote_slot is invalidated already */ if (remote_slot->invalidated != RS_INVAL_NONE) - return; + return false; /* * We create temporary slots instead of ephemeral slots here because @@ -535,9 +622,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) LWLockRelease(ProcArrayLock); update_and_persist_local_synced_slot(remote_slot, remote_dbid); + + slot_updated = true; } ReplicationSlotRelease(); + + return slot_updated; } /* @@ -545,8 +636,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) * * Gets the failover logical slots info from the primary server and updates * the slots locally. Creates the slots if not present on the standby. + * + * Returns TRUE if any of the slots gets updated in this sync-cycle. */ -static void +static bool synchronize_slots(WalReceiverConn *wrconn) { #define SLOTSYNC_COLUMN_COUNT 9 @@ -557,21 +650,30 @@ synchronize_slots(WalReceiverConn *wrconn) TupleTableSlot *tupslot; StringInfoData s; List *remote_slot_list = NIL; + bool some_slot_updated = false; + bool started_tx = false; - SpinLockAcquire(&SlotSyncCtx->mutex); - if (SlotSyncCtx->syncing) + SpinLockAcquire(&SlotSyncWorker->mutex); + if (SlotSyncWorker->syncing) { - SpinLockRelease(&SlotSyncCtx->mutex); + SpinLockRelease(&SlotSyncWorker->mutex); ereport(ERROR, errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), errmsg("cannot synchronize replication slots concurrently")); } - SlotSyncCtx->syncing = true; - SpinLockRelease(&SlotSyncCtx->mutex); + SlotSyncWorker->syncing = true; + SpinLockRelease(&SlotSyncWorker->mutex); syncing_slots = true; + /* The syscache access in walrcv_exec() needs a transaction env. */ + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + initStringInfo(&s); /* Construct query to fetch slots with failover enabled. */ @@ -680,7 +782,7 @@ synchronize_slots(WalReceiverConn *wrconn) */ LockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); - synchronize_one_slot(remote_slot, remote_dbid); + some_slot_updated |= synchronize_one_slot(remote_slot, remote_dbid); UnlockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); } @@ -690,11 +792,16 @@ synchronize_slots(WalReceiverConn *wrconn) walrcv_clear_result(res); - SpinLockAcquire(&SlotSyncCtx->mutex); - SlotSyncCtx->syncing = false; - SpinLockRelease(&SlotSyncCtx->mutex); + if (started_tx) + CommitTransactionCommand(); + + SpinLockAcquire(&SlotSyncWorker->mutex); + SlotSyncWorker->syncing = false; + SpinLockRelease(&SlotSyncWorker->mutex); syncing_slots = false; + + return some_slot_updated; } /* @@ -703,8 +810,8 @@ synchronize_slots(WalReceiverConn *wrconn) * We ensure that the 'primary_slot_name' exists on the remote server and the * remote server is not a standby node. */ -static void -validate_remote_info(WalReceiverConn *wrconn) +static bool +validate_remote_info(WalReceiverConn *wrconn, int elevel) { #define PRIMARY_INFO_OUTPUT_COL_COUNT 2 WalRcvExecResult *res; @@ -714,6 +821,7 @@ validate_remote_info(WalReceiverConn *wrconn) TupleTableSlot *tupslot; bool remote_in_recovery; bool primary_slot_valid; + bool started_tx = false; initStringInfo(&cmd); appendStringInfo(&cmd, @@ -722,6 +830,13 @@ validate_remote_info(WalReceiverConn *wrconn) " WHERE slot_type='physical' AND slot_name=%s", quote_literal_cstr(PrimarySlotName)); + /* The syscache access in walrcv_exec() needs a transaction env. */ + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + res = walrcv_exec(wrconn, cmd.data, PRIMARY_INFO_OUTPUT_COL_COUNT, slotRow); pfree(cmd.data); @@ -740,7 +855,7 @@ validate_remote_info(WalReceiverConn *wrconn) Assert(!isnull); if (remote_in_recovery) - ereport(ERROR, + ereport(elevel, errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot synchronize replication slots from a standby server")); @@ -748,7 +863,7 @@ validate_remote_info(WalReceiverConn *wrconn) Assert(!isnull); if (!primary_slot_valid) - ereport(ERROR, + ereport(elevel, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), /* translator: second %s is a GUC variable name */ @@ -757,14 +872,37 @@ validate_remote_info(WalReceiverConn *wrconn) ExecClearTuple(tupslot); walrcv_clear_result(res); + + if (started_tx) + CommitTransactionCommand(); + + return (primary_slot_valid || !remote_in_recovery); } /* - * Check all necessary GUCs for slot synchronization are set - * appropriately, otherwise, raise ERROR. + * Get database name from the conninfo. + * + * If dbname is extracted already from the conninfo, just return it. */ -void -ValidateSlotSyncParams(void) +static char * +get_dbname_from_conninfo(const char *conninfo) +{ + static char *dbname; + + if (dbname) + return dbname; + else + dbname = walrcv_get_dbname_from_conninfo(conninfo); + + return dbname; +} + +/* + * Return true if all necessary GUCs for slot synchronization are set + * appropriately, otherwise, return false. + */ +bool +ValidateSlotSyncParams(int elevel) { char *dbname; @@ -775,11 +913,14 @@ ValidateSlotSyncParams(void) * be invalidated. */ if (PrimarySlotName == NULL || *PrimarySlotName == '\0') - ereport(ERROR, + { + ereport(elevel, /* translator: %s is a GUC variable name */ errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("\"%s\" must be defined.", "primary_slot_name")); + return false; + } /* * hot_standby_feedback must be enabled to cooperate with the physical @@ -787,37 +928,47 @@ ValidateSlotSyncParams(void) * catalog_xmin values on the standby. */ if (!hot_standby_feedback) - ereport(ERROR, + { + ereport(elevel, /* translator: %s is a GUC variable name */ errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("\"%s\" must be enabled.", "hot_standby_feedback")); + return false; + } /* Logical slot sync/creation requires wal_level >= logical. */ if (wal_level < WAL_LEVEL_LOGICAL) - ereport(ERROR, + { + ereport(elevel, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("\"wal_level\" must be >= logical.")); + return false; + } /* * The primary_conninfo is required to make connection to primary for * getting slots information. */ if (PrimaryConnInfo == NULL || *PrimaryConnInfo == '\0') - ereport(ERROR, + { + ereport(elevel, /* translator: %s is a GUC variable name */ errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("\"%s\" must be defined.", "primary_conninfo")); + return false; + } /* * The slot synchronization needs a database connection for walrcv_exec to * work. */ - dbname = walrcv_get_dbname_from_conninfo(PrimaryConnInfo); + dbname = get_dbname_from_conninfo(PrimaryConnInfo); if (dbname == NULL) - ereport(ERROR, + { + ereport(elevel, /* * translator: 'dbname' is a specific option; %s is a GUC variable @@ -826,41 +977,537 @@ ValidateSlotSyncParams(void) errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("'dbname' must be specified in \"%s\".", "primary_conninfo")); + return false; + } + + return true; +} + +/* + * Check that all necessary GUCs for slot synchronization are set + * appropriately. If not, sleep for MAX_WORKER_NAPTIME_MS and check again. + * The idea is to become no-op until we get valid GUCs values. + * + * If all checks pass, extracts the dbname from the primary_conninfo GUC and + * returns it. + */ +static void +ensure_valid_slotsync_params(void) +{ + int rc; + + /* Sanity check. */ + Assert(sync_replication_slots); + + for (;;) + { + if (ValidateSlotSyncParams(LOG)) + break; + + ereport(LOG, errmsg("skipping slot synchronization")); + + ProcessSlotSyncInterrupts(NULL, false); + + rc = WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + MAX_WORKER_NAPTIME_MS, + WAIT_EVENT_REPL_SLOTSYNC_MAIN); + + if (rc & WL_LATCH_SET) + ResetLatch(MyLatch); + } +} + +/* + * Re-read the config file. + * + * If any of the slot sync GUCs have changed, exit the worker and + * let it get restarted by the postmaster. The worker to be exited for + * restart purpose only if the caller passed restart as true. + */ +static void +slotsync_reread_config(bool restart) +{ + char *old_primary_conninfo = pstrdup(PrimaryConnInfo); + char *old_primary_slotname = pstrdup(PrimarySlotName); + bool old_sync_replication_slots = sync_replication_slots; + bool old_hot_standby_feedback = hot_standby_feedback; + bool conninfo_changed; + bool primary_slotname_changed; + + Assert(sync_replication_slots); + + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + + conninfo_changed = strcmp(old_primary_conninfo, PrimaryConnInfo) != 0; + primary_slotname_changed = strcmp(old_primary_slotname, PrimarySlotName) != 0; + pfree(old_primary_conninfo); + pfree(old_primary_slotname); + + if (old_sync_replication_slots != sync_replication_slots) + { + ereport(LOG, + /* translator: %s is a GUC variable name */ + errmsg("slot sync worker will shutdown because %s is disabled", "sync_replication_slots")); + proc_exit(0); + } + + /* The caller instructed to skip restart */ + if (!restart) + return; + + if (conninfo_changed || + primary_slotname_changed || + (old_hot_standby_feedback != hot_standby_feedback)) + { + ereport(LOG, + errmsg("slot sync worker will restart because of a parameter change")); + + /* + * Reset the last-start time for this worker so that the postmaster + * can restart it without waiting for SLOTSYNC_RESTART_INTERVAL_SEC. + */ + SlotSyncWorker->last_start_time = 0; + + proc_exit(0); + } + +} + +/* + * Interrupt handler for main loop of slot sync worker. + */ +static void +ProcessSlotSyncInterrupts(WalReceiverConn *wrconn, bool restart) +{ + CHECK_FOR_INTERRUPTS(); + + if (ShutdownRequestPending) + { + ereport(LOG, + errmsg("replication slot sync worker is shutting down on receiving SIGINT")); + + proc_exit(0); + } + + if (ConfigReloadPending) + slotsync_reread_config(restart); +} + +/* + * Cleanup function for slotsync worker. + * + * Called on slotsync worker exit. + */ +static void +slotsync_worker_onexit(int code, Datum arg) +{ + SpinLockAcquire(&SlotSyncWorker->mutex); + SlotSyncWorker->pid = InvalidPid; + SpinLockRelease(&SlotSyncWorker->mutex); +} + +/* + * Sleep for long enough that we believe it's likely that the slots on primary + * get updated. + * + * If there is no slot activity the wait time between sync-cycles will double + * (to a maximum of 30s). If there is some slot activity the wait time between + * sync-cycles is reset to the minimum (200ms). + */ +static void +wait_for_slot_activity(bool some_slot_updated, bool recheck_primary_info) +{ + int rc; + + if (recheck_primary_info) + { + /* + * If we are on the cascading standby or primary_slot_name configured + * is not valid, then we will skip the sync and take a longer nap + * before we can do validate_remote_info() again. + */ + sleep_ms = MAX_WORKER_NAPTIME_MS; + } + else if (!some_slot_updated) + { + /* + * No slots were updated, so double the sleep time, but not beyond the + * maximum allowable value. + */ + sleep_ms = Min(sleep_ms * 2, MAX_WORKER_NAPTIME_MS); + } + else + { + /* + * Some slots were updated since the last sleep, so reset the sleep + * time. + */ + sleep_ms = MIN_WORKER_NAPTIME_MS; + } + + rc = WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + sleep_ms, + WAIT_EVENT_REPL_SLOTSYNC_MAIN); + + if (rc & WL_LATCH_SET) + ResetLatch(MyLatch); +} + +/* + * The main loop of our worker process. + * + * It connects to the primary server, fetches logical failover slots + * information periodically in order to create and sync the slots. + */ +NON_EXEC_STATIC void +ReplSlotSyncWorkerMain(int argc, char *argv[]) +{ + WalReceiverConn *wrconn = NULL; + char *dbname; + char *err; + sigjmp_buf local_sigjmp_buf; + StringInfoData app_name; + bool remote_info_valid; + + am_slotsync_worker = true; + + MyBackendType = B_SLOTSYNC_WORKER; + + init_ps_display(NULL); + + SetProcessingMode(InitProcessing); + + /* + * Create a per-backend PGPROC struct in shared memory. We must do this + * before we access any shared memory. + */ + InitProcess(); + + /* + * Early initialization. + */ + BaseInit(); + + Assert(SlotSyncWorker != NULL); + + SpinLockAcquire(&SlotSyncWorker->mutex); + Assert(SlotSyncWorker->pid == InvalidPid); + + /* + * Startup process signaled the slot sync worker to stop, so if meanwhile + * postmaster ended up starting the worker again, exit. + */ + if (SlotSyncWorker->stopSignaled) + { + SpinLockRelease(&SlotSyncWorker->mutex); + proc_exit(0); + } + + /* Advertise our PID so that the startup process can kill us on promotion */ + SlotSyncWorker->pid = MyProcPid; + SpinLockRelease(&SlotSyncWorker->mutex); + + ereport(LOG, errmsg("replication slot sync worker started")); + + on_shmem_exit(slotsync_worker_onexit, (Datum) 0); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGINT, SignalHandlerForShutdownRequest); + pqsignal(SIGTERM, die); + pqsignal(SIGFPE, FloatExceptionHandler); + pqsignal(SIGUSR1, procsignal_sigusr1_handler); + pqsignal(SIGUSR2, SIG_IGN); + pqsignal(SIGPIPE, SIG_IGN); + pqsignal(SIGCHLD, SIG_DFL); + + /* + * Establishes SIGALRM handler and initialize timeout module. It is needed + * by InitPostgres to register different timeouts. + */ + InitializeTimeouts(); + + /* Load the libpq-specific functions */ + load_file("libpqwalreceiver", false); + + /* + * If an exception is encountered, processing resumes here. + * + * We just need to clean up, report the error, and go away. + * + * If we do not have this handling here, then since this worker process + * operates at the bottom of the exception stack, ERRORs turn into FATALs. + * Therefore, we create our own exception handler to catch ERRORs. + */ + if (sigsetjmp(local_sigjmp_buf, 1) != 0) + { + /* since not using PG_TRY, must reset error stack by hand */ + error_context_stack = NULL; + + /* Prevents interrupts while cleaning up */ + HOLD_INTERRUPTS(); + + /* Report the error to the server log */ + EmitErrorReport(); + + /* + * We can now go away. Note that because we called InitProcess, a + * callback was registered to do ProcKill, which will clean up + * necessary state. + */ + proc_exit(0); + } + + /* We can now handle ereport(ERROR) */ + PG_exception_stack = &local_sigjmp_buf; + + /* + * Unblock signals (they were blocked when the postmaster forked us) + */ + sigprocmask(SIG_SETMASK, &UnBlockSig, NULL); + + ensure_valid_slotsync_params(); + + dbname = get_dbname_from_conninfo(PrimaryConnInfo); + + /* + * Connect to the database specified by user in primary_conninfo. We need + * a database connection for walrcv_exec to work. Please see comments atop + * libpqrcv_exec. + */ + InitPostgres(dbname, InvalidOid, NULL, InvalidOid, 0, NULL); + + SetProcessingMode(NormalProcessing); + + initStringInfo(&app_name); + if (cluster_name[0]) + appendStringInfo(&app_name, "%s_%s", cluster_name, "slotsyncworker"); + else + appendStringInfo(&app_name, "%s", "slotsyncworker"); + + /* + * Establish the connection to the primary server for slots + * synchronization. + */ + wrconn = walrcv_connect(PrimaryConnInfo, false, false, false, + app_name.data, + &err); + pfree(app_name.data); + + if (!wrconn) + ereport(ERROR, + errcode(ERRCODE_CONNECTION_FAILURE), + errmsg("could not connect to the primary server: %s", err)); + + before_shmem_exit(slotsync_failure_callback, PointerGetDatum(wrconn)); + + /* + * Using the specified primary server connection, check whether we are + * cascading standby and validates primary_slot_name for + * non-cascading-standbys. + */ + remote_info_valid = validate_remote_info(wrconn, LOG); + + /* Main wait loop */ + for (;;) + { + bool some_slot_updated = false; + + ProcessSlotSyncInterrupts(wrconn, true); + + if (remote_info_valid) + some_slot_updated = synchronize_slots(wrconn); + + wait_for_slot_activity(some_slot_updated, !remote_info_valid); + + /* + * If the standby was promoted then what was previously a cascading + * standby might no longer be one, so recheck each time. + */ + if (!remote_info_valid) + validate_remote_info(wrconn, LOG); + } + + /* + * The slot sync worker can not get here because it will only stop when it + * receives a SIGINT from the startup process, or when there is an error. + */ + Assert(false); +} + +/* + * Main entry point for slot sync worker process, to be called from the + * postmaster. + */ +int +StartSlotSyncWorker(void) +{ + pid_t pid; + +#ifdef EXEC_BACKEND + switch ((pid = slotsyncworker_forkexec())) + { +#else + switch ((pid = fork_process())) + { + case 0: + /* in postmaster child ... */ + InitPostmasterChild(); + + /* Close the postmaster's sockets */ + ClosePostmasterPorts(false); + + ReplSlotSyncWorkerMain(0, NULL); + break; +#endif + case -1: + ereport(LOG, + (errmsg("could not fork slot sync worker process: %m"))); + return 0; + + default: + return (int) pid; + } + + /* shouldn't get here */ + return 0; +} + +#ifdef EXEC_BACKEND +/* + * The forkexec routine for the slot sync worker process. + * + * Format up the arglist, then fork and exec. + */ +static pid_t +slotsyncworker_forkexec(void) +{ + char *av[10]; + int ac = 0; + + av[ac++] = "postgres"; + av[ac++] = "--forkssworker"; + av[ac++] = NULL; /* filled in by postmaster_forkexec */ + av[ac] = NULL; + + Assert(ac < lengthof(av)); + + return postmaster_forkexec(ac, av); +} +#endif + +/* + * Shut down the slot sync worker. + */ +void +ShutDownSlotSync(void) +{ + SpinLockAcquire(&SlotSyncWorker->mutex); + + SlotSyncWorker->stopSignaled = true; + + if (SlotSyncWorker->pid == InvalidPid) + { + SpinLockRelease(&SlotSyncWorker->mutex); + return; + } + SpinLockRelease(&SlotSyncWorker->mutex); + + kill(SlotSyncWorker->pid, SIGINT); + + /* Wait for it to die */ + for (;;) + { + int rc; + + /* Wait a bit, we don't expect to have to wait long */ + rc = WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, WAIT_EVENT_REPL_SLOTSYNC_SHUTDOWN); + + if (rc & WL_LATCH_SET) + { + ResetLatch(MyLatch); + CHECK_FOR_INTERRUPTS(); + } + + SpinLockAcquire(&SlotSyncWorker->mutex); + + /* Is it gone? */ + if (SlotSyncWorker->pid == InvalidPid) + break; + + SpinLockRelease(&SlotSyncWorker->mutex); + } + + SpinLockRelease(&SlotSyncWorker->mutex); +} + +/* + * SlotSyncWorkerCanRestart + * + * Returns true if the worker is allowed to restart if enough time has + * passed (SLOTSYNC_RESTART_INTERVAL_SEC) since it was launched last. + * Otherwise returns false. + * + * This is a safety valve to protect against continuous respawn attempts if the + * worker is dying immediately at launch. Note that since we will retry to + * launch the worker from the postmaster main loop, we will get another + * chance later. + */ +bool +SlotSyncWorkerCanRestart(void) +{ + time_t curtime = time(NULL); + + /* Return false if too soon since last start. */ + if ((unsigned int) (curtime - SlotSyncWorker->last_start_time) < + (unsigned int) SLOTSYNC_RESTART_INTERVAL_SEC) + return false; + + SlotSyncWorker->last_start_time = curtime; + + return true; } /* * Is current process syncing replication slots ? */ bool -IsSyncingReplicationSlots(void) +IsLogicalSlotSyncWorker(void) { - return syncing_slots; + return am_slotsync_worker | syncing_slots; } /* * Amount of shared memory required for slot synchronization. */ Size -SlotSyncShmemSize(void) +SlotSyncWorkerShmemSize(void) { - return sizeof(SlotSyncCtxStruct); + return sizeof(SlotSyncWorkerCtxStruct); } /* * Allocate and initialize the shared memory of slot synchronization. */ void -SlotSyncShmemInit(void) +SlotSyncWorkerShmemInit(void) { + Size size = SlotSyncWorkerShmemSize(); bool found; - SlotSyncCtx = (SlotSyncCtxStruct *) - ShmemInitStruct("Slot Sync Data", SlotSyncShmemSize(), &found); + SlotSyncWorker = (SlotSyncWorkerCtxStruct *) + ShmemInitStruct("Slot Sync Worker Data", size, &found); if (!found) { - SlotSyncCtx->syncing = false; - SpinLockInit(&SlotSyncCtx->mutex); + memset(SlotSyncWorker, 0, size); + SlotSyncWorker->pid = InvalidPid; + SpinLockInit(&SlotSyncWorker->mutex); } } @@ -879,9 +1526,9 @@ slotsync_failure_callback(int code, Datum arg) * without resetting the flag. So, we need to clean up shared memory * and reset the flag here. */ - SpinLockAcquire(&SlotSyncCtx->mutex); - SlotSyncCtx->syncing = false; - SpinLockRelease(&SlotSyncCtx->mutex); + SpinLockAcquire(&SlotSyncWorker->mutex); + SlotSyncWorker->syncing = false; + SpinLockRelease(&SlotSyncWorker->mutex); syncing_slots = false; } @@ -898,7 +1545,7 @@ SyncReplicationSlots(WalReceiverConn *wrconn) { PG_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); { - validate_remote_info(wrconn); + validate_remote_info(wrconn, ERROR); synchronize_slots(wrconn); } diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index 2180a38063..db4b5a06f1 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -274,7 +274,7 @@ ReplicationSlotCreate(const char *name, bool db_specific, * synchronization because we need to retain the same values as the * remote slot. */ - if (RecoveryInProgress() && !IsSyncingReplicationSlots()) + if (RecoveryInProgress() && !IsLogicalSlotSyncWorker()) ereport(ERROR, errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot enable failover for a replication slot created on the standby")); @@ -286,7 +286,7 @@ ReplicationSlotCreate(const char *name, bool db_specific, * However, failover enabled temporary slots can be created during * slot synchronization. See the comments atop slotsync.c for details. */ - if (persistency == RS_TEMPORARY && !IsSyncingReplicationSlots()) + if (persistency == RS_TEMPORARY && !IsLogicalSlotSyncWorker()) ereport(ERROR, errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot enable failover for a temporary replication slot")); @@ -1236,6 +1236,20 @@ restart: * concurrently being dropped by a backend connected to another DB. * * That's fairly unlikely in practice, so we'll just bail out. + * + * The slot sync worker holds a shared lock on the database before + * operating on synced logical slots to avoid conflict with the drop + * happening here. The persistent synced slots are thus safe but there + * is a possibility that the slot sync worker has created a temporary + * slot (which stays active even on release) and we are trying to drop + * the same here. In practice, the chances of hitting this scenario is + * very less as during slot synchronization, the temporary slot is + * immediately converted to persistent and thus is safe due to the + * shared lock taken on the database. So for the time being, we'll + * just bail out in such a scenario. + * + * XXX: If needed, we can consider shutting down slot sync worker + * before trying to drop synced temporary slots here. */ if (active_pid) ereport(ERROR, diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index d2fa5e669a..c4a0bf99ee 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -975,7 +975,7 @@ pg_sync_replication_slots(PG_FUNCTION_ARGS) /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); - ValidateSlotSyncParams(); + ValidateSlotSyncParams(ERROR); initStringInfo(&app_name); if (cluster_name[0]) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index e5477c1de1..96898bc4ee 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -3404,7 +3404,7 @@ GetStandbyFlushRecPtr(TimeLineID *tli) TimeLineID receiveTLI; XLogRecPtr result; - Assert(am_cascading_walsender || IsSyncingReplicationSlots()); + Assert(am_cascading_walsender || IsLogicalSlotSyncWorker()); /* * We can safely send what's already been replayed. Also, if walreceiver diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index 7e7941d625..8ecd89f8ea 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -154,7 +154,7 @@ CalculateShmemSize(int *num_semaphores) size = add_size(size, StatsShmemSize()); size = add_size(size, WaitEventExtensionShmemSize()); size = add_size(size, InjectionPointShmemSize()); - size = add_size(size, SlotSyncShmemSize()); + size = add_size(size, SlotSyncWorkerShmemSize()); #ifdef EXEC_BACKEND size = add_size(size, ShmemBackendArraySize()); #endif @@ -349,7 +349,7 @@ CreateOrAttachShmemStructs(void) WalSummarizerShmemInit(); PgArchShmemInit(); ApplyLauncherShmemInit(); - SlotSyncShmemInit(); + SlotSyncWorkerShmemInit(); /* * Set up other modules that need some shared memory space diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c index e5977548fe..937e626040 100644 --- a/src/backend/storage/lmgr/proc.c +++ b/src/backend/storage/lmgr/proc.c @@ -40,6 +40,7 @@ #include "pgstat.h" #include "postmaster/autovacuum.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/syncrep.h" #include "replication/walsender.h" #include "storage/condition_variable.h" @@ -364,8 +365,12 @@ InitProcess(void) * child; this is so that the postmaster can detect it if we exit without * cleaning up. (XXX autovac launcher currently doesn't participate in * this; it probably should.) + * + * Slot sync worker also does not participate in it, see comments atop + * 'struct bkend' in postmaster.c. */ - if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess()) + if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && + !IsLogicalSlotSyncWorker()) MarkPostmasterChildActive(); /* @@ -934,8 +939,12 @@ ProcKill(int code, Datum arg) * This process is no longer present in shared memory in any meaningful * way, so tell the postmaster we've cleaned up acceptably well. (XXX * autovac launcher should be included here someday) + * + * Slot sync worker is also not a postmaster child, so skip this shared + * memory related processing here. */ - if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess()) + if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && + !IsLogicalSlotSyncWorker()) MarkPostmasterChildInactive(); /* wake autovac launcher if needed -- see comments in FreeWorkerInfo */ diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 43c393d6fe..9d6e067382 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -338,6 +338,7 @@ pgstat_tracks_io_bktype(BackendType bktype) case B_BG_WORKER: case B_BG_WRITER: case B_CHECKPOINTER: + case B_SLOTSYNC_WORKER: case B_STANDALONE_BACKEND: case B_STARTUP: case B_WAL_SENDER: diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 6464386b77..b52afd4eac 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -53,6 +53,8 @@ LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process." LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process." LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process." RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery." +REPL_SLOTSYNC_MAIN "Waiting in main loop of slot sync worker." +REPL_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down." SYSLOGGER_MAIN "Waiting in main loop of syslogger process." WAL_RECEIVER_MAIN "Waiting in main loop of WAL receiver process." WAL_SENDER_MAIN "Waiting in main loop of WAL sender process." diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c index 23f77a59e5..26aaf292c5 100644 --- a/src/backend/utils/init/miscinit.c +++ b/src/backend/utils/init/miscinit.c @@ -40,6 +40,7 @@ #include "postmaster/interrupt.h" #include "postmaster/pgarch.h" #include "postmaster/postmaster.h" +#include "replication/slotsync.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/latch.h" @@ -293,6 +294,9 @@ GetBackendTypeDesc(BackendType backendType) case B_LOGGER: backendDesc = "logger"; break; + case B_SLOTSYNC_WORKER: + backendDesc = "slotsyncworker"; + break; case B_STANDALONE_BACKEND: backendDesc = "standalone backend"; break; @@ -835,9 +839,10 @@ InitializeSessionUserIdStandalone(void) { /* * This function should only be called in single-user mode, in autovacuum - * workers, and in background workers. + * workers, in slot sync worker and in background workers. */ - Assert(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || IsBackgroundWorker); + Assert(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || + IsLogicalSlotSyncWorker() || IsBackgroundWorker); /* call only once */ Assert(!OidIsValid(AuthenticatedUserId)); diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 1ad3367159..e21cc59e32 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -43,6 +43,7 @@ #include "postmaster/autovacuum.h" #include "postmaster/postmaster.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/walsender.h" #include "storage/bufmgr.h" #include "storage/fd.h" @@ -874,10 +875,11 @@ InitPostgres(const char *in_dbname, Oid dboid, * Perform client authentication if necessary, then figure out our * postgres user ID, and see if we are a superuser. * - * In standalone mode and in autovacuum worker processes, we use a fixed - * ID, otherwise we figure it out from the authenticated user name. + * In standalone mode, autovacuum worker processes and slot sync worker + * process, we use a fixed ID, otherwise we figure it out from the + * authenticated user name. */ - if (bootstrap || IsAutoVacuumWorkerProcess()) + if (bootstrap || IsAutoVacuumWorkerProcess() || IsLogicalSlotSyncWorker()) { InitializeSessionUserIdStandalone(); am_superuser = true; diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 7fe58518d7..83136e35a6 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -67,6 +67,7 @@ #include "postmaster/walwriter.h" #include "replication/logicallauncher.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/syncrep.h" #include "storage/bufmgr.h" #include "storage/large_object.h" @@ -2054,6 +2055,15 @@ struct config_bool ConfigureNamesBool[] = NULL, NULL, NULL }, + { + {"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY, + gettext_noop("Enables a physical standby to synchronize logical failover slots from the primary server."), + }, + &sync_replication_slots, + false, + NULL, NULL, NULL + }, + /* End-of-list marker */ { {NULL, 0, 0, NULL, NULL}, NULL, false, NULL, NULL, NULL diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index da10b43dac..dfd1313c94 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -361,6 +361,7 @@ #wal_retrieve_retry_interval = 5s # time to wait before retrying to # retrieve WAL after a failed attempt #recovery_min_apply_delay = 0 # minimum delay for applying changes during recovery +#sync_replication_slots = off # enables slot synchronization on the physical standby from the primary # - Subscribers - diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index 0b01c1f093..65819cb7a7 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -332,6 +332,7 @@ typedef enum BackendType B_BG_WRITER, B_CHECKPOINTER, B_LOGGER, + B_SLOTSYNC_WORKER, B_STANDALONE_BACKEND, B_STARTUP, B_WAL_RECEIVER, diff --git a/src/include/replication/slotsync.h b/src/include/replication/slotsync.h index e86d8a47b8..30f03c1305 100644 --- a/src/include/replication/slotsync.h +++ b/src/include/replication/slotsync.h @@ -14,10 +14,27 @@ #include "replication/walreceiver.h" -extern void ValidateSlotSyncParams(void); -extern bool IsSyncingReplicationSlots(void); -extern Size SlotSyncShmemSize(void); -extern void SlotSyncShmemInit(void); +extern PGDLLIMPORT bool sync_replication_slots; + +/* + * GUCs needed by slot sync worker to connect to the primary + * server and carry on with slots synchronization. + */ +extern PGDLLIMPORT char *PrimaryConnInfo; +extern PGDLLIMPORT char *PrimarySlotName; + +extern bool ValidateSlotSyncParams(int elevel); + +#ifdef EXEC_BACKEND +extern void ReplSlotSyncWorkerMain(int argc, char *argv[]) pg_attribute_noreturn(); +#endif +extern int StartSlotSyncWorker(void); + +extern void ShutDownSlotSync(void); +extern bool SlotSyncWorkerCanRestart(void); +extern bool IsLogicalSlotSyncWorker(void); +extern Size SlotSyncWorkerShmemSize(void); +extern void SlotSyncWorkerShmemInit(void); extern void SyncReplicationSlots(WalReceiverConn *wrconn); #endif /* SLOTSYNC_H */ diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index 7ad5f2eb11..e964e5b4c7 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -18,7 +18,8 @@ $publisher->init(allows_streaming => 'logical'); $publisher->start; $publisher->safe_psql('postgres', - "CREATE PUBLICATION regress_mypub FOR ALL TABLES;"); + "CREATE PUBLICATION regress_mypub FOR ALL TABLES;" +); my $publisher_connstr = $publisher->connstr . ' dbname=postgres'; @@ -305,6 +306,10 @@ ok( $stderr =~ /HINT: 'dbname' must be specified in "primary_conninfo"/, "cannot sync slots if dbname is not specified in primary_conninfo"); +# Add the dbname back to the primary_conninfo for further tests +$standby1->append_conf('postgresql.conf', "primary_conninfo = '$connstr_1 dbname=postgres'"); +$standby1->reload; + ################################################## # Test that we cannot synchronize slots to a cascading standby server. ################################################## @@ -338,4 +343,85 @@ ok( $stderr =~ /ERROR: cannot synchronize replication slots from a standby server/, "cannot sync slots to a cascading standby server"); +$cascading_standby->stop; + +################################################## +# Test to confirm that restart_lsn and confirmed_flush_lsn of the logical slot +# on the primary is synced to the standby via the slot sync worker. +################################################## + +$standby1->append_conf('postgresql.conf', "sync_replication_slots = on"); +$standby1->reload; + +# Insert data on the primary +$primary->safe_psql( + 'postgres', qq[ + CREATE TABLE tab_int (a int PRIMARY KEY); + INSERT INTO tab_int SELECT generate_series(1, 10); +]); + +# Subscribe to the new table data and wait for it to arrive +$subscriber1->safe_psql( + 'postgres', qq[ + CREATE TABLE tab_int (a int PRIMARY KEY); + ALTER SUBSCRIPTION regress_mysub1 ENABLE; + ALTER SUBSCRIPTION regress_mysub1 REFRESH PUBLICATION; +]); + +$subscriber1->wait_for_subscription_sync; + +# Do not allow any further advancement of the restart_lsn and +# confirmed_flush_lsn for the lsub1_slot. +$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); + +# Wait for the replication slot to become inactive on the publisher +$primary->poll_query_until( + 'postgres', + "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active='f'", + 1); + +# Get the restart_lsn for the logical slot lsub1_slot on the primary +my $primary_restart_lsn = $primary->safe_psql('postgres', + "SELECT restart_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot';"); + +# Get the confirmed_flush_lsn for the logical slot lsub1_slot on the primary +my $primary_flush_lsn = $primary->safe_psql('postgres', + "SELECT confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot';"); + +# Confirm that restart_lsn and of confirmed_flush_lsn lsub1_slot slot are synced +# to the standby +ok( $standby1->poll_query_until( + 'postgres', + "SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot';"), + 'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby'); + +################################################## +# Promote the standby1 to primary. Confirm that: +# a) the slot 'lsub1_slot' is retained on the new primary +# b) logical replication for regress_mysub1 is resumed successfully after failover +################################################## +$standby1->promote; + +# Update subscription with the new primary's connection info +my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo'; + ALTER SUBSCRIPTION regress_mysub1 ENABLE; "); + +# Confirm the synced slot 'lsub1_slot' is retained on the new primary +is($standby1->safe_psql('postgres', + q{SELECT slot_name FROM pg_replication_slots WHERE slot_name = 'lsub1_slot';}), + 'lsub1_slot', + 'synced slot retained on the new primary'); + +# Insert data on the new primary +$standby1->safe_psql('postgres', + "INSERT INTO tab_int SELECT generate_series(11, 20);"); +$standby1->wait_for_catchup('regress_mysub1'); + +# Confirm that data in tab_int replicated on the subscriber +is( $subscriber1->safe_psql('postgres', q{SELECT count(*) FROM tab_int;}), + "20", + 'data replicated from the new primary'); + done_testing(); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index d808aad8b0..a35e399f0c 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2585,7 +2585,7 @@ SlabBlock SlabContext SlabSlot SlotNumber -SlotSyncCtxStruct +SlotSyncWorkerCtxStruct SlruCtl SlruCtlData SlruErrorCause -- 2.30.0.windows.2 [application/octet-stream] v88-0002-Allow-logical-walsenders-to-wait-for-the-physica.patch (43.2K, ../../OS0PR01MB571618061B7774189B3493D7944D2@OS0PR01MB5716.jpnprd01.prod.outlook.com/4-v88-0002-Allow-logical-walsenders-to-wait-for-the-physica.patch) download | inline diff: From 3d8bb4ad2936cabec6e129572b99a3335282e146 Mon Sep 17 00:00:00 2001 From: Hou Zhijie <[email protected]> Date: Thu, 15 Feb 2024 12:43:55 +0800 Subject: [PATCH v88 2/3] Allow logical walsenders to wait for the physical This patch introduces a mechanism to ensure that physical standby servers, which are potential failover candidates, have received and flushed changes before making them visible to subscribers. By doing so, it guarantees that the promoted standby server is not lagging behind the subscribers when a failover is necessary. A new parameter named standby_slot_names is introduced. The logical walsender now guarantees that all local changes are sent and flushed to the standby servers corresponding to the replication slots specified in standby_slot_names before sending those changes to the subscriber. Additionally, The SQL functions pg_logical_slot_get_changes and pg_replication_slot_advance are modified to wait for the replication slots mentioned in standby_slot_names to catch up before returning the changes to the user. --- doc/src/sgml/config.sgml | 24 ++ doc/src/sgml/logicaldecoding.sgml | 8 + .../replication/logical/logicalfuncs.c | 13 + src/backend/replication/logical/slotsync.c | 13 + src/backend/replication/slot.c | 342 +++++++++++++++++- src/backend/replication/slotfuncs.c | 9 + src/backend/replication/walsender.c | 110 +++++- .../utils/activity/wait_event_names.txt | 1 + src/backend/utils/misc/guc_tables.c | 14 + src/backend/utils/misc/postgresql.conf.sample | 2 + src/include/replication/slot.h | 7 + src/include/replication/walsender.h | 1 + src/include/replication/walsender_private.h | 7 + src/include/utils/guc_hooks.h | 3 + src/test/recovery/t/006_logical_decoding.pl | 3 +- .../t/040_standby_failover_slots_sync.pl | 261 +++++++++++-- 16 files changed, 761 insertions(+), 57 deletions(-) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 02d28a0be9..8743c60c11 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4420,6 +4420,30 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows </listitem> </varlistentry> + <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names"> + <term><varname>standby_slot_names</varname> (<type>string</type>) + <indexterm> + <primary><varname>standby_slot_names</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + List of physical slots guarantees that logical replication slots with + failover enabled do not consume changes until those changes are received + and flushed to corresponding physical standbys. If a logical replication + connection is meant to switch to a physical standby after the standby is + promoted, the physical replication slot for the standby should be listed + here. + </para> + <para> + The standbys corresponding to the physical replication slots in + <varname>standby_slot_names</varname> must configure + <literal>sync_replication_slots = true</literal> so they can receive + failover logical slots changes from the primary. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index 930c0fa8a6..73b2abeda5 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -384,6 +384,14 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU must be enabled on the standby. It is also necessary to specify a valid <literal>dbname</literal> in the <link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>. + It's also highly recommended that the said physical replication slot + is named in + <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link> + list on the primary, to prevent the subscriber from consuming changes + faster than the hot standby. But once we configure it, then certain latency + is expected in sending changes to logical subscribers due to wait on + physical replication slots in + <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link> </para> <para> diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c index b0081d3ce5..5ff761dd65 100644 --- a/src/backend/replication/logical/logicalfuncs.c +++ b/src/backend/replication/logical/logicalfuncs.c @@ -30,6 +30,7 @@ #include "replication/decode.h" #include "replication/logical.h" #include "replication/message.h" +#include "replication/walsender.h" #include "storage/fd.h" #include "utils/array.h" #include "utils/builtins.h" @@ -109,6 +110,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin MemoryContext per_query_ctx; MemoryContext oldcontext; XLogRecPtr end_of_wal; + XLogRecPtr wait_for_wal_lsn; LogicalDecodingContext *ctx; ResourceOwner old_resowner = CurrentResourceOwner; ArrayType *arr; @@ -228,6 +230,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin NameStr(MyReplicationSlot->data.plugin), format_procedure(fcinfo->flinfo->fn_oid)))); + if (XLogRecPtrIsInvalid(upto_lsn)) + wait_for_wal_lsn = end_of_wal; + else + wait_for_wal_lsn = Min(upto_lsn, end_of_wal); + + /* + * Wait for specified streaming replication standby servers (if any) + * to confirm receipt of WAL up to wait_for_wal_lsn. + */ + WaitForStandbyConfirmation(wait_for_wal_lsn); + ctx->output_writer_private = p; /* diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 6492582156..727b89239e 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -478,6 +478,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) latestFlushPtr = GetStandbyFlushRecPtr(NULL); if (remote_slot->confirmed_lsn > latestFlushPtr) { + /* + * Can get here only if GUC 'standby_slot_names' on the primary server + * was not configured correctly. + */ elog(am_slotsync_worker ? LOG : ERROR, "skipping slot synchronization as the received slot sync" " LSN %X/%X for slot \"%s\" is ahead of the standby position %X/%X", @@ -854,6 +858,15 @@ validate_remote_info(WalReceiverConn *wrconn, int elevel) remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull)); Assert(!isnull); + /* + * Slot sync is currently not supported on the cascading standby. This is + * because if we allow it, the primary server needs to wait for all the + * cascading standbys, otherwise, logical subscribers can still be ahead + * of one of the cascading standbys which we plan to promote. Thus, to + * avoid this additional complexity, we restrict it for the time being. + * + * XXX: If needed, this can be attempted in future. + */ if (remote_in_recovery) ereport(elevel, errcode(ERRCODE_FEATURE_NOT_SUPPORTED), diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index db4b5a06f1..5bed673046 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -46,13 +46,18 @@ #include "common/string.h" #include "miscadmin.h" #include "pgstat.h" +#include "postmaster/interrupt.h" #include "replication/slotsync.h" #include "replication/slot.h" +#include "replication/walsender_private.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/proc.h" #include "storage/procarray.h" #include "utils/builtins.h" +#include "utils/guc_hooks.h" +#include "utils/memutils.h" +#include "utils/varlena.h" /* * Replication slot on-disk data structure. @@ -99,10 +104,19 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL; /* My backend's replication slot in the shared memory array */ ReplicationSlot *MyReplicationSlot = NULL; -/* GUC variable */ +/* GUC variables */ int max_replication_slots = 10; /* the maximum number of replication * slots */ +/* + * This GUC lists streaming replication standby server slot names that + * logical WAL sender processes will wait for. + */ +char *standby_slot_names; + +/* This is parsed and cached list for raw standby_slot_names. */ +static List *standby_slot_names_list = NIL; + static void ReplicationSlotShmemExit(int code, Datum arg); static void ReplicationSlotDropPtr(ReplicationSlot *slot); @@ -2297,3 +2311,329 @@ GetSlotInvalidationCause(char *conflict_reason) /* Keep compiler quiet */ return RS_INVAL_NONE; } + +/* + * A helper function to validate slots specified in GUC standby_slot_names. + */ +static bool +validate_standby_slots(char **newval) +{ + char *rawname; + List *elemlist; + ListCell *lc; + bool ok; + + /* Need a modifiable copy of string */ + rawname = pstrdup(*newval); + + /* Verify syntax and parse string into a list of identifiers */ + ok = SplitIdentifierString(rawname, ',', &elemlist); + + if (!ok) + GUC_check_errdetail("List syntax is invalid."); + + /* + * If there is a syntax error in the name or if the replication slots' + * data is not initialized yet (i.e., we are in the startup process), skip + * the slot verification. + */ + if (!ok || !ReplicationSlotCtl) + { + pfree(rawname); + list_free(elemlist); + return ok; + } + + foreach(lc, elemlist) + { + char *name = lfirst(lc); + ReplicationSlot *slot; + + slot = SearchNamedReplicationSlot(name, true); + + if (!slot) + { + GUC_check_errdetail("replication slot \"%s\" does not exist", + name); + ok = false; + break; + } + + if (!SlotIsPhysical(slot)) + { + GUC_check_errdetail("\"%s\" is not a physical replication slot", + name); + ok = false; + break; + } + } + + pfree(rawname); + list_free(elemlist); + return ok; +} + +/* + * GUC check_hook for standby_slot_names + */ +bool +check_standby_slot_names(char **newval, void **extra, GucSource source) +{ + if (strcmp(*newval, "") == 0) + return true; + + /* + * "*" is not accepted as in that case primary will not be able to know + * for which all standbys to wait for. Even if we have physical-slots + * info, there is no way to confirm whether there is any standby + * configured for the known physical slots. + */ + if (strcmp(*newval, "*") == 0) + { + GUC_check_errdetail("\"%s\" is not accepted for standby_slot_names", + *newval); + return false; + } + + /* Now verify if the specified slots really exist and have correct type */ + if (!validate_standby_slots(newval)) + return false; + + *extra = guc_strdup(ERROR, *newval); + + return true; +} + +/* + * GUC assign_hook for standby_slot_names + */ +void +assign_standby_slot_names(const char *newval, void *extra) +{ + List *standby_slots; + MemoryContext oldcxt; + char *standby_slot_names_cpy = extra; + + list_free(standby_slot_names_list); + standby_slot_names_list = NIL; + + /* No value is specified for standby_slot_names. */ + if (standby_slot_names_cpy == NULL) + return; + + if (!SplitIdentifierString(standby_slot_names_cpy, ',', &standby_slots)) + { + /* This should not happen if GUC checked check_standby_slot_names. */ + elog(ERROR, "invalid list syntax"); + } + + /* + * Switch to the same memory context under which GUC variables are + * allocated (GUCMemoryContext). + */ + oldcxt = MemoryContextSwitchTo(GetMemoryChunkContext(standby_slot_names_cpy)); + standby_slot_names_list = list_copy(standby_slots); + MemoryContextSwitchTo(oldcxt); +} + +/* + * Return a copy of standby_slot_names_list if the copy flag is set to true, + * otherwise return the original list. + */ +List * +GetStandbySlotList(bool copy) +{ + /* + * Since we do not support syncing slots to cascading standbys, we return + * NIL here if we are running in a standby to indicate that no standby + * slots need to be waited for. + */ + if (RecoveryInProgress()) + return NIL; + + if (copy) + return list_copy(standby_slot_names_list); + else + return standby_slot_names_list; +} + +/* + * Reload the config file and reinitialize the standby slot list if the GUC + * standby_slot_names has changed. + */ +void +RereadConfigAndReInitSlotList(List **standby_slots) +{ + char *pre_standby_slot_names; + + /* + * If we are running on a standby, there is no need to reload + * standby_slot_names since we do not support syncing slots to cascading + * standbys. + */ + if (RecoveryInProgress()) + { + ProcessConfigFile(PGC_SIGHUP); + return; + } + + pre_standby_slot_names = pstrdup(standby_slot_names); + + ProcessConfigFile(PGC_SIGHUP); + + if (strcmp(pre_standby_slot_names, standby_slot_names) != 0) + { + list_free(*standby_slots); + *standby_slots = GetStandbySlotList(true); + } + + pfree(pre_standby_slot_names); +} + +/* + * Filter the standby slots based on the specified log sequence number + * (wait_for_lsn). + * + * This function updates the passed standby_slots list, removing any slots that + * have already caught up to or surpassed the given wait_for_lsn. Additionally, + * it removes slots that have been invalidated, dropped, or converted to + * logical slots. + */ +void +FilterStandbySlots(XLogRecPtr wait_for_lsn, List **standby_slots) +{ + ListCell *lc; + List *standby_slots_cpy = *standby_slots; + + foreach(lc, standby_slots_cpy) + { + char *name = lfirst(lc); + char *warningfmt = NULL; + ReplicationSlot *slot; + + slot = SearchNamedReplicationSlot(name, true); + + if (!slot) + { + /* + * It may happen that the slot specified in standby_slot_names GUC + * value is dropped, so let's skip over it. + */ + warningfmt = _("replication slot \"%s\" specified in parameter \"%s\" does not exist, ignoring"); + } + else if (SlotIsLogical(slot)) + { + /* + * If a logical slot name is provided in standby_slot_names, issue + * a WARNING and skip it. Although logical slots are disallowed in + * the GUC check_hook(validate_standby_slots), it is still + * possible for a user to drop an existing physical slot and + * recreate a logical slot with the same name. Since it is + * harmless, a WARNING should be enough, no need to error-out. + */ + warningfmt = _("cannot have logical replication slot \"%s\" in parameter \"%s\", ignoring"); + } + else + { + SpinLockAcquire(&slot->mutex); + + if (slot->data.invalidated != RS_INVAL_NONE) + { + /* + * Specified physical slot have been invalidated, so no point + * in waiting for it. + */ + warningfmt = _("physical slot \"%s\" specified in parameter \"%s\" has been invalidated, ignoring"); + } + else if (XLogRecPtrIsInvalid(slot->data.restart_lsn) || + slot->data.restart_lsn < wait_for_lsn) + { + bool inactive = (slot->active_pid == 0); + + SpinLockRelease(&slot->mutex); + + /* Log warning if no active_pid for this physical slot */ + if (inactive) + ereport(WARNING, + errmsg("replication slot \"%s\" specified in parameter \"%s\" does not have active_pid", + name, "standby_slot_names"), + errdetail("Logical replication is waiting on the " + "standby associated with \"%s\".", name), + errhint("Consider starting standby associated with " + "\"%s\" or amend standby_slot_names.", name)); + + /* Continue if the current slot hasn't caught up. */ + continue; + } + else + { + Assert(slot->data.restart_lsn >= wait_for_lsn); + } + + SpinLockRelease(&slot->mutex); + } + + /* + * Reaching here indicates that either the slot has passed the + * wait_for_lsn or there is an issue with the slot that requires a + * warning to be reported. + */ + if (warningfmt) + ereport(WARNING, errmsg(warningfmt, name, "standby_slot_names")); + + standby_slots_cpy = foreach_delete_current(standby_slots_cpy, lc); + } + + *standby_slots = standby_slots_cpy; +} + +/* + * Wait for physical standby to confirm receiving the given lsn. + * + * Used by logical decoding SQL functions that acquired slot with failover + * enabled. It waits for physical standbys corresponding to the physical slots + * specified in the standby_slot_names GUC. + */ +void +WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn) +{ + List *standby_slots; + + if (!MyReplicationSlot->data.failover) + return; + + standby_slots = GetStandbySlotList(true); + + if (standby_slots == NIL) + return; + + ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv); + + for (;;) + { + CHECK_FOR_INTERRUPTS(); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + RereadConfigAndReInitSlotList(&standby_slots); + } + + FilterStandbySlots(wait_for_lsn, &standby_slots); + + /* Exit if done waiting for every slot. */ + if (standby_slots == NIL) + break; + + /* + * We wait for the slots in the standby_slot_names to catch up, but we + * use a timeout so we can also check the if the standby_slot_names + * has been changed. + */ + ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000, + WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION); + } + + ConditionVariableCancelSleep(); + list_free(standby_slots); +} diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index c4a0bf99ee..4f0a1fddc2 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -22,6 +22,7 @@ #include "replication/logical.h" #include "replication/slot.h" #include "replication/slotsync.h" +#include "replication/walsender.h" #include "utils/builtins.h" #include "utils/guc.h" #include "utils/inval.h" @@ -476,6 +477,8 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto) * crash, but this makes the data consistent after a clean shutdown. */ ReplicationSlotMarkDirty(); + + PhysicalWakeupLogicalWalSnd(); } return retlsn; @@ -516,6 +519,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto) .segment_close = wal_segment_close), NULL, NULL, NULL); + /* + * Wait for specified streaming replication standby servers (if any) + * to confirm receipt of WAL up to moveto lsn. + */ + WaitForStandbyConfirmation(moveto); + /* * Start reading at the slot's restart_lsn, which we know to point to * a valid record. diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 96898bc4ee..f88b865593 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1728,27 +1728,78 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId ProcessPendingWrites(); } +/* + * Wake up the logical walsender processes with failover-enabled slots if the + * currently acquired physical slot is specified in standby_slot_names + * GUC. + */ +void +PhysicalWakeupLogicalWalSnd(void) +{ + ListCell *lc; + List *standby_slots; + + Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot)); + + standby_slots = GetStandbySlotList(false); + + foreach(lc, standby_slots) + { + char *name = lfirst(lc); + + if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0) + { + ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv); + return; + } + } +} + /* * Wait till WAL < loc is flushed to disk so it can be safely sent to client. * - * Returns end LSN of flushed WAL. Normally this will be >= loc, but - * if we detect a shutdown request (either from postmaster or client) - * we will return early, so caller must always check. + * If the walsender holds a logical slot that has enabled failover, we also + * wait for all the specified streaming replication standby servers to + * confirm receipt of WAL up to RecentFlushPtr. + * + * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we + * detect a shutdown request (either from postmaster or client) we will return + * early, so caller must always check. */ static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc) { int wakeEvents; + bool wait_for_standby = false; + uint32 wait_event; + List *standby_slots = NIL; static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr; + if (MyReplicationSlot->data.failover && replication_active) + standby_slots = GetStandbySlotList(true); + /* - * Fast path to avoid acquiring the spinlock in case we already know we - * have enough WAL available. This is particularly interesting if we're - * far behind. + * Check if all the standby servers have confirmed receipt of WAL up to + * RecentFlushPtr even when we already know we have enough WAL available. + * + * Note that we cannot directly return without checking the status of + * standby servers because the standby_slot_names may have changed, which + * means there could be new standby slots in the list that have not yet + * caught up to the RecentFlushPtr. */ - if (RecentFlushPtr != InvalidXLogRecPtr && - loc <= RecentFlushPtr) - return RecentFlushPtr; + if (!XLogRecPtrIsInvalid(RecentFlushPtr) && loc <= RecentFlushPtr) + { + FilterStandbySlots(RecentFlushPtr, &standby_slots); + + /* + * Fast path to avoid acquiring the spinlock in case we already know + * we have enough WAL available and all the standby servers have + * confirmed receipt of WAL up to RecentFlushPtr. This is particularly + * interesting if we're far behind. + */ + if (standby_slots == NIL) + return RecentFlushPtr; + } /* Get a more recent flush pointer. */ if (!RecoveryInProgress()) @@ -1769,7 +1820,7 @@ WalSndWaitForWal(XLogRecPtr loc) if (ConfigReloadPending) { ConfigReloadPending = false; - ProcessConfigFile(PGC_SIGHUP); + RereadConfigAndReInitSlotList(&standby_slots); SyncRepInitConfig(); } @@ -1784,8 +1835,18 @@ WalSndWaitForWal(XLogRecPtr loc) if (got_STOPPING) XLogBackgroundFlush(); + /* + * Update the standby slots that have not yet caught up to the flushed + * position. It is good to wait up to RecentFlushPtr and then let it + * send the changes to logical subscribers one by one which are + * already covered in RecentFlushPtr without needing to wait on every + * change for standby confirmation. + */ + if (wait_for_standby) + FilterStandbySlots(RecentFlushPtr, &standby_slots); + /* Update our idea of the currently flushed position. */ - if (!RecoveryInProgress()) + else if (!RecoveryInProgress()) RecentFlushPtr = GetFlushRecPtr(NULL); else RecentFlushPtr = GetXLogReplayRecPtr(NULL); @@ -1813,9 +1874,18 @@ WalSndWaitForWal(XLogRecPtr loc) !waiting_for_ping_response) WalSndKeepalive(false, InvalidXLogRecPtr); - /* check whether we're done */ - if (loc <= RecentFlushPtr) + if (loc > RecentFlushPtr) + wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL; + else if (standby_slots) + { + wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION; + wait_for_standby = true; + } + else + { + /* Already caught up and doesn't need to wait for standby_slots. */ break; + } /* Waiting for new WAL. Since we need to wait, we're now caught up. */ WalSndCaughtUp = true; @@ -1855,9 +1925,11 @@ WalSndWaitForWal(XLogRecPtr loc) if (pq_is_send_pending()) wakeEvents |= WL_SOCKET_WRITEABLE; - WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL); + WalSndWait(wakeEvents, sleeptime, wait_event); } + list_free(standby_slots); + /* reactivate latch so WalSndLoop knows to continue */ SetLatch(MyLatch); return RecentFlushPtr; @@ -2265,6 +2337,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn) { ReplicationSlotMarkDirty(); ReplicationSlotsComputeRequiredLSN(); + PhysicalWakeupLogicalWalSnd(); } /* @@ -3538,6 +3611,7 @@ WalSndShmemInit(void) ConditionVariableInit(&WalSndCtl->wal_flush_cv); ConditionVariableInit(&WalSndCtl->wal_replay_cv); + ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv); } } @@ -3607,8 +3681,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event) * * And, we use separate shared memory CVs for physical and logical * walsenders for selective wake ups, see WalSndWakeup() for more details. + * + * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV + * until awakened by physical walsenders after the walreceiver confirms + * the receipt of the LSN. */ - if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL) + if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION) + ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv); + else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL) ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv); else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL) ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index b52afd4eac..2f99649581 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server." LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server." SSL_OPEN_SERVER "Waiting for SSL while attempting connection." +WAIT_FOR_STANDBY_CONFIRMATION "Waiting for the WAL to be received by physical standby." WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process." WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process." diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 83136e35a6..cf2537235e 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -4628,6 +4628,20 @@ struct config_string ConfigureNamesString[] = check_debug_io_direct, assign_debug_io_direct, NULL }, + { + {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY, + gettext_noop("Lists streaming replication standby server slot " + "names that logical WAL sender processes will wait for."), + gettext_noop("Decoded changes are sent out to plugins by logical " + "WAL sender processes only after specified " + "replication slots confirm receiving WAL."), + GUC_LIST_INPUT | GUC_LIST_QUOTE + }, + &standby_slot_names, + "", + check_standby_slot_names, assign_standby_slot_names, NULL + }, + /* End-of-list marker */ { {NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index dfd1313c94..399602b06e 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -334,6 +334,8 @@ # method to choose sync standbys, number of sync standbys, # and comma-separated list of application_name # from standby(s); '*' = all +#standby_slot_names = '' # streaming replication standby server slot names that + # logical walsender processes will wait for # - Standby Servers - diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index e706ca834c..7b754a1f48 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -229,6 +229,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot; /* GUCs */ extern PGDLLIMPORT int max_replication_slots; +extern PGDLLIMPORT char *standby_slot_names; /* shmem initialization functions */ extern Size ReplicationSlotsShmemSize(void); @@ -277,4 +278,10 @@ extern void CheckSlotPermissions(void); extern ReplicationSlotInvalidationCause GetSlotInvalidationCause(char *conflict_reason); +extern List *GetStandbySlotList(bool copy); +extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn); +extern void FilterStandbySlots(XLogRecPtr wait_for_lsn, + List **standby_slots); +extern void RereadConfigAndReInitSlotList(List **standby_slots); + #endif /* SLOT_H */ diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h index 0c3996e926..f2d8297f01 100644 --- a/src/include/replication/walsender.h +++ b/src/include/replication/walsender.h @@ -39,6 +39,7 @@ extern void InitWalSender(void); extern bool exec_replication_command(const char *cmd_string); extern void WalSndErrorCleanup(void); extern void WalSndResourceCleanup(bool isCommit); +extern void PhysicalWakeupLogicalWalSnd(void); extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); extern void WalSndSignals(void); extern Size WalSndShmemSize(void); diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h index 3113e9ea47..0f962b0c72 100644 --- a/src/include/replication/walsender_private.h +++ b/src/include/replication/walsender_private.h @@ -113,6 +113,13 @@ typedef struct ConditionVariable wal_flush_cv; ConditionVariable wal_replay_cv; + /* + * Used by physical walsenders holding slots specified in + * standby_slot_names to wake up logical walsenders holding + * failover-enabled slots when a walreceiver confirms the receipt of LSN. + */ + ConditionVariable wal_confirm_rcv_cv; + WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER]; } WalSndCtlData; diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h index 5300c44f3b..464996b4f0 100644 --- a/src/include/utils/guc_hooks.h +++ b/src/include/utils/guc_hooks.h @@ -162,5 +162,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra, extern void assign_wal_consistency_checking(const char *newval, void *extra); extern bool check_wal_segment_size(int *newval, void **extra, GucSource source); extern void assign_wal_sync_method(int new_wal_sync_method, void *extra); +extern bool check_standby_slot_names(char **newval, void **extra, + GucSource source); +extern void assign_standby_slot_names(const char *newval, void *extra); #endif /* GUC_HOOKS_H */ diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl index 5c7b4ca5e3..85f019774c 100644 --- a/src/test/recovery/t/006_logical_decoding.pl +++ b/src/test/recovery/t/006_logical_decoding.pl @@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'}, undef, 'logical slot was actually dropped with DB'); # Test logical slot advancing and its durability. +# Pass failover=true (last-arg), it should not have any impact on advancing. my $logical_slot = 'logical_slot'; $node_primary->safe_psql('postgres', - "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);" + "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);" ); $node_primary->psql( 'postgres', " diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index e964e5b4c7..a9f60007c1 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -109,19 +109,30 @@ ok( $stderr =~ "cannot sync slots on a non-standby server"); ################################################## -# Test logical failover slots on the standby -# Configure standby1 to replicate and synchronize logical slots configured -# for failover on the primary +# Test primary disallowing specified logical replication slots getting ahead of +# specified physical replication slots. It uses the following set up: # -# failover slot lsub1_slot ->| ----> subscriber1 (connected via logical replication) -# failover slot lsub2_slot | inactive -# primary ---> | -# physical slot sb1_slot --->| ----> standby1 (connected via streaming replication) -# | lsub1_slot, lsub2_slot (synced_slot) +# | ----> standby1 (primary_slot_name = sb1_slot) +# | ----> standby2 (primary_slot_name = sb2_slot) +# primary ----- | +# | ----> subscriber1 (failover = true) +# | ----> subscriber2 (failover = false) +# +# standby_slot_names = 'sb1_slot' +# +# Set up is configured in such a way that the logical slot of subscriber1 is +# enabled failover, thus it will wait for the physical slot of +# standby1(sb1_slot) to catch up before sending decoded changes to subscriber1. ################################################## my $primary = $publisher; my $backup_name = 'backup'; + +$primary->psql('postgres', + q{SELECT pg_create_physical_replication_slot('sb1_slot');}); +$primary->psql('postgres', + q{SELECT pg_create_physical_replication_slot('sb2_slot');}); + $primary->backup($backup_name); # Create a standby @@ -131,38 +142,214 @@ $standby1->init_from_backup( has_streaming => 1, has_restoring => 1); +$standby1->append_conf( + 'postgresql.conf', qq( +primary_slot_name = 'sb1_slot' +)); +$standby1->start; +$primary->wait_for_replay_catchup($standby1); + +# Speed up the slot creation +$primary->safe_psql('postgres', "SELECT pg_log_standby_snapshot()"); + +# Create a slot to save the old xmin info, which speeds up the persistence of +# the newly created slot by obtaining an existing old xmin value. +$standby1->psql('postgres', + "SELECT pg_create_logical_replication_slot('save_xmin', 'test_decoding');"); + +# Create another standby +my $standby2 = PostgreSQL::Test::Cluster->new('standby2'); +$standby2->init_from_backup( + $primary, $backup_name, + has_streaming => 1, + has_restoring => 1); +$standby2->append_conf( + 'postgresql.conf', qq( +primary_slot_name = 'sb2_slot' +)); +$standby2->start; +$primary->wait_for_replay_catchup($standby2); + +# Configure primary to disallow any logical slots that enabled failover from +# getting ahead of specified physical replication slot (sb1_slot). +$primary->append_conf( + 'postgresql.conf', qq( +standby_slot_names = 'sb1_slot' +)); +$primary->reload; + +$primary->safe_psql('postgres', "CREATE TABLE tab_int (a int PRIMARY KEY);"); + +# Create a table and refresh the publication +$subscriber1->safe_psql( + 'postgres', qq[ + CREATE TABLE tab_int (a int PRIMARY KEY); + ALTER SUBSCRIPTION regress_mysub1 REFRESH PUBLICATION WITH (copy_data = false); +]); + +# Create another subscriber node without enabling failover, wait for sync to +# complete +my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2'); +$subscriber2->init; +$subscriber2->start; +$subscriber2->safe_psql( + 'postgres', qq[ + CREATE TABLE tab_int (a int PRIMARY KEY); + CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot, copy_data = false); +]); + +# Stop the standby associated with the specified physical replication slot so +# that the logical replication slot won't receive changes until the standby +# comes up. +$standby1->stop; + +# Create some data on the primary +my $primary_row_count = 10; +$primary->safe_psql('postgres', + "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);"); + +# Wait for the standby that's up and running gets the data from primary +$primary->wait_for_replay_catchup($standby2); +$result = $standby2->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', "standby2 gets data from primary"); + +# Wait for the subscription that's up and running and is not enabled for failover. +# It gets the data from primary without waiting for any standbys. +$publisher->wait_for_catchup('regress_mysub2'); +$result = $subscriber2->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', "subscriber2 gets data from primary"); + +# The subscription that's up and running and is enabled for failover +# doesn't get the data from primary and keeps waiting for the +# standby specified in standby_slot_names. +$result = + $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;"); +is($result, 't', + "subscriber1 doesn't get data from primary until standby1 acknowledges changes" +); + +# Start the standby specified in standby_slot_names and wait for it to catch +# up with the primary. +$standby1->start; +$primary->wait_for_replay_catchup($standby1); +$result = $standby1->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', "standby1 gets data from primary"); + +# Now that the standby specified in standby_slot_names is up and running, +# primary must send the decoded changes to subscription enabled for failover +# While the standby was down, this subscriber didn't receive any data from +# primary i.e. the primary didn't allow it to go ahead of standby. +$publisher->wait_for_catchup('regress_mysub1'); +$result = $subscriber1->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', + "subscriber1 gets data from primary after standby1 acknowledges changes"); + +# Stop the standby associated with the specified physical replication slot so +# that the logical replication slot won't receive changes until the standby +# slot's restart_lsn is advanced or the slot is removed from the +# standby_slot_names list. +$publisher->safe_psql('postgres', "TRUNCATE tab_int;"); +$publisher->wait_for_catchup('regress_mysub1'); +$standby1->stop; + +################################################## +# Verify that when using pg_logical_slot_get_changes to consume changes from a +# logical slot with failover enabled, it will also wait for the slots specified +# in standby_slot_names to catch up. +################################################## + +# Create a logical 'test_decoding' replication slot with failover enabled +$publisher->safe_psql('postgres', + "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);" +); + +my $back_q = $primary->background_psql('postgres', on_error_stop => 0); +my $pid = $back_q->query('SELECT pg_backend_pid()'); + +# Try and get changes from the logical slot with failover enabled. +my $offset = -s $primary->logfile; +$back_q->query_until(qr//, + "SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);\n"); + +# Wait until the primary server logs a warning indicating that it is waiting +# for the sb1_slot to catch up. +$primary->wait_for_log( + qr/WARNING: ( [A-Z0-9]+:)? replication slot \"sb1_slot\" specified in parameter \"standby_slot_names\" does not have active_pid/, + $offset); + +ok($primary->safe_psql('postgres', "SELECT pg_cancel_backend($pid)"), + "cancelling pg_logical_slot_get_changes command"); + +$back_q->quit; + +$publisher->safe_psql('postgres', + "SELECT pg_drop_replication_slot('test_slot');" +); + +################################################## +# Test that logical replication will wait for the user-created inactive +# physical slot to catch up until we remove the slot from standby_slot_names. +################################################## + +# Create some data on the primary +$primary_row_count = 10; +$primary->safe_psql('postgres', + "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);"); + +$result = + $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;"); +is($result, 't', + "subscriber1 doesn't get data as the sb1_slot doesn't catch up"); + +# Remove the standby from the standby_slot_names list and reload the +# configuration. +$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''"); +$primary->reload; + +# Since there are no slots in standby_slot_names, the primary server should now +# send the decoded changes to the subscription. +$publisher->wait_for_catchup('regress_mysub1'); +$result = $subscriber1->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', + "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list" +); + +# Put the standby back on the primary_slot_name for the rest of the tests +$primary->adjust_conf('postgresql.conf', 'standby_slot_names', 'sb1_slot'); +$primary->reload; + +################################################## +# Test logical failover slots on the standby +# Configure standby1 to replicate and synchronize logical slots configured +# for failover on the primary +# +# failover slot lsub1_slot ->| ----> subscriber1 (connected via logical replication) +# failover slot lsub3_slot | inactive +# primary ---> | +# physical slot sb1_slot --->| ----> standby1 (connected via streaming replication) +# | lsub1_slot, lsub3_slot (synced_slot) +################################################## + +# Configure the standby my $connstr_1 = $primary->connstr; $standby1->append_conf( 'postgresql.conf', qq( hot_standby_feedback = on -primary_slot_name = 'sb1_slot' primary_conninfo = '$connstr_1 dbname=postgres' )); $primary->psql('postgres', - q{SELECT pg_create_logical_replication_slot('lsub2_slot', 'test_decoding', false, false, true);} + q{SELECT pg_create_logical_replication_slot('lsub3_slot', 'test_decoding', false, false, true);} ); -$primary->psql('postgres', - q{SELECT pg_create_physical_replication_slot('sb1_slot');}); - # Start the standby so that slot syncing can begin $standby1->start; -$primary->wait_for_catchup('regress_mysub1'); - -# Do not allow any further advancement of the restart_lsn for the lsub1_slot. -$subscriber1->safe_psql('postgres', - "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); - -# Wait for the replication slot to become inactive on the publisher -$primary->poll_query_until( - 'postgres', - "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", - 1); - -# Wait for the standby to catch up so that the standby is not lagging behind -# the subscriber. $primary->wait_for_replay_catchup($standby1); # Synchronize the primary server slots to the standby. @@ -172,7 +359,7 @@ $standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); # flagged as 'synced' is( $standby1->safe_psql( 'postgres', - q{SELECT count(*) = 2 FROM pg_replication_slots WHERE slot_name IN ('lsub1_slot', 'lsub2_slot') AND synced AND NOT temporary;} + q{SELECT count(*) = 2 FROM pg_replication_slots WHERE slot_name IN ('lsub1_slot', 'lsub3_slot') AND synced AND NOT temporary;} ), "t", 'logical slots have synced as true on standby'); @@ -182,13 +369,13 @@ is( $standby1->safe_psql( # slot on the primary server has been dropped. ################################################## -$primary->psql('postgres', "SELECT pg_drop_replication_slot('lsub2_slot');"); +$primary->psql('postgres', "SELECT pg_drop_replication_slot('lsub3_slot');"); $standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); is( $standby1->safe_psql( 'postgres', - q{SELECT count(*) = 0 FROM pg_replication_slots WHERE slot_name = 'lsub2_slot';} + q{SELECT count(*) = 0 FROM pg_replication_slots WHERE slot_name = 'lsub3_slot';} ), "t", 'synchronized slot has been dropped'); @@ -356,19 +543,13 @@ $standby1->reload; # Insert data on the primary $primary->safe_psql( 'postgres', qq[ - CREATE TABLE tab_int (a int PRIMARY KEY); + TRUNCATE TABLE tab_int; INSERT INTO tab_int SELECT generate_series(1, 10); ]); -# Subscribe to the new table data and wait for it to arrive -$subscriber1->safe_psql( - 'postgres', qq[ - CREATE TABLE tab_int (a int PRIMARY KEY); - ALTER SUBSCRIPTION regress_mysub1 ENABLE; - ALTER SUBSCRIPTION regress_mysub1 REFRESH PUBLICATION; -]); - -$subscriber1->wait_for_subscription_sync; +# Enable the subscription to let it catch up to the latest wal position +$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE"); +$primary->wait_for_catchup('regress_mysub1'); # Do not allow any further advancement of the restart_lsn and # confirmed_flush_lsn for the lsub1_slot. -- 2.30.0.windows.2 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-15 12:43 Amit Kapila <[email protected]> parent: Zhijie Hou (Fujitsu) <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Amit Kapila @ 2024-02-15 12:43 UTC (permalink / raw) To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Bertrand Drouvot <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Thu, Feb 15, 2024 at 12:07 PM Zhijie Hou (Fujitsu) <[email protected]> wrote: > > Since the slotsync function is committed, I rebased remaining patches. > And here is the V88 patch set. > Please find the improvements in some of the comments in v88_0001* attached. Kindly include these in next version, if you are okay with it. -- With Regards, Amit Kapila. diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 68f48c052c..6173143bcf 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1469,16 +1469,16 @@ FinishWalRecovery(void) XLogShutdownWalRcv(); /* - * Shutdown the slot sync workers to prevent potential conflicts between - * user processes and slotsync workers after a promotion. + * Shutdown the slot sync worker to prevent it from keep trying to fetch + * slots and drop any temporary slots. * * We do not update the 'synced' column from true to false here, as any * failed update could leave 'synced' column false for some slots. This * could cause issues during slot sync after restarting the server as a - * standby. While updating after switching to the new timeline is an - * option, it does not simplify the handling for 'synced' column. - * Therefore, we retain the 'synced' column as true after promotion as it - * may provide useful information about the slot origin. + * standby. While updating the 'synced' column after switching to the new + * timeline is an option, it does not simplify the handling for the + * 'synced' column. Therefore, we retain the 'synced' column as true after + * promotion as it may provide useful information about the slot origin. */ ShutDownSlotSync(); diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index b0334ce449..011b5f4828 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -168,7 +168,7 @@ * they will never become live backends. dead_end children are not assigned a * PMChildSlot. dead_end children have bkend_type NORMAL. * - * "Special" children such as the startup, bgwriter, autovacuum launcher and + * "Special" children such as the startup, bgwriter, autovacuum launcher, and * slot sync worker tasks are not in this list. They are tracked via StartupPID * and other pid_t variables below. (Thus, there can't be more than one of any * given "special" child process type. We use BackendList entries for any @@ -3415,7 +3415,7 @@ CleanupBackend(int pid, /* * HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer, - * walwriter, autovacuum, archiver, slot sync worker or background worker. + * walwriter, autovacuum, archiver, slot sync worker, or background worker. * * The objectives here are to clean up our local state about the child * process, and to signal all other remaining children to quickdie. diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index e40cdbe4d9..04271ee703 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -6,8 +6,8 @@ * loaded as a dynamic module to avoid linking the main server binary with * libpq. * - * Apart from walreceiver, the libpq-specific routines here are now being used - * by logical replication workers and slot sync worker as well. + * Apart from walreceiver, the libpq-specific routines are now being used by + * logical replication workers and slot synchronization. * * Portions Copyright (c) 2010-2024, PostgreSQL Global Development Group * diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 6492582156..56eb0ea11b 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -22,10 +22,10 @@ * which we can call pg_sync_replication_slots() periodically to perform * syncs. * - * The slot sync worker worker waits for a period of time before the next - * synchronization, with the duration varying based on whether any slots were - * updated during the last cycle. Refer to the comments above - * wait_for_slot_activity() for more details. + * The slot sync worker waits for some time before the next synchronization, + * with the duration varying based on whether any slots were updated during + * the last cycle. Refer to the comments above wait_for_slot_activity() for + * more details. * * Any standby synchronized slots will be dropped if they no longer need * to be synchronized. See comment atop drop_local_obsolete_slots() for more @@ -1449,9 +1449,8 @@ ShutDownSlotSync(void) /* * SlotSyncWorkerCanRestart * - * Returns true if the worker is allowed to restart if enough time has - * passed (SLOTSYNC_RESTART_INTERVAL_SEC) since it was launched last. - * Otherwise returns false. + * Returns true if enough time has passed (SLOTSYNC_RESTART_INTERVAL_SEC) + * since it was launched last. Otherwise returns false. * * This is a safety valve to protect against continuous respawn attempts if the * worker is dying immediately at launch. Note that since we will retry to Attachments: [text/plain] v88_amit_1.patch.txt (4.8K, ../../CAA4eK1LrPcWHgPjFe__tF1z119=Hrw0w463jmv+quOaDzSjctw@mail.gmail.com/2-v88_amit_1.patch.txt) download | inline diff: diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 68f48c052c..6173143bcf 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -1469,16 +1469,16 @@ FinishWalRecovery(void) XLogShutdownWalRcv(); /* - * Shutdown the slot sync workers to prevent potential conflicts between - * user processes and slotsync workers after a promotion. + * Shutdown the slot sync worker to prevent it from keep trying to fetch + * slots and drop any temporary slots. * * We do not update the 'synced' column from true to false here, as any * failed update could leave 'synced' column false for some slots. This * could cause issues during slot sync after restarting the server as a - * standby. While updating after switching to the new timeline is an - * option, it does not simplify the handling for 'synced' column. - * Therefore, we retain the 'synced' column as true after promotion as it - * may provide useful information about the slot origin. + * standby. While updating the 'synced' column after switching to the new + * timeline is an option, it does not simplify the handling for the + * 'synced' column. Therefore, we retain the 'synced' column as true after + * promotion as it may provide useful information about the slot origin. */ ShutDownSlotSync(); diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index b0334ce449..011b5f4828 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -168,7 +168,7 @@ * they will never become live backends. dead_end children are not assigned a * PMChildSlot. dead_end children have bkend_type NORMAL. * - * "Special" children such as the startup, bgwriter, autovacuum launcher and + * "Special" children such as the startup, bgwriter, autovacuum launcher, and * slot sync worker tasks are not in this list. They are tracked via StartupPID * and other pid_t variables below. (Thus, there can't be more than one of any * given "special" child process type. We use BackendList entries for any @@ -3415,7 +3415,7 @@ CleanupBackend(int pid, /* * HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer, - * walwriter, autovacuum, archiver, slot sync worker or background worker. + * walwriter, autovacuum, archiver, slot sync worker, or background worker. * * The objectives here are to clean up our local state about the child * process, and to signal all other remaining children to quickdie. diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index e40cdbe4d9..04271ee703 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -6,8 +6,8 @@ * loaded as a dynamic module to avoid linking the main server binary with * libpq. * - * Apart from walreceiver, the libpq-specific routines here are now being used - * by logical replication workers and slot sync worker as well. + * Apart from walreceiver, the libpq-specific routines are now being used by + * logical replication workers and slot synchronization. * * Portions Copyright (c) 2010-2024, PostgreSQL Global Development Group * diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 6492582156..56eb0ea11b 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -22,10 +22,10 @@ * which we can call pg_sync_replication_slots() periodically to perform * syncs. * - * The slot sync worker worker waits for a period of time before the next - * synchronization, with the duration varying based on whether any slots were - * updated during the last cycle. Refer to the comments above - * wait_for_slot_activity() for more details. + * The slot sync worker waits for some time before the next synchronization, + * with the duration varying based on whether any slots were updated during + * the last cycle. Refer to the comments above wait_for_slot_activity() for + * more details. * * Any standby synchronized slots will be dropped if they no longer need * to be synchronized. See comment atop drop_local_obsolete_slots() for more @@ -1449,9 +1449,8 @@ ShutDownSlotSync(void) /* * SlotSyncWorkerCanRestart * - * Returns true if the worker is allowed to restart if enough time has - * passed (SLOTSYNC_RESTART_INTERVAL_SEC) since it was launched last. - * Otherwise returns false. + * Returns true if enough time has passed (SLOTSYNC_RESTART_INTERVAL_SEC) + * since it was launched last. Otherwise returns false. * * This is a safety valve to protect against continuous respawn attempts if the * worker is dying immediately at launch. Note that since we will retry to ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-15 17:18 Bertrand Drouvot <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Bertrand Drouvot @ 2024-02-15 17:18 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> Hi, On Thu, Feb 15, 2024 at 06:13:38PM +0530, Amit Kapila wrote: > On Thu, Feb 15, 2024 at 12:07 PM Zhijie Hou (Fujitsu) > <[email protected]> wrote: > > > > Since the slotsync function is committed, I rebased remaining patches. > > And here is the V88 patch set. > > Thanks! > > Please find the improvements in some of the comments in v88_0001* > attached. Kindly include these in next version, if you are okay with > it. Looking at v88_0001, random comments: 1 === Commit message "Be enabling slot synchronization" Typo? s:Be/By 2 === + It enables a physical standby to synchronize logical failover slots + from the primary server so that logical subscribers are not blocked + after failover. Not sure "not blocked" is the right wording. "can be resumed from the new primary" maybe? (was discussed in [1]) 3 === +#define SlotSyncWorkerAllowed() \ + (sync_replication_slots && pmState == PM_HOT_STANDBY && \ + SlotSyncWorkerCanRestart()) Maybe add a comment above the macro explaining the logic? 4 === +#include "replication/walreceiver.h" #include "replication/slotsync.h" should be reverse order? 5 === + if (SlotSyncWorker->syncing) { - SpinLockRelease(&SlotSyncCtx->mutex); + SpinLockRelease(&SlotSyncWorker->mutex); ereport(ERROR, errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), errmsg("cannot synchronize replication slots concurrently")); } worth to add a test in 040_standby_failover_slots_sync.pl for it? 6 === +static void +slotsync_reread_config(bool restart) +{ worth to add test(s) in 040_standby_failover_slots_sync.pl for it? [1]: https://www.postgresql.org/message-id/CAA4eK1JcBG6TJ3o5iUd4z0BuTbciLV3dK4aKgb7OgrNGoLcfSQ%40mail.gma... Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-16 10:40 shveta malik <[email protected]> parent: Bertrand Drouvot <[email protected]> 0 siblings, 2 replies; 46+ messages in thread From: shveta malik @ 2024-02-16 10:40 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: Amit Kapila <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>; shveta malik <[email protected]> On Thu, Feb 15, 2024 at 10:48 PM Bertrand Drouvot <[email protected]> wrote: > > Looking at v88_0001, random comments: Thanks for the feedback. > > 1 === > > Commit message "Be enabling slot synchronization" > > Typo? s:Be/By Modified. > 2 === > > + It enables a physical standby to synchronize logical failover slots > + from the primary server so that logical subscribers are not blocked > + after failover. > > Not sure "not blocked" is the right wording. > "can be resumed from the new primary" maybe? (was discussed in [1]) Modified. > 3 === > > +#define SlotSyncWorkerAllowed() \ > + (sync_replication_slots && pmState == PM_HOT_STANDBY && \ > + SlotSyncWorkerCanRestart()) > > Maybe add a comment above the macro explaining the logic? Done. > 4 === > > +#include "replication/walreceiver.h" > #include "replication/slotsync.h" > > should be reverse order? Removed walreceiver.h inclusion as it was not needed. > 5 === > > + if (SlotSyncWorker->syncing) > { > - SpinLockRelease(&SlotSyncCtx->mutex); > + SpinLockRelease(&SlotSyncWorker->mutex); > ereport(ERROR, > errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), > errmsg("cannot synchronize replication slots concurrently")); > } > > worth to add a test in 040_standby_failover_slots_sync.pl for it? It will be very difficult to stabilize this test as we have to make sure that the concurrent users (SQL function(s) and/or worker(s)) are in that target function at the same time to hit it. > > 6 === > > +static void > +slotsync_reread_config(bool restart) > +{ > > worth to add test(s) in 040_standby_failover_slots_sync.pl for it? Added test. Please find v89 patch set. The other changes are: patch001: 1) Addressed some comments by Amit and Ajin given off-list. 2) Removed redundant header inclusions from slotsync.c. 3) Corrected the value returned by validate_remote_info(). 4) Restructured code around validate_remote_info. 5) Improved comments and commit msg. patch002: Rebased it. thanks Shveta Attachments: [application/octet-stream] v89-0003-Document-the-steps-to-check-if-the-standby-is-re.patch (7.0K, ../../CAJpy0uBA4efiWHhS9_FOmXxvdERk0bbMk0qMjo8ZpSTWE4DjVA@mail.gmail.com/2-v89-0003-Document-the-steps-to-check-if-the-standby-is-re.patch) download | inline diff: From f0b421a0e32d73a8aa5721d5038ecc375887ac53 Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Fri, 19 Jan 2024 11:04:16 +0530 Subject: [PATCH v89 3/3] Document the steps to check if the standby is ready for failover --- doc/src/sgml/high-availability.sgml | 9 ++ doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++ 2 files changed, 145 insertions(+) diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml index 236c0af65f..36215aa68c 100644 --- a/doc/src/sgml/high-availability.sgml +++ b/doc/src/sgml/high-availability.sgml @@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)' Written administration procedures are advised. </para> + <para> + If you have opted for synchronization of logical slots (see + <xref linkend="logicaldecoding-replication-slots-synchronization"/>), + then before switching to the standby server, it is recommended to check + if the logical slots synchronized on the standby server are ready + for failover. This can be done by following the steps described in + <xref linkend="logical-replication-failover"/>. + </para> + <para> To trigger failover of a log-shipping standby server, run <command>pg_ctl promote</command> or call <function>pg_promote()</function>. diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index ec2130669e..be59d306a1 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -687,6 +687,142 @@ ALTER SUBSCRIPTION </sect1> + <sect1 id="logical-replication-failover"> + <title>Logical Replication Failover</title> + + <para> + When the publisher server is the primary server of a streaming replication, + the logical slots on that primary server can be synchronized to the standby + server by specifying <literal>failover = true</literal> when creating + subscriptions for those publications. Enabling failover ensures a seamless + transition of those subscriptions after the standby is promoted. They can + continue subscribing to publications now on the new primary server without + any data loss. + </para> + + <para> + Because the slot synchronization logic copies asynchronously, it is + necessary to confirm that replication slots have been synced to the standby + server before the failover happens. Furthermore, to ensure a successful + failover, the standby server must not be lagging behind the subscriber. It + is highly recommended to use + <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link> + to prevent the subscriber from consuming changes faster than the hot standby. + To confirm that the standby server is indeed ready for failover, follow + these 2 steps: + </para> + + <procedure> + <step performance="required"> + <para> + Confirm that all the necessary logical replication slots have been synced to + the standby server. + </para> + <substeps> + <step performance="required"> + <para> + Firstly, on the subscriber node, use the following SQL to identify + which slots should be synced to the standby that we plan to promote. +<programlisting> +test_sub=# SELECT + array_agg(slotname) AS slots + FROM + (( + SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname + FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s + WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover + ) UNION ( + SELECT s.oid AS subid, s.subslotname as slotname + FROM pg_subscription s + WHERE s.subfailover + )); + slots +------- + {sub1,sub2,sub3} +(1 row) +</programlisting></para> + </step> + <step performance="required"> + <para> + Next, check that the logical replication slots identified above exist on + the standby server and are ready for failover. +<programlisting> +test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready + FROM pg_replication_slots + WHERE slot_name IN ('sub1','sub2','sub3'); + slot_name | failover_ready +-------------+---------------- + sub1 | t + sub2 | t + sub3 | t +(3 rows) +</programlisting></para> + </step> + </substeps> + </step> + + <step performance="required"> + <para> + Confirm that the standby server is not lagging behind the subscribers. + This step can be skipped if + <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link> + has been correctly configured. If standby_slot_names is not configured + correctly, it is highly recommended to run this step after the primary + server is down, otherwise the results of the query may vary at different + points of time due to the ongoing replication on the logical subscribers + from the primary server. + </para> + <substeps> + <step performance="required"> + <para> + Firstly, on the subscriber node check the last replayed WAL. + This step needs to be run on the database(s) that includes the failover + enabled subscription(s), to find the last replayed WAL on each database. +<programlisting> +test_sub=# SELECT + MAX(remote_lsn) AS remote_lsn_on_subscriber + FROM + (( + SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false) + WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn + FROM pg_subscription_rel r, pg_subscription s + WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover + ) UNION ( + SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn + FROM pg_subscription s + WHERE s.subfailover + )); + remote_lsn_on_subscriber +-------------------------- + 0/3000388 +</programlisting></para> + </step> + <step performance="required"> + <para> + Next, on the standby server check that the last-received WAL location + is ahead of the replayed WAL location(s) on the subscriber identified + above. If the above SQL result was NULL, it means the subscriber has not + yet replayed any WAL, so the standby server must be ahead of the + subscriber, and this step can be skipped. +<programlisting> +test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready; + failover_ready +---------------- + t +(1 row) +</programlisting></para> + </step> + </substeps> + </step> + </procedure> + + <para> + If the result (<literal>failover_ready</literal>) of both above steps is + true, existing subscriptions will be able to continue without data loss. + </para> + + </sect1> + <sect1 id="logical-replication-row-filter"> <title>Row Filters</title> -- 2.34.1 [application/octet-stream] v89-0002-Allow-logical-walsenders-to-wait-for-the-physica.patch (43.5K, ../../CAJpy0uBA4efiWHhS9_FOmXxvdERk0bbMk0qMjo8ZpSTWE4DjVA@mail.gmail.com/3-v89-0002-Allow-logical-walsenders-to-wait-for-the-physica.patch) download | inline diff: From a8678d75e84fe11f190ba4d68597b3df747d13cc Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Fri, 16 Feb 2024 15:21:48 +0530 Subject: [PATCH v89 2/3] Allow logical walsenders to wait for the physical This patch introduces a mechanism to ensure that physical standby servers, which are potential failover candidates, have received and flushed changes before making them visible to subscribers. By doing so, it guarantees that the promoted standby server is not lagging behind the subscribers when a failover is necessary. A new parameter named standby_slot_names is introduced. The logical walsender now guarantees that all local changes are sent and flushed to the standby servers corresponding to the replication slots specified in standby_slot_names before sending those changes to the subscriber. Additionally, The SQL functions pg_logical_slot_get_changes and pg_replication_slot_advance are modified to wait for the replication slots mentioned in standby_slot_names to catch up before returning the changes to the user. --- doc/src/sgml/config.sgml | 24 ++ doc/src/sgml/logicaldecoding.sgml | 8 + .../replication/logical/logicalfuncs.c | 13 + src/backend/replication/logical/slotsync.c | 13 + src/backend/replication/slot.c | 342 +++++++++++++++++- src/backend/replication/slotfuncs.c | 9 + src/backend/replication/walsender.c | 110 +++++- .../utils/activity/wait_event_names.txt | 1 + src/backend/utils/misc/guc_tables.c | 14 + src/backend/utils/misc/postgresql.conf.sample | 2 + src/include/replication/slot.h | 7 + src/include/replication/walsender.h | 1 + src/include/replication/walsender_private.h | 7 + src/include/utils/guc_hooks.h | 3 + src/test/recovery/t/006_logical_decoding.pl | 3 +- .../t/040_standby_failover_slots_sync.pl | 261 +++++++++++-- 16 files changed, 761 insertions(+), 57 deletions(-) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index ff184003fe..1c84d35c96 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4420,6 +4420,30 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows </listitem> </varlistentry> + <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names"> + <term><varname>standby_slot_names</varname> (<type>string</type>) + <indexterm> + <primary><varname>standby_slot_names</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + List of physical slots guarantees that logical replication slots with + failover enabled do not consume changes until those changes are received + and flushed to corresponding physical standbys. If a logical replication + connection is meant to switch to a physical standby after the standby is + promoted, the physical replication slot for the standby should be listed + here. + </para> + <para> + The standbys corresponding to the physical replication slots in + <varname>standby_slot_names</varname> must configure + <literal>sync_replication_slots = true</literal> so they can receive + failover logical slots changes from the primary. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index 930c0fa8a6..73b2abeda5 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -384,6 +384,14 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU must be enabled on the standby. It is also necessary to specify a valid <literal>dbname</literal> in the <link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>. + It's also highly recommended that the said physical replication slot + is named in + <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link> + list on the primary, to prevent the subscriber from consuming changes + faster than the hot standby. But once we configure it, then certain latency + is expected in sending changes to logical subscribers due to wait on + physical replication slots in + <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link> </para> <para> diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c index b0081d3ce5..5ff761dd65 100644 --- a/src/backend/replication/logical/logicalfuncs.c +++ b/src/backend/replication/logical/logicalfuncs.c @@ -30,6 +30,7 @@ #include "replication/decode.h" #include "replication/logical.h" #include "replication/message.h" +#include "replication/walsender.h" #include "storage/fd.h" #include "utils/array.h" #include "utils/builtins.h" @@ -109,6 +110,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin MemoryContext per_query_ctx; MemoryContext oldcontext; XLogRecPtr end_of_wal; + XLogRecPtr wait_for_wal_lsn; LogicalDecodingContext *ctx; ResourceOwner old_resowner = CurrentResourceOwner; ArrayType *arr; @@ -228,6 +230,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin NameStr(MyReplicationSlot->data.plugin), format_procedure(fcinfo->flinfo->fn_oid)))); + if (XLogRecPtrIsInvalid(upto_lsn)) + wait_for_wal_lsn = end_of_wal; + else + wait_for_wal_lsn = Min(upto_lsn, end_of_wal); + + /* + * Wait for specified streaming replication standby servers (if any) + * to confirm receipt of WAL up to wait_for_wal_lsn. + */ + WaitForStandbyConfirmation(wait_for_wal_lsn); + ctx->output_writer_private = p; /* diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 1f6ba0762e..d7bf61fd10 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -487,6 +487,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) latestFlushPtr = GetStandbyFlushRecPtr(NULL); if (remote_slot->confirmed_lsn > latestFlushPtr) { + /* + * Can get here only if GUC 'standby_slot_names' on the primary server + * was not configured correctly. + */ elog(am_slotsync_worker ? LOG : ERROR, "skipping slot synchronization as the received slot sync" " LSN %X/%X for slot \"%s\" is ahead of the standby position %X/%X", @@ -863,6 +867,15 @@ validate_remote_info(WalReceiverConn *wrconn, int elevel) remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull)); Assert(!isnull); + /* + * Slot sync is currently not supported on the cascading standby. This is + * because if we allow it, the primary server needs to wait for all the + * cascading standbys, otherwise, logical subscribers can still be ahead + * of one of the cascading standbys which we plan to promote. Thus, to + * avoid this additional complexity, we restrict it for the time being. + * + * XXX: If needed, this can be attempted in future. + */ if (remote_in_recovery) { /* diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index dd4ea0388a..1ab0f758ec 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -46,13 +46,18 @@ #include "common/string.h" #include "miscadmin.h" #include "pgstat.h" +#include "postmaster/interrupt.h" #include "replication/slotsync.h" #include "replication/slot.h" +#include "replication/walsender_private.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/proc.h" #include "storage/procarray.h" #include "utils/builtins.h" +#include "utils/guc_hooks.h" +#include "utils/memutils.h" +#include "utils/varlena.h" /* * Replication slot on-disk data structure. @@ -99,10 +104,19 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL; /* My backend's replication slot in the shared memory array */ ReplicationSlot *MyReplicationSlot = NULL; -/* GUC variable */ +/* GUC variables */ int max_replication_slots = 10; /* the maximum number of replication * slots */ +/* + * This GUC lists streaming replication standby server slot names that + * logical WAL sender processes will wait for. + */ +char *standby_slot_names; + +/* This is parsed and cached list for raw standby_slot_names. */ +static List *standby_slot_names_list = NIL; + static void ReplicationSlotShmemExit(int code, Datum arg); static void ReplicationSlotDropPtr(ReplicationSlot *slot); @@ -2297,3 +2311,329 @@ GetSlotInvalidationCause(char *conflict_reason) /* Keep compiler quiet */ return RS_INVAL_NONE; } + +/* + * A helper function to validate slots specified in GUC standby_slot_names. + */ +static bool +validate_standby_slots(char **newval) +{ + char *rawname; + List *elemlist; + ListCell *lc; + bool ok; + + /* Need a modifiable copy of string */ + rawname = pstrdup(*newval); + + /* Verify syntax and parse string into a list of identifiers */ + ok = SplitIdentifierString(rawname, ',', &elemlist); + + if (!ok) + GUC_check_errdetail("List syntax is invalid."); + + /* + * If there is a syntax error in the name or if the replication slots' + * data is not initialized yet (i.e., we are in the startup process), skip + * the slot verification. + */ + if (!ok || !ReplicationSlotCtl) + { + pfree(rawname); + list_free(elemlist); + return ok; + } + + foreach(lc, elemlist) + { + char *name = lfirst(lc); + ReplicationSlot *slot; + + slot = SearchNamedReplicationSlot(name, true); + + if (!slot) + { + GUC_check_errdetail("replication slot \"%s\" does not exist", + name); + ok = false; + break; + } + + if (!SlotIsPhysical(slot)) + { + GUC_check_errdetail("\"%s\" is not a physical replication slot", + name); + ok = false; + break; + } + } + + pfree(rawname); + list_free(elemlist); + return ok; +} + +/* + * GUC check_hook for standby_slot_names + */ +bool +check_standby_slot_names(char **newval, void **extra, GucSource source) +{ + if (strcmp(*newval, "") == 0) + return true; + + /* + * "*" is not accepted as in that case primary will not be able to know + * for which all standbys to wait for. Even if we have physical-slots + * info, there is no way to confirm whether there is any standby + * configured for the known physical slots. + */ + if (strcmp(*newval, "*") == 0) + { + GUC_check_errdetail("\"%s\" is not accepted for standby_slot_names", + *newval); + return false; + } + + /* Now verify if the specified slots really exist and have correct type */ + if (!validate_standby_slots(newval)) + return false; + + *extra = guc_strdup(ERROR, *newval); + + return true; +} + +/* + * GUC assign_hook for standby_slot_names + */ +void +assign_standby_slot_names(const char *newval, void *extra) +{ + List *standby_slots; + MemoryContext oldcxt; + char *standby_slot_names_cpy = extra; + + list_free(standby_slot_names_list); + standby_slot_names_list = NIL; + + /* No value is specified for standby_slot_names. */ + if (standby_slot_names_cpy == NULL) + return; + + if (!SplitIdentifierString(standby_slot_names_cpy, ',', &standby_slots)) + { + /* This should not happen if GUC checked check_standby_slot_names. */ + elog(ERROR, "invalid list syntax"); + } + + /* + * Switch to the same memory context under which GUC variables are + * allocated (GUCMemoryContext). + */ + oldcxt = MemoryContextSwitchTo(GetMemoryChunkContext(standby_slot_names_cpy)); + standby_slot_names_list = list_copy(standby_slots); + MemoryContextSwitchTo(oldcxt); +} + +/* + * Return a copy of standby_slot_names_list if the copy flag is set to true, + * otherwise return the original list. + */ +List * +GetStandbySlotList(bool copy) +{ + /* + * Since we do not support syncing slots to cascading standbys, we return + * NIL here if we are running in a standby to indicate that no standby + * slots need to be waited for. + */ + if (RecoveryInProgress()) + return NIL; + + if (copy) + return list_copy(standby_slot_names_list); + else + return standby_slot_names_list; +} + +/* + * Reload the config file and reinitialize the standby slot list if the GUC + * standby_slot_names has changed. + */ +void +RereadConfigAndReInitSlotList(List **standby_slots) +{ + char *pre_standby_slot_names; + + /* + * If we are running on a standby, there is no need to reload + * standby_slot_names since we do not support syncing slots to cascading + * standbys. + */ + if (RecoveryInProgress()) + { + ProcessConfigFile(PGC_SIGHUP); + return; + } + + pre_standby_slot_names = pstrdup(standby_slot_names); + + ProcessConfigFile(PGC_SIGHUP); + + if (strcmp(pre_standby_slot_names, standby_slot_names) != 0) + { + list_free(*standby_slots); + *standby_slots = GetStandbySlotList(true); + } + + pfree(pre_standby_slot_names); +} + +/* + * Filter the standby slots based on the specified log sequence number + * (wait_for_lsn). + * + * This function updates the passed standby_slots list, removing any slots that + * have already caught up to or surpassed the given wait_for_lsn. Additionally, + * it removes slots that have been invalidated, dropped, or converted to + * logical slots. + */ +void +FilterStandbySlots(XLogRecPtr wait_for_lsn, List **standby_slots) +{ + ListCell *lc; + List *standby_slots_cpy = *standby_slots; + + foreach(lc, standby_slots_cpy) + { + char *name = lfirst(lc); + char *warningfmt = NULL; + ReplicationSlot *slot; + + slot = SearchNamedReplicationSlot(name, true); + + if (!slot) + { + /* + * It may happen that the slot specified in standby_slot_names GUC + * value is dropped, so let's skip over it. + */ + warningfmt = _("replication slot \"%s\" specified in parameter \"%s\" does not exist, ignoring"); + } + else if (SlotIsLogical(slot)) + { + /* + * If a logical slot name is provided in standby_slot_names, issue + * a WARNING and skip it. Although logical slots are disallowed in + * the GUC check_hook(validate_standby_slots), it is still + * possible for a user to drop an existing physical slot and + * recreate a logical slot with the same name. Since it is + * harmless, a WARNING should be enough, no need to error-out. + */ + warningfmt = _("cannot have logical replication slot \"%s\" in parameter \"%s\", ignoring"); + } + else + { + SpinLockAcquire(&slot->mutex); + + if (slot->data.invalidated != RS_INVAL_NONE) + { + /* + * Specified physical slot have been invalidated, so no point + * in waiting for it. + */ + warningfmt = _("physical slot \"%s\" specified in parameter \"%s\" has been invalidated, ignoring"); + } + else if (XLogRecPtrIsInvalid(slot->data.restart_lsn) || + slot->data.restart_lsn < wait_for_lsn) + { + bool inactive = (slot->active_pid == 0); + + SpinLockRelease(&slot->mutex); + + /* Log warning if no active_pid for this physical slot */ + if (inactive) + ereport(WARNING, + errmsg("replication slot \"%s\" specified in parameter \"%s\" does not have active_pid", + name, "standby_slot_names"), + errdetail("Logical replication is waiting on the " + "standby associated with \"%s\".", name), + errhint("Consider starting standby associated with " + "\"%s\" or amend standby_slot_names.", name)); + + /* Continue if the current slot hasn't caught up. */ + continue; + } + else + { + Assert(slot->data.restart_lsn >= wait_for_lsn); + } + + SpinLockRelease(&slot->mutex); + } + + /* + * Reaching here indicates that either the slot has passed the + * wait_for_lsn or there is an issue with the slot that requires a + * warning to be reported. + */ + if (warningfmt) + ereport(WARNING, errmsg(warningfmt, name, "standby_slot_names")); + + standby_slots_cpy = foreach_delete_current(standby_slots_cpy, lc); + } + + *standby_slots = standby_slots_cpy; +} + +/* + * Wait for physical standby to confirm receiving the given lsn. + * + * Used by logical decoding SQL functions that acquired slot with failover + * enabled. It waits for physical standbys corresponding to the physical slots + * specified in the standby_slot_names GUC. + */ +void +WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn) +{ + List *standby_slots; + + if (!MyReplicationSlot->data.failover) + return; + + standby_slots = GetStandbySlotList(true); + + if (standby_slots == NIL) + return; + + ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv); + + for (;;) + { + CHECK_FOR_INTERRUPTS(); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + RereadConfigAndReInitSlotList(&standby_slots); + } + + FilterStandbySlots(wait_for_lsn, &standby_slots); + + /* Exit if done waiting for every slot. */ + if (standby_slots == NIL) + break; + + /* + * We wait for the slots in the standby_slot_names to catch up, but we + * use a timeout so we can also check the if the standby_slot_names + * has been changed. + */ + ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000, + WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION); + } + + ConditionVariableCancelSleep(); + list_free(standby_slots); +} diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index c4a0bf99ee..4f0a1fddc2 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -22,6 +22,7 @@ #include "replication/logical.h" #include "replication/slot.h" #include "replication/slotsync.h" +#include "replication/walsender.h" #include "utils/builtins.h" #include "utils/guc.h" #include "utils/inval.h" @@ -476,6 +477,8 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto) * crash, but this makes the data consistent after a clean shutdown. */ ReplicationSlotMarkDirty(); + + PhysicalWakeupLogicalWalSnd(); } return retlsn; @@ -516,6 +519,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto) .segment_close = wal_segment_close), NULL, NULL, NULL); + /* + * Wait for specified streaming replication standby servers (if any) + * to confirm receipt of WAL up to moveto lsn. + */ + WaitForStandbyConfirmation(moveto); + /* * Start reading at the slot's restart_lsn, which we know to point to * a valid record. diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index b33044e10f..54dbad7158 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1728,27 +1728,78 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId ProcessPendingWrites(); } +/* + * Wake up the logical walsender processes with failover-enabled slots if the + * currently acquired physical slot is specified in standby_slot_names + * GUC. + */ +void +PhysicalWakeupLogicalWalSnd(void) +{ + ListCell *lc; + List *standby_slots; + + Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot)); + + standby_slots = GetStandbySlotList(false); + + foreach(lc, standby_slots) + { + char *name = lfirst(lc); + + if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0) + { + ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv); + return; + } + } +} + /* * Wait till WAL < loc is flushed to disk so it can be safely sent to client. * - * Returns end LSN of flushed WAL. Normally this will be >= loc, but - * if we detect a shutdown request (either from postmaster or client) - * we will return early, so caller must always check. + * If the walsender holds a logical slot that has enabled failover, we also + * wait for all the specified streaming replication standby servers to + * confirm receipt of WAL up to RecentFlushPtr. + * + * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we + * detect a shutdown request (either from postmaster or client) we will return + * early, so caller must always check. */ static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc) { int wakeEvents; + bool wait_for_standby = false; + uint32 wait_event; + List *standby_slots = NIL; static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr; + if (MyReplicationSlot->data.failover && replication_active) + standby_slots = GetStandbySlotList(true); + /* - * Fast path to avoid acquiring the spinlock in case we already know we - * have enough WAL available. This is particularly interesting if we're - * far behind. + * Check if all the standby servers have confirmed receipt of WAL up to + * RecentFlushPtr even when we already know we have enough WAL available. + * + * Note that we cannot directly return without checking the status of + * standby servers because the standby_slot_names may have changed, which + * means there could be new standby slots in the list that have not yet + * caught up to the RecentFlushPtr. */ - if (RecentFlushPtr != InvalidXLogRecPtr && - loc <= RecentFlushPtr) - return RecentFlushPtr; + if (!XLogRecPtrIsInvalid(RecentFlushPtr) && loc <= RecentFlushPtr) + { + FilterStandbySlots(RecentFlushPtr, &standby_slots); + + /* + * Fast path to avoid acquiring the spinlock in case we already know + * we have enough WAL available and all the standby servers have + * confirmed receipt of WAL up to RecentFlushPtr. This is particularly + * interesting if we're far behind. + */ + if (standby_slots == NIL) + return RecentFlushPtr; + } /* Get a more recent flush pointer. */ if (!RecoveryInProgress()) @@ -1769,7 +1820,7 @@ WalSndWaitForWal(XLogRecPtr loc) if (ConfigReloadPending) { ConfigReloadPending = false; - ProcessConfigFile(PGC_SIGHUP); + RereadConfigAndReInitSlotList(&standby_slots); SyncRepInitConfig(); } @@ -1784,8 +1835,18 @@ WalSndWaitForWal(XLogRecPtr loc) if (got_STOPPING) XLogBackgroundFlush(); + /* + * Update the standby slots that have not yet caught up to the flushed + * position. It is good to wait up to RecentFlushPtr and then let it + * send the changes to logical subscribers one by one which are + * already covered in RecentFlushPtr without needing to wait on every + * change for standby confirmation. + */ + if (wait_for_standby) + FilterStandbySlots(RecentFlushPtr, &standby_slots); + /* Update our idea of the currently flushed position. */ - if (!RecoveryInProgress()) + else if (!RecoveryInProgress()) RecentFlushPtr = GetFlushRecPtr(NULL); else RecentFlushPtr = GetXLogReplayRecPtr(NULL); @@ -1813,9 +1874,18 @@ WalSndWaitForWal(XLogRecPtr loc) !waiting_for_ping_response) WalSndKeepalive(false, InvalidXLogRecPtr); - /* check whether we're done */ - if (loc <= RecentFlushPtr) + if (loc > RecentFlushPtr) + wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL; + else if (standby_slots) + { + wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION; + wait_for_standby = true; + } + else + { + /* Already caught up and doesn't need to wait for standby_slots. */ break; + } /* Waiting for new WAL. Since we need to wait, we're now caught up. */ WalSndCaughtUp = true; @@ -1855,9 +1925,11 @@ WalSndWaitForWal(XLogRecPtr loc) if (pq_is_send_pending()) wakeEvents |= WL_SOCKET_WRITEABLE; - WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL); + WalSndWait(wakeEvents, sleeptime, wait_event); } + list_free(standby_slots); + /* reactivate latch so WalSndLoop knows to continue */ SetLatch(MyLatch); return RecentFlushPtr; @@ -2265,6 +2337,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn) { ReplicationSlotMarkDirty(); ReplicationSlotsComputeRequiredLSN(); + PhysicalWakeupLogicalWalSnd(); } /* @@ -3538,6 +3611,7 @@ WalSndShmemInit(void) ConditionVariableInit(&WalSndCtl->wal_flush_cv); ConditionVariableInit(&WalSndCtl->wal_replay_cv); + ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv); } } @@ -3607,8 +3681,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event) * * And, we use separate shared memory CVs for physical and logical * walsenders for selective wake ups, see WalSndWakeup() for more details. + * + * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV + * until awakened by physical walsenders after the walreceiver confirms + * the receipt of the LSN. */ - if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL) + if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION) + ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv); + else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL) ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv); else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL) ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index b52afd4eac..2f99649581 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server." LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server." SSL_OPEN_SERVER "Waiting for SSL while attempting connection." +WAIT_FOR_STANDBY_CONFIRMATION "Waiting for the WAL to be received by physical standby." WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process." WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process." diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 37be0669bb..d3bbdbe94e 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -4639,6 +4639,20 @@ struct config_string ConfigureNamesString[] = check_debug_io_direct, assign_debug_io_direct, NULL }, + { + {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY, + gettext_noop("Lists streaming replication standby server slot " + "names that logical WAL sender processes will wait for."), + gettext_noop("Decoded changes are sent out to plugins by logical " + "WAL sender processes only after specified " + "replication slots confirm receiving WAL."), + GUC_LIST_INPUT | GUC_LIST_QUOTE + }, + &standby_slot_names, + "", + check_standby_slot_names, assign_standby_slot_names, NULL + }, + /* End-of-list marker */ { {NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index c97f9a25f0..cadfe10958 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -334,6 +334,8 @@ # method to choose sync standbys, number of sync standbys, # and comma-separated list of application_name # from standby(s); '*' = all +#standby_slot_names = '' # streaming replication standby server slot names that + # logical walsender processes will wait for # - Standby Servers - diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index e706ca834c..7b754a1f48 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -229,6 +229,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot; /* GUCs */ extern PGDLLIMPORT int max_replication_slots; +extern PGDLLIMPORT char *standby_slot_names; /* shmem initialization functions */ extern Size ReplicationSlotsShmemSize(void); @@ -277,4 +278,10 @@ extern void CheckSlotPermissions(void); extern ReplicationSlotInvalidationCause GetSlotInvalidationCause(char *conflict_reason); +extern List *GetStandbySlotList(bool copy); +extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn); +extern void FilterStandbySlots(XLogRecPtr wait_for_lsn, + List **standby_slots); +extern void RereadConfigAndReInitSlotList(List **standby_slots); + #endif /* SLOT_H */ diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h index 0c3996e926..f2d8297f01 100644 --- a/src/include/replication/walsender.h +++ b/src/include/replication/walsender.h @@ -39,6 +39,7 @@ extern void InitWalSender(void); extern bool exec_replication_command(const char *cmd_string); extern void WalSndErrorCleanup(void); extern void WalSndResourceCleanup(bool isCommit); +extern void PhysicalWakeupLogicalWalSnd(void); extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); extern void WalSndSignals(void); extern Size WalSndShmemSize(void); diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h index 3113e9ea47..0f962b0c72 100644 --- a/src/include/replication/walsender_private.h +++ b/src/include/replication/walsender_private.h @@ -113,6 +113,13 @@ typedef struct ConditionVariable wal_flush_cv; ConditionVariable wal_replay_cv; + /* + * Used by physical walsenders holding slots specified in + * standby_slot_names to wake up logical walsenders holding + * failover-enabled slots when a walreceiver confirms the receipt of LSN. + */ + ConditionVariable wal_confirm_rcv_cv; + WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER]; } WalSndCtlData; diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h index 339c490300..dcf9a040d1 100644 --- a/src/include/utils/guc_hooks.h +++ b/src/include/utils/guc_hooks.h @@ -163,5 +163,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra, extern void assign_wal_consistency_checking(const char *newval, void *extra); extern bool check_wal_segment_size(int *newval, void **extra, GucSource source); extern void assign_wal_sync_method(int new_wal_sync_method, void *extra); +extern bool check_standby_slot_names(char **newval, void **extra, + GucSource source); +extern void assign_standby_slot_names(const char *newval, void *extra); #endif /* GUC_HOOKS_H */ diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl index 5c7b4ca5e3..85f019774c 100644 --- a/src/test/recovery/t/006_logical_decoding.pl +++ b/src/test/recovery/t/006_logical_decoding.pl @@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'}, undef, 'logical slot was actually dropped with DB'); # Test logical slot advancing and its durability. +# Pass failover=true (last-arg), it should not have any impact on advancing. my $logical_slot = 'logical_slot'; $node_primary->safe_psql('postgres', - "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);" + "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);" ); $node_primary->psql( 'postgres', " diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index 950f6b3738..632d99dae4 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -113,19 +113,30 @@ ok( $stderr =~ "cannot sync slots on a non-standby server"); ################################################## -# Test logical failover slots on the standby -# Configure standby1 to replicate and synchronize logical slots configured -# for failover on the primary +# Test primary disallowing specified logical replication slots getting ahead of +# specified physical replication slots. It uses the following set up: # -# failover slot lsub1_slot ->| ----> subscriber1 (connected via logical replication) -# failover slot lsub2_slot | inactive -# primary ---> | -# physical slot sb1_slot --->| ----> standby1 (connected via streaming replication) -# | lsub1_slot, lsub2_slot (synced_slot) +# | ----> standby1 (primary_slot_name = sb1_slot) +# | ----> standby2 (primary_slot_name = sb2_slot) +# primary ----- | +# | ----> subscriber1 (failover = true) +# | ----> subscriber2 (failover = false) +# +# standby_slot_names = 'sb1_slot' +# +# Set up is configured in such a way that the logical slot of subscriber1 is +# enabled failover, thus it will wait for the physical slot of +# standby1(sb1_slot) to catch up before sending decoded changes to subscriber1. ################################################## my $primary = $publisher; my $backup_name = 'backup'; + +$primary->psql('postgres', + q{SELECT pg_create_physical_replication_slot('sb1_slot');}); +$primary->psql('postgres', + q{SELECT pg_create_physical_replication_slot('sb2_slot');}); + $primary->backup($backup_name); # Create a standby @@ -135,13 +146,206 @@ $standby1->init_from_backup( has_streaming => 1, has_restoring => 1); +$standby1->append_conf( + 'postgresql.conf', qq( +primary_slot_name = 'sb1_slot' +)); +$standby1->start; +$primary->wait_for_replay_catchup($standby1); + +# Speed up the slot creation +$primary->safe_psql('postgres', "SELECT pg_log_standby_snapshot()"); + +# Create a slot to save the old xmin info, which speeds up the persistence of +# the newly created slot by obtaining an existing old xmin value. +$standby1->psql('postgres', + "SELECT pg_create_logical_replication_slot('save_xmin', 'test_decoding');"); + +# Create another standby +my $standby2 = PostgreSQL::Test::Cluster->new('standby2'); +$standby2->init_from_backup( + $primary, $backup_name, + has_streaming => 1, + has_restoring => 1); +$standby2->append_conf( + 'postgresql.conf', qq( +primary_slot_name = 'sb2_slot' +)); +$standby2->start; +$primary->wait_for_replay_catchup($standby2); + +# Configure primary to disallow any logical slots that enabled failover from +# getting ahead of specified physical replication slot (sb1_slot). +$primary->append_conf( + 'postgresql.conf', qq( +standby_slot_names = 'sb1_slot' +)); +$primary->reload; + +$primary->safe_psql('postgres', "CREATE TABLE tab_int (a int PRIMARY KEY);"); + +# Create a table and refresh the publication +$subscriber1->safe_psql( + 'postgres', qq[ + CREATE TABLE tab_int (a int PRIMARY KEY); + ALTER SUBSCRIPTION regress_mysub1 REFRESH PUBLICATION WITH (copy_data = false); +]); + +# Create another subscriber node without enabling failover, wait for sync to +# complete +my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2'); +$subscriber2->init; +$subscriber2->start; +$subscriber2->safe_psql( + 'postgres', qq[ + CREATE TABLE tab_int (a int PRIMARY KEY); + CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot, copy_data = false); +]); + +# Stop the standby associated with the specified physical replication slot so +# that the logical replication slot won't receive changes until the standby +# comes up. +$standby1->stop; + +# Create some data on the primary +my $primary_row_count = 10; +$primary->safe_psql('postgres', + "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);"); + +# Wait for the standby that's up and running gets the data from primary +$primary->wait_for_replay_catchup($standby2); +$result = $standby2->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', "standby2 gets data from primary"); + +# Wait for the subscription that's up and running and is not enabled for failover. +# It gets the data from primary without waiting for any standbys. +$publisher->wait_for_catchup('regress_mysub2'); +$result = $subscriber2->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', "subscriber2 gets data from primary"); + +# The subscription that's up and running and is enabled for failover +# doesn't get the data from primary and keeps waiting for the +# standby specified in standby_slot_names. +$result = + $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;"); +is($result, 't', + "subscriber1 doesn't get data from primary until standby1 acknowledges changes" +); + +# Start the standby specified in standby_slot_names and wait for it to catch +# up with the primary. +$standby1->start; +$primary->wait_for_replay_catchup($standby1); +$result = $standby1->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', "standby1 gets data from primary"); + +# Now that the standby specified in standby_slot_names is up and running, +# primary must send the decoded changes to subscription enabled for failover +# While the standby was down, this subscriber didn't receive any data from +# primary i.e. the primary didn't allow it to go ahead of standby. +$publisher->wait_for_catchup('regress_mysub1'); +$result = $subscriber1->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', + "subscriber1 gets data from primary after standby1 acknowledges changes"); + +# Stop the standby associated with the specified physical replication slot so +# that the logical replication slot won't receive changes until the standby +# slot's restart_lsn is advanced or the slot is removed from the +# standby_slot_names list. +$publisher->safe_psql('postgres', "TRUNCATE tab_int;"); +$publisher->wait_for_catchup('regress_mysub1'); +$standby1->stop; + +################################################## +# Verify that when using pg_logical_slot_get_changes to consume changes from a +# logical slot with failover enabled, it will also wait for the slots specified +# in standby_slot_names to catch up. +################################################## + +# Create a logical 'test_decoding' replication slot with failover enabled +$publisher->safe_psql('postgres', + "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);" +); + +my $back_q = $primary->background_psql('postgres', on_error_stop => 0); +my $pid = $back_q->query('SELECT pg_backend_pid()'); + +# Try and get changes from the logical slot with failover enabled. +my $offset = -s $primary->logfile; +$back_q->query_until(qr//, + "SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);\n"); + +# Wait until the primary server logs a warning indicating that it is waiting +# for the sb1_slot to catch up. +$primary->wait_for_log( + qr/WARNING: ( [A-Z0-9]+:)? replication slot \"sb1_slot\" specified in parameter \"standby_slot_names\" does not have active_pid/, + $offset); + +ok($primary->safe_psql('postgres', "SELECT pg_cancel_backend($pid)"), + "cancelling pg_logical_slot_get_changes command"); + +$back_q->quit; + +$publisher->safe_psql('postgres', + "SELECT pg_drop_replication_slot('test_slot');" +); + +################################################## +# Test that logical replication will wait for the user-created inactive +# physical slot to catch up until we remove the slot from standby_slot_names. +################################################## + +# Create some data on the primary +$primary_row_count = 10; +$primary->safe_psql('postgres', + "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);"); + +$result = + $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;"); +is($result, 't', + "subscriber1 doesn't get data as the sb1_slot doesn't catch up"); + +# Remove the standby from the standby_slot_names list and reload the +# configuration. +$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''"); +$primary->reload; + +# Since there are no slots in standby_slot_names, the primary server should now +# send the decoded changes to the subscription. +$publisher->wait_for_catchup('regress_mysub1'); +$result = $subscriber1->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', + "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list" +); + +# Put the standby back on the primary_slot_name for the rest of the tests +$primary->adjust_conf('postgresql.conf', 'standby_slot_names', 'sb1_slot'); +$primary->reload; + +################################################## +# Test logical failover slots on the standby +# Configure standby1 to replicate and synchronize logical slots configured +# for failover on the primary +# +# failover slot lsub1_slot ->| ----> subscriber1 (connected via logical replication) +# failover slot lsub3_slot | inactive +# primary ---> | +# physical slot sb1_slot --->| ----> standby1 (connected via streaming replication) +# | lsub1_slot, lsub3_slot (synced_slot) +################################################## + +# Configure the standby # Increase the log_min_messages setting to DEBUG2 on both the standby and # primary to debug test failures, if any. my $connstr_1 = $primary->connstr; $standby1->append_conf( 'postgresql.conf', qq( hot_standby_feedback = on -primary_slot_name = 'sb1_slot' primary_conninfo = '$connstr_1 dbname=postgres' log_min_messages = 'debug2' )); @@ -150,29 +354,12 @@ $primary->append_conf('postgresql.conf', "log_min_messages = 'debug2'"); $primary->reload; $primary->psql('postgres', - q{SELECT pg_create_logical_replication_slot('lsub2_slot', 'test_decoding', false, false, true);} + q{SELECT pg_create_logical_replication_slot('lsub3_slot', 'test_decoding', false, false, true);} ); -$primary->psql('postgres', - q{SELECT pg_create_physical_replication_slot('sb1_slot');}); - # Start the standby so that slot syncing can begin $standby1->start; -$primary->wait_for_catchup('regress_mysub1'); - -# Do not allow any further advancement of the restart_lsn for the lsub1_slot. -$subscriber1->safe_psql('postgres', - "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); - -# Wait for the replication slot to become inactive on the publisher -$primary->poll_query_until( - 'postgres', - "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", - 1); - -# Wait for the standby to catch up so that the standby is not lagging behind -# the subscriber. $primary->wait_for_replay_catchup($standby1); # Synchronize the primary server slots to the standby. @@ -182,7 +369,7 @@ $standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); # flagged as 'synced' is( $standby1->safe_psql( 'postgres', - q{SELECT count(*) = 2 FROM pg_replication_slots WHERE slot_name IN ('lsub1_slot', 'lsub2_slot') AND synced AND NOT temporary;} + q{SELECT count(*) = 2 FROM pg_replication_slots WHERE slot_name IN ('lsub1_slot', 'lsub3_slot') AND synced AND NOT temporary;} ), "t", 'logical slots have synced as true on standby'); @@ -192,13 +379,13 @@ is( $standby1->safe_psql( # slot on the primary server has been dropped. ################################################## -$primary->psql('postgres', "SELECT pg_drop_replication_slot('lsub2_slot');"); +$primary->psql('postgres', "SELECT pg_drop_replication_slot('lsub3_slot');"); $standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); is( $standby1->safe_psql( 'postgres', - q{SELECT count(*) = 0 FROM pg_replication_slots WHERE slot_name = 'lsub2_slot';} + q{SELECT count(*) = 0 FROM pg_replication_slots WHERE slot_name = 'lsub3_slot';} ), "t", 'synchronized slot has been dropped'); @@ -398,19 +585,13 @@ $standby1->wait_for_log(qr/LOG: parameters validation done for slot synchroniza # Insert data on the primary $primary->safe_psql( 'postgres', qq[ - CREATE TABLE tab_int (a int PRIMARY KEY); + TRUNCATE TABLE tab_int; INSERT INTO tab_int SELECT generate_series(1, 10); ]); -# Subscribe to the new table data and wait for it to arrive -$subscriber1->safe_psql( - 'postgres', qq[ - CREATE TABLE tab_int (a int PRIMARY KEY); - ALTER SUBSCRIPTION regress_mysub1 ENABLE; - ALTER SUBSCRIPTION regress_mysub1 REFRESH PUBLICATION; -]); - -$subscriber1->wait_for_subscription_sync; +# Enable the subscription to let it catch up to the latest wal position +$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE"); +$primary->wait_for_catchup('regress_mysub1'); # Do not allow any further advancement of the restart_lsn and # confirmed_flush_lsn for the lsub1_slot. -- 2.34.1 [application/octet-stream] v89-0001-Add-a-new-slotsync-worker.patch (62.7K, ../../CAJpy0uBA4efiWHhS9_FOmXxvdERk0bbMk0qMjo8ZpSTWE4DjVA@mail.gmail.com/4-v89-0001-Add-a-new-slotsync-worker.patch) download | inline diff: From e430f5563f3cf9ff56ffd4c773a5cf785d39bc18 Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Fri, 16 Feb 2024 12:44:58 +0530 Subject: [PATCH v89 1/3] Add a new slotsync worker By enabling slot synchronization, all the failover logical replication slots on the primary (assuming configurations are appropriate) are automatically created on the physical standbys and are synced periodically. Slot-sync worker on the standby server ping the primary server at regular intervals to get the necessary failover logical slots information and create/update the slots locally. The slots that no longer require synchronization are automatically dropped by the worker. The nap time of the worker is tuned according to the activity on the primary. The worker waits for a period of time before the next synchronization, with the duration varying based on whether any slots were updated during the last cycle. A new parameter sync_replication_slots enables or disables this new process. On promotion, the slot sync worker is shut down by the startup process to drop any temporary slots acquired by the slot sync worker and to prevent the worker from keep trying to fetch the failover slots. --- doc/src/sgml/config.sgml | 18 + doc/src/sgml/logicaldecoding.sgml | 5 +- src/backend/access/transam/xlogrecovery.c | 15 + src/backend/postmaster/postmaster.c | 88 +- .../libpqwalreceiver/libpqwalreceiver.c | 3 + src/backend/replication/logical/slotsync.c | 809 ++++++++++++++++-- src/backend/replication/slot.c | 14 + src/backend/replication/slotfuncs.c | 2 +- src/backend/replication/walsender.c | 2 +- src/backend/storage/ipc/ipci.c | 4 +- src/backend/storage/lmgr/proc.c | 13 +- src/backend/utils/activity/pgstat_io.c | 1 + .../utils/activity/wait_event_names.txt | 2 + src/backend/utils/init/miscinit.c | 9 +- src/backend/utils/init/postinit.c | 8 +- src/backend/utils/misc/guc_tables.c | 10 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/miscadmin.h | 1 + src/include/replication/slotsync.h | 24 +- .../t/040_standby_failover_slots_sync.pl | 113 ++- src/tools/pgindent/typedefs.list | 2 +- 21 files changed, 1044 insertions(+), 100 deletions(-) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index ffd711b7f2..ff184003fe 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4943,6 +4943,24 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-sync-replication-slots" xreflabel="sync_replication_slots"> + <term><varname>sync_replication_slots</varname> (<type>boolean</type>) + <indexterm> + <primary><varname>sync_replication_slots</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + It enables a physical standby to synchronize logical failover slots + from the primary server so that logical subscribers can resume + replication from the new primary server after failover. + </para> + <para> + It is disabled by default. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command line. + </para> + </listitem> + </varlistentry> </variablelist> </sect2> diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index eceaaaa273..930c0fa8a6 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -373,7 +373,10 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU <command>CREATE SUBSCRIPTION</command> during slot creation, and then calling <link linkend="pg-sync-replication-slots"> <function>pg_sync_replication_slots</function></link> - on the standby. For the synchronization to work, it is mandatory to + on the standby. By setting <link linkend="guc-sync-replication-slots"> + <varname>sync_replication_slots</varname></link> + on the standby, the failover slots can be synchronized periodically in + the slotsync worker. For the synchronization to work, it is mandatory to have a physical replication slot between the primary and the standby aka <link linkend="guc-primary-slot-name"><varname>primary_slot_name</varname></link> should be configured on the standby, and diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 0bb472da27..d73a49b3e8 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -49,6 +49,7 @@ #include "postmaster/bgwriter.h" #include "postmaster/startup.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/walreceiver.h" #include "storage/fd.h" #include "storage/ipc.h" @@ -1467,6 +1468,20 @@ FinishWalRecovery(void) */ XLogShutdownWalRcv(); + /* + * Shutdown the slot sync worker to drop any temporary slots acquired by + * it and to prevent it from keep trying to fetch the failover slots. + * + * We do not update the 'synced' column from true to false here, as any + * failed update could leave 'synced' column false for some slots. This + * could cause issues during slot sync after restarting the server as a + * standby. While updating the 'synced' column after switching to the new + * timeline is an option, it does not simplify the handling for the + * 'synced' column. Therefore, we retain the 'synced' column as true after + * promotion as it may provide useful information about the slot origin. + */ + ShutDownSlotSync(); + /* * We are now done reading the xlog from stream. Turn off streaming * recovery to force fetching the files (which would be required at end of diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index df945a5ac4..3080d4aa53 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -115,6 +115,7 @@ #include "postmaster/syslogger.h" #include "postmaster/walsummarizer.h" #include "replication/logicallauncher.h" +#include "replication/slotsync.h" #include "replication/walsender.h" #include "storage/fd.h" #include "storage/ipc.h" @@ -167,11 +168,11 @@ * they will never become live backends. dead_end children are not assigned a * PMChildSlot. dead_end children have bkend_type NORMAL. * - * "Special" children such as the startup, bgwriter and autovacuum launcher - * tasks are not in this list. They are tracked via StartupPID and other - * pid_t variables below. (Thus, there can't be more than one of any given - * "special" child process type. We use BackendList entries for any child - * process there can be more than one of.) + * "Special" children such as the startup, bgwriter, autovacuum launcher, and + * slot sync worker tasks are not in this list. They are tracked via StartupPID + * and other pid_t variables below. (Thus, there can't be more than one of any + * given "special" child process type. We use BackendList entries for any + * child process there can be more than one of.) */ typedef struct bkend { @@ -254,7 +255,8 @@ static pid_t StartupPID = 0, WalSummarizerPID = 0, AutoVacPID = 0, PgArchPID = 0, - SysLoggerPID = 0; + SysLoggerPID = 0, + SlotSyncWorkerPID = 0; /* Startup process's status */ typedef enum @@ -458,6 +460,17 @@ static void InitPostmasterDeathWatchHandle(void); (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \ PgArchCanRestart()) +/* + * Slot sync worker allowed to start up? + * + * If we are on a hot standby, slot sync parameter is enabled, and it is + * the first time of worker's launch, or enough time has passed since the + * worker was launched last, then it is allowed to be started. + */ +#define SlotSyncWorkerAllowed() \ + (sync_replication_slots && pmState == PM_HOT_STANDBY && \ + SlotSyncWorkerCanRestart()) + #ifdef EXEC_BACKEND #ifdef WIN32 @@ -1822,6 +1835,10 @@ ServerLoop(void) if (PgArchPID == 0 && PgArchStartupAllowed()) PgArchPID = StartChildProcess(ArchiverProcess); + /* If we need to start a slot sync worker, try to do that now */ + if (SlotSyncWorkerPID == 0 && SlotSyncWorkerAllowed()) + SlotSyncWorkerPID = StartSlotSyncWorker(); + /* If we need to signal the autovacuum launcher, do so now */ if (avlauncher_needs_signal) { @@ -2661,6 +2678,8 @@ process_pm_reload_request(void) signal_child(PgArchPID, SIGHUP); if (SysLoggerPID != 0) signal_child(SysLoggerPID, SIGHUP); + if (SlotSyncWorkerPID != 0) + signal_child(SlotSyncWorkerPID, SIGHUP); /* Reload authentication config files too */ if (!load_hba()) @@ -3011,6 +3030,9 @@ process_pm_child_exit(void) if (PgArchStartupAllowed() && PgArchPID == 0) PgArchPID = StartChildProcess(ArchiverProcess); + if (SlotSyncWorkerAllowed() && SlotSyncWorkerPID == 0) + SlotSyncWorkerPID = StartSlotSyncWorker(); + /* workers may be scheduled to start now */ maybe_start_bgworkers(); @@ -3180,6 +3202,22 @@ process_pm_child_exit(void) continue; } + /* + * Was it the slot sync worker? Normal exit or FATAL exit can be + * ignored (FATAL can be caused by libpqwalreceiver on receiving + * shutdown request by the startup process during promotion); we'll + * start a new one at the next iteration of the postmaster's main + * loop, if necessary. Any other exit condition is treated as a crash. + */ + if (pid == SlotSyncWorkerPID) + { + SlotSyncWorkerPID = 0; + if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus)) + HandleChildCrash(pid, exitstatus, + _("slot sync worker process")); + continue; + } + /* Was it one of our background workers? */ if (CleanupBackgroundWorker(pid, exitstatus)) { @@ -3384,7 +3422,7 @@ CleanupBackend(int pid, /* * HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer, - * walwriter, autovacuum, archiver or background worker. + * walwriter, autovacuum, archiver, slot sync worker, or background worker. * * The objectives here are to clean up our local state about the child * process, and to signal all other remaining children to quickdie. @@ -3546,6 +3584,12 @@ HandleChildCrash(int pid, int exitstatus, const char *procname) else if (PgArchPID != 0 && take_action) sigquit_child(PgArchPID); + /* Take care of the slot sync worker too */ + if (pid == SlotSyncWorkerPID) + SlotSyncWorkerPID = 0; + else if (SlotSyncWorkerPID != 0 && take_action) + sigquit_child(SlotSyncWorkerPID); + /* We do NOT restart the syslogger */ if (Shutdown != ImmediateShutdown) @@ -3686,6 +3730,8 @@ PostmasterStateMachine(void) signal_child(WalReceiverPID, SIGTERM); if (WalSummarizerPID != 0) signal_child(WalSummarizerPID, SIGTERM); + if (SlotSyncWorkerPID != 0) + signal_child(SlotSyncWorkerPID, SIGTERM); /* checkpointer, archiver, stats, and syslogger may continue for now */ /* Now transition to PM_WAIT_BACKENDS state to wait for them to die */ @@ -3701,13 +3747,13 @@ PostmasterStateMachine(void) /* * PM_WAIT_BACKENDS state ends when we have no regular backends * (including autovac workers), no bgworkers (including unconnected - * ones), and no walwriter, autovac launcher or bgwriter. If we are - * doing crash recovery or an immediate shutdown then we expect the - * checkpointer to exit as well, otherwise not. The stats and - * syslogger processes are disregarded since they are not connected to - * shared memory; we also disregard dead_end children here. Walsenders - * and archiver are also disregarded, they will be terminated later - * after writing the checkpoint record. + * ones), and no walwriter, autovac launcher, bgwriter or slot sync + * worker. If we are doing crash recovery or an immediate shutdown + * then we expect the checkpointer to exit as well, otherwise not. The + * stats and syslogger processes are disregarded since they are not + * connected to shared memory; we also disregard dead_end children + * here. Walsenders and archiver are also disregarded, they will be + * terminated later after writing the checkpoint record. */ if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 && StartupPID == 0 && @@ -3717,7 +3763,8 @@ PostmasterStateMachine(void) (CheckpointerPID == 0 || (!FatalError && Shutdown < ImmediateShutdown)) && WalWriterPID == 0 && - AutoVacPID == 0) + AutoVacPID == 0 && + SlotSyncWorkerPID == 0) { if (Shutdown >= ImmediateShutdown || FatalError) { @@ -3815,6 +3862,7 @@ PostmasterStateMachine(void) Assert(CheckpointerPID == 0); Assert(WalWriterPID == 0); Assert(AutoVacPID == 0); + Assert(SlotSyncWorkerPID == 0); /* syslogger is not considered here */ pmState = PM_NO_CHILDREN; } @@ -4038,6 +4086,8 @@ TerminateChildren(int signal) signal_child(AutoVacPID, signal); if (PgArchPID != 0) signal_child(PgArchPID, signal); + if (SlotSyncWorkerPID != 0) + signal_child(SlotSyncWorkerPID, signal); } /* @@ -4850,6 +4900,7 @@ SubPostmasterMain(int argc, char *argv[]) */ if (strcmp(argv[1], "--forkbackend") == 0 || strcmp(argv[1], "--forkavlauncher") == 0 || + strcmp(argv[1], "--forkssworker") == 0 || strcmp(argv[1], "--forkavworker") == 0 || strcmp(argv[1], "--forkaux") == 0 || strcmp(argv[1], "--forkbgworker") == 0) @@ -4953,6 +5004,13 @@ SubPostmasterMain(int argc, char *argv[]) AutoVacWorkerMain(argc - 2, argv + 2); /* does not return */ } + if (strcmp(argv[1], "--forkssworker") == 0) + { + /* Restore basic shared memory pointers */ + InitShmemAccess(UsedShmemSegAddr); + + ReplSlotSyncWorkerMain(argc - 2, argv + 2); /* does not return */ + } if (strcmp(argv[1], "--forkbgworker") == 0) { /* do this as early as possible; in particular, before InitProcess() */ diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index 9270d7b855..04271ee703 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -6,6 +6,9 @@ * loaded as a dynamic module to avoid linking the main server binary with * libpq. * + * Apart from walreceiver, the libpq-specific routines are now being used by + * logical replication workers and slot synchronization. + * * Portions Copyright (c) 2010-2024, PostgreSQL Global Development Group * * diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 4cab7b7101..1f6ba0762e 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -10,18 +10,25 @@ * * This file contains the code for slot synchronization on a physical standby * to fetch logical failover slots information from the primary server, create - * the slots on the standby and synchronize them. This is done by a call to SQL - * function pg_sync_replication_slots. + * the slots on the standby and synchronize them periodically. * - * If on physical standby, the WAL corresponding to the remote's restart_lsn - * is not available or the remote's catalog_xmin precedes the oldest xid for which - * it is guaranteed that rows wouldn't have been removed then we cannot create - * the local standby slot because that would mean moving the local slot + * Slot synchronization can be performed either automatically by enabling slot + * sync worker or manually by calling SQL function pg_sync_replication_slots(). + * + * If the WAL corresponding to the remote's restart_lsn is not available on the + * physical standby or the remote's catalog_xmin precedes the oldest xid for + * which it is guaranteed that rows wouldn't have been removed then we cannot + * create the local standby slot because that would mean moving the local slot * backward and decoding won't be possible via such a slot. In this case, the * slot will be marked as RS_TEMPORARY. Once the primary server catches up, * the slot will be marked as RS_PERSISTENT (which means sync-ready) after - * which we can call pg_sync_replication_slots() periodically to perform - * syncs. + * which slot sync worker can perform the sync periodically or user can call + * pg_sync_replication_slots() periodically to perform the syncs. + * + * The slot sync worker waits for some time before the next synchronization, + * with the duration varying based on whether any slots were updated during + * the last cycle. Refer to the comments above wait_for_slot_activity() for + * more details. * * Any standby synchronized slots will be dropped if they no longer need * to be synchronized. See comment atop drop_local_obsolete_slots() for more @@ -31,31 +38,80 @@ #include "postgres.h" +#include <time.h> + #include "access/xlog_internal.h" #include "access/xlogrecovery.h" #include "catalog/pg_database.h" #include "commands/dbcommands.h" +#include "libpq/pqsignal.h" +#include "pgstat.h" +#include "postmaster/fork_process.h" +#include "postmaster/interrupt.h" +#include "postmaster/postmaster.h" #include "replication/logical.h" #include "replication/slotsync.h" #include "storage/ipc.h" #include "storage/lmgr.h" +#include "storage/proc.h" #include "storage/procarray.h" +#include "tcop/tcopprot.h" #include "utils/builtins.h" #include "utils/pg_lsn.h" +#include "utils/ps_status.h" +#include "utils/timeout.h" -/* Struct for sharing information to control slot synchronization. */ -typedef struct SlotSyncCtxStruct +/* + * Struct for sharing information to control slot synchronization. + * + * Slot sync worker's pid is needed by the startup process in order to shut it + * down during promotion. Startup process shuts down the slot sync worker and + * also sets stopSignaled=true to handle the race condition when postmaster has + * not noticed the promotion yet and thus may end up restarting slot sync + * worker. If stopSignaled is set, the worker will exit in such a case. + * + * The 'syncing' flag is needed to prevent concurrent slot syncs to avoid slot + * overwrites. + * + * The last_start_time is needed by postmaster to start the slot sync worker + * once per SLOTSYNC_RESTART_INTERVAL_SEC. In cases where a immediate restart + * is expected (e.g., slot sync GUCs change), slot sync worker will reset + * last_start_time before exiting, so that postmaster can start the worker + * without waiting for SLOTSYNC_RESTART_INTERVAL_SEC. + */ +typedef struct SlotSyncWorkerCtxStruct { - /* prevents concurrent slot syncs to avoid slot overwrites */ + pid_t pid; + bool stopSignaled; bool syncing; + time_t last_start_time; slock_t mutex; -} SlotSyncCtxStruct; +} SlotSyncWorkerCtxStruct; + +SlotSyncWorkerCtxStruct *SlotSyncWorker = NULL; -SlotSyncCtxStruct *SlotSyncCtx = NULL; +/* GUC variable */ +bool sync_replication_slots = false; + +/* + * The sleep time (ms) between slot-sync cycles varies dynamically + * (within a MIN/MAX range) according to slot activity. See + * wait_for_slot_activity() for details. + */ +#define MIN_WORKER_NAPTIME_MS 200 +#define MAX_WORKER_NAPTIME_MS 30000 /* 30s */ + +static long sleep_ms = MIN_WORKER_NAPTIME_MS; + +/* The restart interval for slot sync work used by postmaster */ +#define SLOTSYNC_RESTART_INTERVAL_SEC 10 + +/* Flag to tell if we are in a slot sync worker process */ +static bool am_slotsync_worker = false; /* * Flag to tell if we are syncing replication slots. Unlike the 'syncing' flag - * in SlotSyncCtxStruct, this flag is true only if the current process is + * in SlotSyncWorkerCtxStruct, this flag is true only if the current process is * performing slot synchronization. */ static bool syncing_slots = false; @@ -79,6 +135,15 @@ typedef struct RemoteSlot ReplicationSlotInvalidationCause invalidated; } RemoteSlot; +static void ProcessSlotSyncInterrupts(WalReceiverConn *wrconn, bool restart); + +#ifdef EXEC_BACKEND +static pid_t slotsyncworker_forkexec(void); +#endif +NON_EXEC_STATIC void ReplSlotSyncWorkerMain(int argc, char *argv[]) pg_attribute_noreturn(); + +static void slotsync_failure_callback(int code, Datum arg); + /* * If necessary, update the local synced slot's metadata based on the data * from the remote slot. @@ -343,8 +408,11 @@ reserve_wal_for_local_slot(XLogRecPtr restart_lsn) * If the remote restart_lsn and catalog_xmin have caught up with the * local ones, then update the LSNs and persist the local synced slot for * future synchronization; otherwise, do nothing. + * + * Return true if the slot is marked as RS_PERSISTENT (sync-ready), otherwise + * false. */ -static void +static bool update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) { ReplicationSlot *slot = MyReplicationSlot; @@ -376,7 +444,7 @@ update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) LSN_FORMAT_ARGS(slot->data.restart_lsn), slot->data.catalog_xmin)); - return; + return false; } /* First time slot update, the function must return true */ @@ -388,6 +456,8 @@ update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) ereport(LOG, errmsg("newly created slot \"%s\" is sync-ready now", remote_slot->name)); + + return true; } /* @@ -400,12 +470,15 @@ update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) * the remote_slot catches up with locally reserved position and local slot is * updated. The slot is then persisted and is considered as sync-ready for * periodic syncs. + * + * Returns TRUE if the local slot is updated. */ -static void +static bool synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) { ReplicationSlot *slot; XLogRecPtr latestFlushPtr; + bool slot_updated = false; /* * Make sure that concerned WAL is received and flushed before syncing @@ -413,13 +486,17 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) */ latestFlushPtr = GetStandbyFlushRecPtr(NULL); if (remote_slot->confirmed_lsn > latestFlushPtr) - elog(ERROR, + { + elog(am_slotsync_worker ? LOG : ERROR, "skipping slot synchronization as the received slot sync" " LSN %X/%X for slot \"%s\" is ahead of the standby position %X/%X", LSN_FORMAT_ARGS(remote_slot->confirmed_lsn), remote_slot->name, LSN_FORMAT_ARGS(latestFlushPtr)); + return false; + } + /* Search for the named slot */ if ((slot = SearchNamedReplicationSlot(remote_slot->name, true))) { @@ -466,19 +543,22 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) /* Make sure the invalidated state persists across server restart */ ReplicationSlotMarkDirty(); ReplicationSlotSave(); + + slot_updated = true; } /* Skip the sync of an invalidated slot */ if (slot->data.invalidated != RS_INVAL_NONE) { ReplicationSlotRelease(); - return; + return slot_updated; } /* Slot not ready yet, let's attempt to make it sync-ready now. */ if (slot->data.persistency == RS_TEMPORARY) { - update_and_persist_local_synced_slot(remote_slot, remote_dbid); + slot_updated = update_and_persist_local_synced_slot(remote_slot, + remote_dbid); } /* Slot ready for sync, so sync it. */ @@ -501,6 +581,8 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) { ReplicationSlotMarkDirty(); ReplicationSlotSave(); + + slot_updated = true; } } } @@ -512,7 +594,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) /* Skip creating the local slot if remote_slot is invalidated already */ if (remote_slot->invalidated != RS_INVAL_NONE) - return; + return false; /* * We create temporary slots instead of ephemeral slots here because @@ -549,9 +631,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) LWLockRelease(ProcArrayLock); update_and_persist_local_synced_slot(remote_slot, remote_dbid); + + slot_updated = true; } ReplicationSlotRelease(); + + return slot_updated; } /* @@ -559,8 +645,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) * * Gets the failover logical slots info from the primary server and updates * the slots locally. Creates the slots if not present on the standby. + * + * Returns TRUE if any of the slots gets updated in this sync-cycle. */ -static void +static bool synchronize_slots(WalReceiverConn *wrconn) { #define SLOTSYNC_COLUMN_COUNT 9 @@ -571,21 +659,30 @@ synchronize_slots(WalReceiverConn *wrconn) TupleTableSlot *tupslot; StringInfoData s; List *remote_slot_list = NIL; + bool some_slot_updated = false; + bool started_tx = false; - SpinLockAcquire(&SlotSyncCtx->mutex); - if (SlotSyncCtx->syncing) + SpinLockAcquire(&SlotSyncWorker->mutex); + if (SlotSyncWorker->syncing) { - SpinLockRelease(&SlotSyncCtx->mutex); + SpinLockRelease(&SlotSyncWorker->mutex); ereport(ERROR, errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), errmsg("cannot synchronize replication slots concurrently")); } - SlotSyncCtx->syncing = true; - SpinLockRelease(&SlotSyncCtx->mutex); + SlotSyncWorker->syncing = true; + SpinLockRelease(&SlotSyncWorker->mutex); syncing_slots = true; + /* The syscache access in walrcv_exec() needs a transaction env. */ + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + initStringInfo(&s); /* Construct query to fetch slots with failover enabled. */ @@ -694,7 +791,7 @@ synchronize_slots(WalReceiverConn *wrconn) */ LockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); - synchronize_one_slot(remote_slot, remote_dbid); + some_slot_updated |= synchronize_one_slot(remote_slot, remote_dbid); UnlockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); } @@ -704,21 +801,26 @@ synchronize_slots(WalReceiverConn *wrconn) walrcv_clear_result(res); - SpinLockAcquire(&SlotSyncCtx->mutex); - SlotSyncCtx->syncing = false; - SpinLockRelease(&SlotSyncCtx->mutex); + if (started_tx) + CommitTransactionCommand(); + + SpinLockAcquire(&SlotSyncWorker->mutex); + SlotSyncWorker->syncing = false; + SpinLockRelease(&SlotSyncWorker->mutex); syncing_slots = false; + + return some_slot_updated; } /* * Checks the remote server info. * - * We ensure that the 'primary_slot_name' exists on the remote server and the - * remote server is not a standby node. + * Check whether we are a cascading standby. For non-cascading standbys, it + * also ensures that the 'primary_slot_name' exists on the remote server. */ -static void -validate_remote_info(WalReceiverConn *wrconn) +static bool +validate_remote_info(WalReceiverConn *wrconn, int elevel) { #define PRIMARY_INFO_OUTPUT_COL_COUNT 2 WalRcvExecResult *res; @@ -728,6 +830,7 @@ validate_remote_info(WalReceiverConn *wrconn) TupleTableSlot *tupslot; bool remote_in_recovery; bool primary_slot_valid; + bool started_tx = false; initStringInfo(&cmd); appendStringInfo(&cmd, @@ -736,6 +839,13 @@ validate_remote_info(WalReceiverConn *wrconn) " WHERE slot_type='physical' AND slot_name=%s", quote_literal_cstr(PrimarySlotName)); + /* The syscache access in walrcv_exec() needs a transaction env. */ + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + res = walrcv_exec(wrconn, cmd.data, PRIMARY_INFO_OUTPUT_COL_COUNT, slotRow); pfree(cmd.data); @@ -754,31 +864,89 @@ validate_remote_info(WalReceiverConn *wrconn) Assert(!isnull); if (remote_in_recovery) - ereport(ERROR, + { + /* + * If we are a cascading standby, no need to check further for + * 'primary_slot_name'. + */ + ereport(elevel, errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot synchronize replication slots from a standby server")); + } + else + { + primary_slot_valid = DatumGetBool(slot_getattr(tupslot, 2, &isnull)); + Assert(!isnull); - primary_slot_valid = DatumGetBool(slot_getattr(tupslot, 2, &isnull)); - Assert(!isnull); - - if (!primary_slot_valid) - ereport(ERROR, - errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("bad configuration for slot synchronization"), - /* translator: second %s is a GUC variable name */ - errdetail("The replication slot \"%s\" specified by \"%s\" does not exist on the primary server.", - PrimarySlotName, "primary_slot_name")); + if (!primary_slot_valid) + ereport(elevel, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + /* translator: second %s is a GUC variable name */ + errdetail("The replication slot \"%s\" specified by \"%s\" does not exist on the primary server.", + PrimarySlotName, "primary_slot_name")); + } ExecClearTuple(tupslot); walrcv_clear_result(res); + + if (started_tx) + CommitTransactionCommand(); + + return (!remote_in_recovery && primary_slot_valid); } /* - * Check all necessary GUCs for slot synchronization are set - * appropriately, otherwise, raise ERROR. + * Check that we are not a cascading standby and 'primary_slot_name' exists + * on the remote server. If not, sleep for MAX_WORKER_NAPTIME_MS and check + * again. The idea is to become no-op until we get valid remote info. */ -void -ValidateSlotSyncParams(void) +static void +ensure_valid_remote_info(WalReceiverConn *wrconn) +{ + int rc; + + for (;;) + { + if (validate_remote_info(wrconn, LOG)) + break; + + rc = WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + MAX_WORKER_NAPTIME_MS, + WAIT_EVENT_REPL_SLOTSYNC_MAIN); + + if (rc & WL_LATCH_SET) + ResetLatch(MyLatch); + + ProcessSlotSyncInterrupts(NULL, true); + } +} + +/* + * Get database name from the conninfo. + * + * If dbname is extracted already from the conninfo, just return it. + */ +static char * +get_dbname_from_conninfo(const char *conninfo) +{ + static char *dbname; + + if (dbname) + return dbname; + else + dbname = walrcv_get_dbname_from_conninfo(conninfo); + + return dbname; +} + +/* + * Return true if all necessary GUCs for slot synchronization are set + * appropriately, otherwise, return false. + */ +bool +ValidateSlotSyncParams(int elevel) { char *dbname; @@ -789,11 +957,14 @@ ValidateSlotSyncParams(void) * be invalidated. */ if (PrimarySlotName == NULL || *PrimarySlotName == '\0') - ereport(ERROR, + { + ereport(elevel, /* translator: %s is a GUC variable name */ errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("\"%s\" must be defined.", "primary_slot_name")); + return false; + } /* * hot_standby_feedback must be enabled to cooperate with the physical @@ -801,37 +972,47 @@ ValidateSlotSyncParams(void) * catalog_xmin values on the standby. */ if (!hot_standby_feedback) - ereport(ERROR, + { + ereport(elevel, /* translator: %s is a GUC variable name */ errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("\"%s\" must be enabled.", "hot_standby_feedback")); + return false; + } /* Logical slot sync/creation requires wal_level >= logical. */ if (wal_level < WAL_LEVEL_LOGICAL) - ereport(ERROR, + { + ereport(elevel, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("\"wal_level\" must be >= logical.")); + return false; + } /* * The primary_conninfo is required to make connection to primary for * getting slots information. */ if (PrimaryConnInfo == NULL || *PrimaryConnInfo == '\0') - ereport(ERROR, + { + ereport(elevel, /* translator: %s is a GUC variable name */ errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("\"%s\" must be defined.", "primary_conninfo")); + return false; + } /* * The slot synchronization needs a database connection for walrcv_exec to * work. */ - dbname = walrcv_get_dbname_from_conninfo(PrimaryConnInfo); + dbname = get_dbname_from_conninfo(PrimaryConnInfo); if (dbname == NULL) - ereport(ERROR, + { + ereport(elevel, /* * translator: 'dbname' is a specific option; %s is a GUC variable @@ -840,10 +1021,491 @@ ValidateSlotSyncParams(void) errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("'dbname' must be specified in \"%s\".", "primary_conninfo")); + return false; + } + + return true; +} + +/* + * Check that all necessary GUCs for slot synchronization are set + * appropriately. If not, sleep for MAX_WORKER_NAPTIME_MS and check again. + * The idea is to become no-op until we get valid GUCs values. + */ +static void +ensure_valid_slotsync_params(void) +{ + int rc; + + /* Sanity check. */ + Assert(sync_replication_slots); + + for (;;) + { + if (ValidateSlotSyncParams(LOG)) + { + ereport(LOG, errmsg("parameters validation done for slot synchronization")); + break; + } + + ereport(LOG, errmsg("skipping slot synchronization")); + + rc = WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + MAX_WORKER_NAPTIME_MS, + WAIT_EVENT_REPL_SLOTSYNC_MAIN); + + if (rc & WL_LATCH_SET) + ResetLatch(MyLatch); + + /* + * Process the interrupts. Pass 'restart' as false as we do not want + * the worker to be restarted on GUC change. + */ + ProcessSlotSyncInterrupts(NULL, false); + } +} + +/* + * Re-read the config file. + * + * If any of the slot sync GUCs have changed, exit the worker and + * let it get restarted by the postmaster. The worker to be exited for + * restart purpose only if the caller passed restart as true. + */ +static void +slotsync_reread_config(bool restart) +{ + char *old_primary_conninfo = pstrdup(PrimaryConnInfo); + char *old_primary_slotname = pstrdup(PrimarySlotName); + bool old_sync_replication_slots = sync_replication_slots; + bool old_hot_standby_feedback = hot_standby_feedback; + bool conninfo_changed; + bool primary_slotname_changed; + + Assert(sync_replication_slots); + + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + + conninfo_changed = strcmp(old_primary_conninfo, PrimaryConnInfo) != 0; + primary_slotname_changed = strcmp(old_primary_slotname, PrimarySlotName) != 0; + pfree(old_primary_conninfo); + pfree(old_primary_slotname); + + if (old_sync_replication_slots != sync_replication_slots) + { + ereport(LOG, + /* translator: %s is a GUC variable name */ + errmsg("slot sync worker will shutdown because %s is disabled", "sync_replication_slots")); + proc_exit(0); + } + + /* The caller instructed to skip restart */ + if (!restart) + return; + + if (conninfo_changed || + primary_slotname_changed || + (old_hot_standby_feedback != hot_standby_feedback)) + { + ereport(LOG, + errmsg("slot sync worker will restart because of a parameter change")); + + /* + * Reset the last-start time for this worker so that the postmaster + * can restart it without waiting for SLOTSYNC_RESTART_INTERVAL_SEC. + */ + SlotSyncWorker->last_start_time = 0; + + proc_exit(0); + } + } /* - * Is current process syncing replication slots ? + * Interrupt handler for main loop of slot sync worker. + */ +static void +ProcessSlotSyncInterrupts(WalReceiverConn *wrconn, bool restart) +{ + CHECK_FOR_INTERRUPTS(); + + if (ShutdownRequestPending) + { + ereport(LOG, + errmsg("replication slot sync worker is shutting down on receiving SIGINT")); + + proc_exit(0); + } + + if (ConfigReloadPending) + slotsync_reread_config(restart); +} + +/* + * Cleanup function for slotsync worker. + * + * Called on slotsync worker exit. + */ +static void +slotsync_worker_onexit(int code, Datum arg) +{ + SpinLockAcquire(&SlotSyncWorker->mutex); + SlotSyncWorker->pid = InvalidPid; + SpinLockRelease(&SlotSyncWorker->mutex); +} + +/* + * Sleep for long enough that we believe it's likely that the slots on primary + * get updated. + * + * If there is no slot activity the wait time between sync-cycles will double + * (to a maximum of 30s). If there is some slot activity the wait time between + * sync-cycles is reset to the minimum (200ms). + */ +static void +wait_for_slot_activity(bool some_slot_updated) +{ + int rc; + + if (!some_slot_updated) + { + /* + * No slots were updated, so double the sleep time, but not beyond the + * maximum allowable value. + */ + sleep_ms = Min(sleep_ms * 2, MAX_WORKER_NAPTIME_MS); + } + else + { + /* + * Some slots were updated since the last sleep, so reset the sleep + * time. + */ + sleep_ms = MIN_WORKER_NAPTIME_MS; + } + + rc = WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + sleep_ms, + WAIT_EVENT_REPL_SLOTSYNC_MAIN); + + if (rc & WL_LATCH_SET) + ResetLatch(MyLatch); +} + +/* + * The main loop of our worker process. + * + * It connects to the primary server, fetches logical failover slots + * information periodically in order to create and sync the slots. + */ +NON_EXEC_STATIC void +ReplSlotSyncWorkerMain(int argc, char *argv[]) +{ + WalReceiverConn *wrconn = NULL; + char *dbname; + char *err; + sigjmp_buf local_sigjmp_buf; + StringInfoData app_name; + + am_slotsync_worker = true; + + MyBackendType = B_SLOTSYNC_WORKER; + + init_ps_display(NULL); + + SetProcessingMode(InitProcessing); + + /* + * Create a per-backend PGPROC struct in shared memory. We must do this + * before we access any shared memory. + */ + InitProcess(); + + /* + * Early initialization. + */ + BaseInit(); + + Assert(SlotSyncWorker != NULL); + + SpinLockAcquire(&SlotSyncWorker->mutex); + Assert(SlotSyncWorker->pid == InvalidPid); + + /* + * Startup process signaled the slot sync worker to stop, so if meanwhile + * postmaster ended up starting the worker again, exit. + */ + if (SlotSyncWorker->stopSignaled) + { + SpinLockRelease(&SlotSyncWorker->mutex); + proc_exit(0); + } + + /* Advertise our PID so that the startup process can kill us on promotion */ + SlotSyncWorker->pid = MyProcPid; + SpinLockRelease(&SlotSyncWorker->mutex); + + ereport(LOG, errmsg("replication slot sync worker started")); + + on_shmem_exit(slotsync_worker_onexit, (Datum) 0); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGINT, SignalHandlerForShutdownRequest); + pqsignal(SIGTERM, die); + pqsignal(SIGFPE, FloatExceptionHandler); + pqsignal(SIGUSR1, procsignal_sigusr1_handler); + pqsignal(SIGUSR2, SIG_IGN); + pqsignal(SIGPIPE, SIG_IGN); + pqsignal(SIGCHLD, SIG_DFL); + + /* + * Establishes SIGALRM handler and initialize timeout module. It is needed + * by InitPostgres to register different timeouts. + */ + InitializeTimeouts(); + + /* Load the libpq-specific functions */ + load_file("libpqwalreceiver", false); + + /* + * If an exception is encountered, processing resumes here. + * + * We just need to clean up, report the error, and go away. + * + * If we do not have this handling here, then since this worker process + * operates at the bottom of the exception stack, ERRORs turn into FATALs. + * Therefore, we create our own exception handler to catch ERRORs. + */ + if (sigsetjmp(local_sigjmp_buf, 1) != 0) + { + /* since not using PG_TRY, must reset error stack by hand */ + error_context_stack = NULL; + + /* Prevents interrupts while cleaning up */ + HOLD_INTERRUPTS(); + + /* Report the error to the server log */ + EmitErrorReport(); + + /* + * We can now go away. Note that because we called InitProcess, a + * callback was registered to do ProcKill, which will clean up + * necessary state. + */ + proc_exit(0); + } + + /* We can now handle ereport(ERROR) */ + PG_exception_stack = &local_sigjmp_buf; + + /* + * Unblock signals (they were blocked when the postmaster forked us) + */ + sigprocmask(SIG_SETMASK, &UnBlockSig, NULL); + + ensure_valid_slotsync_params(); + + dbname = get_dbname_from_conninfo(PrimaryConnInfo); + + /* + * Connect to the database specified by user in primary_conninfo. We need + * a database connection for walrcv_exec to work. Please see comments atop + * libpqrcv_exec. + */ + InitPostgres(dbname, InvalidOid, NULL, InvalidOid, 0, NULL); + + SetProcessingMode(NormalProcessing); + + initStringInfo(&app_name); + if (cluster_name[0]) + appendStringInfo(&app_name, "%s_%s", cluster_name, "slotsyncworker"); + else + appendStringInfo(&app_name, "%s", "slotsyncworker"); + + /* + * Establish the connection to the primary server for slots + * synchronization. + */ + wrconn = walrcv_connect(PrimaryConnInfo, false, false, false, + app_name.data, + &err); + pfree(app_name.data); + + if (!wrconn) + ereport(ERROR, + errcode(ERRCODE_CONNECTION_FAILURE), + errmsg("could not connect to the primary server: %s", err)); + + before_shmem_exit(slotsync_failure_callback, PointerGetDatum(wrconn)); + + /* + * Using the specified primary server connection, ensure that we are not a + * cascading standby and slot configured in 'primary_slot_name' exists on + * the primary server. + */ + ensure_valid_remote_info(wrconn); + + /* Main wait loop */ + for (;;) + { + bool some_slot_updated = false; + + ProcessSlotSyncInterrupts(wrconn, true); + + some_slot_updated = synchronize_slots(wrconn); + + wait_for_slot_activity(some_slot_updated); + } + + /* + * The slot sync worker can not get here because it will only stop when it + * receives a SIGINT from the startup process, or when there is an error. + */ + Assert(false); +} + +/* + * Main entry point for slot sync worker process, to be called from the + * postmaster. + */ +int +StartSlotSyncWorker(void) +{ + pid_t pid; + +#ifdef EXEC_BACKEND + switch ((pid = slotsyncworker_forkexec())) + { +#else + switch ((pid = fork_process())) + { + case 0: + /* in postmaster child ... */ + InitPostmasterChild(); + + /* Close the postmaster's sockets */ + ClosePostmasterPorts(false); + + ReplSlotSyncWorkerMain(0, NULL); + break; +#endif + case -1: + ereport(LOG, + (errmsg("could not fork slot sync worker process: %m"))); + return 0; + + default: + return (int) pid; + } + + /* shouldn't get here */ + return 0; +} + +#ifdef EXEC_BACKEND +/* + * The forkexec routine for the slot sync worker process. + * + * Format up the arglist, then fork and exec. + */ +static pid_t +slotsyncworker_forkexec(void) +{ + char *av[10]; + int ac = 0; + + av[ac++] = "postgres"; + av[ac++] = "--forkssworker"; + av[ac++] = NULL; /* filled in by postmaster_forkexec */ + av[ac] = NULL; + + Assert(ac < lengthof(av)); + + return postmaster_forkexec(ac, av); +} +#endif + +/* + * Shut down the slot sync worker. + */ +void +ShutDownSlotSync(void) +{ + SpinLockAcquire(&SlotSyncWorker->mutex); + + SlotSyncWorker->stopSignaled = true; + + if (SlotSyncWorker->pid == InvalidPid) + { + SpinLockRelease(&SlotSyncWorker->mutex); + return; + } + SpinLockRelease(&SlotSyncWorker->mutex); + + kill(SlotSyncWorker->pid, SIGINT); + + /* Wait for it to die */ + for (;;) + { + int rc; + + /* Wait a bit, we don't expect to have to wait long */ + rc = WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, WAIT_EVENT_REPL_SLOTSYNC_SHUTDOWN); + + if (rc & WL_LATCH_SET) + { + ResetLatch(MyLatch); + CHECK_FOR_INTERRUPTS(); + } + + SpinLockAcquire(&SlotSyncWorker->mutex); + + /* Is it gone? */ + if (SlotSyncWorker->pid == InvalidPid) + break; + + SpinLockRelease(&SlotSyncWorker->mutex); + } + + SpinLockRelease(&SlotSyncWorker->mutex); +} + +/* + * SlotSyncWorkerCanRestart + * + * Returns true if enough time has passed (SLOTSYNC_RESTART_INTERVAL_SEC) + * since it was launched last. Otherwise returns false. + * + * This is a safety valve to protect against continuous respawn attempts if the + * worker is dying immediately at launch. Note that since we will retry to + * launch the worker from the postmaster main loop, we will get another + * chance later. + */ +bool +SlotSyncWorkerCanRestart(void) +{ + time_t curtime = time(NULL); + + /* Return false if too soon since last start. */ + if ((unsigned int) (curtime - SlotSyncWorker->last_start_time) < + (unsigned int) SLOTSYNC_RESTART_INTERVAL_SEC) + return false; + + SlotSyncWorker->last_start_time = curtime; + + return true; +} + +/* + * Is current process syncing replication slots? + * + * Could be either backend executing SQL function or slot sync worker. */ bool IsSyncingReplicationSlots(void) @@ -851,30 +1513,41 @@ IsSyncingReplicationSlots(void) return syncing_slots; } +/* + * Is current process a slot sync worker? + */ +bool +IsLogicalSlotSyncWorker(void) +{ + return am_slotsync_worker; +} + /* * Amount of shared memory required for slot synchronization. */ Size -SlotSyncShmemSize(void) +SlotSyncWorkerShmemSize(void) { - return sizeof(SlotSyncCtxStruct); + return sizeof(SlotSyncWorkerCtxStruct); } /* * Allocate and initialize the shared memory of slot synchronization. */ void -SlotSyncShmemInit(void) +SlotSyncWorkerShmemInit(void) { + Size size = SlotSyncWorkerShmemSize(); bool found; - SlotSyncCtx = (SlotSyncCtxStruct *) - ShmemInitStruct("Slot Sync Data", SlotSyncShmemSize(), &found); + SlotSyncWorker = (SlotSyncWorkerCtxStruct *) + ShmemInitStruct("Slot Sync Worker Data", size, &found); if (!found) { - SlotSyncCtx->syncing = false; - SpinLockInit(&SlotSyncCtx->mutex); + memset(SlotSyncWorker, 0, size); + SlotSyncWorker->pid = InvalidPid; + SpinLockInit(&SlotSyncWorker->mutex); } } @@ -893,9 +1566,9 @@ slotsync_failure_callback(int code, Datum arg) * without resetting the flag. So, we need to clean up shared memory * and reset the flag here. */ - SpinLockAcquire(&SlotSyncCtx->mutex); - SlotSyncCtx->syncing = false; - SpinLockRelease(&SlotSyncCtx->mutex); + SpinLockAcquire(&SlotSyncWorker->mutex); + SlotSyncWorker->syncing = false; + SpinLockRelease(&SlotSyncWorker->mutex); syncing_slots = false; } @@ -912,7 +1585,7 @@ SyncReplicationSlots(WalReceiverConn *wrconn) { PG_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); { - validate_remote_info(wrconn); + validate_remote_info(wrconn, ERROR); synchronize_slots(wrconn); } diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index 2180a38063..dd4ea0388a 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -1236,6 +1236,20 @@ restart: * concurrently being dropped by a backend connected to another DB. * * That's fairly unlikely in practice, so we'll just bail out. + * + * The slot sync worker holds a shared lock on the database before + * operating on synced logical slots to avoid conflict with the drop + * happening here. The persistent synced slots are thus safe but there + * is a possibility that the slot sync worker has created a temporary + * slot (which stays active even on release) and we are trying to drop + * the same here. In practice, the chances of hitting this scenario is + * very less as during slot synchronization, the temporary slot is + * immediately converted to persistent and thus is safe due to the + * shared lock taken on the database. So for the time being, we'll + * just bail out in such a scenario. + * + * XXX: If needed, we can consider shutting down slot sync worker + * before trying to drop synced temporary slots here. */ if (active_pid) ereport(ERROR, diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index d2fa5e669a..c4a0bf99ee 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -975,7 +975,7 @@ pg_sync_replication_slots(PG_FUNCTION_ARGS) /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); - ValidateSlotSyncParams(); + ValidateSlotSyncParams(ERROR); initStringInfo(&app_name); if (cluster_name[0]) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index e5477c1de1..b33044e10f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -3389,7 +3389,7 @@ WalSndDone(WalSndSendDataCallback send_data) * This should only be called when in recovery. * * This is called either by cascading walsender to find WAL postion to be sent - * to a cascaded standby or by slot synchronization function to validate remote + * to a cascaded standby or by slot synchronization operation to validate remote * slot's lsn before syncing it locally. * * As a side-effect, *tli is updated to the TLI of the last diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index 7e7941d625..8ecd89f8ea 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -154,7 +154,7 @@ CalculateShmemSize(int *num_semaphores) size = add_size(size, StatsShmemSize()); size = add_size(size, WaitEventExtensionShmemSize()); size = add_size(size, InjectionPointShmemSize()); - size = add_size(size, SlotSyncShmemSize()); + size = add_size(size, SlotSyncWorkerShmemSize()); #ifdef EXEC_BACKEND size = add_size(size, ShmemBackendArraySize()); #endif @@ -349,7 +349,7 @@ CreateOrAttachShmemStructs(void) WalSummarizerShmemInit(); PgArchShmemInit(); ApplyLauncherShmemInit(); - SlotSyncShmemInit(); + SlotSyncWorkerShmemInit(); /* * Set up other modules that need some shared memory space diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c index 1afcbfc052..ac83331bbc 100644 --- a/src/backend/storage/lmgr/proc.c +++ b/src/backend/storage/lmgr/proc.c @@ -40,6 +40,7 @@ #include "pgstat.h" #include "postmaster/autovacuum.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/syncrep.h" #include "replication/walsender.h" #include "storage/condition_variable.h" @@ -365,8 +366,12 @@ InitProcess(void) * child; this is so that the postmaster can detect it if we exit without * cleaning up. (XXX autovac launcher currently doesn't participate in * this; it probably should.) + * + * Slot sync worker also does not participate in it, see comments atop + * 'struct bkend' in postmaster.c. */ - if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess()) + if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && + !IsLogicalSlotSyncWorker()) MarkPostmasterChildActive(); /* @@ -935,8 +940,12 @@ ProcKill(int code, Datum arg) * This process is no longer present in shared memory in any meaningful * way, so tell the postmaster we've cleaned up acceptably well. (XXX * autovac launcher should be included here someday) + * + * Slot sync worker is also not a postmaster child, so skip this shared + * memory related processing here. */ - if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess()) + if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && + !IsLogicalSlotSyncWorker()) MarkPostmasterChildInactive(); /* wake autovac launcher if needed -- see comments in FreeWorkerInfo */ diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 43c393d6fe..9d6e067382 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -338,6 +338,7 @@ pgstat_tracks_io_bktype(BackendType bktype) case B_BG_WORKER: case B_BG_WRITER: case B_CHECKPOINTER: + case B_SLOTSYNC_WORKER: case B_STANDALONE_BACKEND: case B_STARTUP: case B_WAL_SENDER: diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 6464386b77..b52afd4eac 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -53,6 +53,8 @@ LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process." LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process." LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process." RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery." +REPL_SLOTSYNC_MAIN "Waiting in main loop of slot sync worker." +REPL_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down." SYSLOGGER_MAIN "Waiting in main loop of syslogger process." WAL_RECEIVER_MAIN "Waiting in main loop of WAL receiver process." WAL_SENDER_MAIN "Waiting in main loop of WAL sender process." diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c index 23f77a59e5..26aaf292c5 100644 --- a/src/backend/utils/init/miscinit.c +++ b/src/backend/utils/init/miscinit.c @@ -40,6 +40,7 @@ #include "postmaster/interrupt.h" #include "postmaster/pgarch.h" #include "postmaster/postmaster.h" +#include "replication/slotsync.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/latch.h" @@ -293,6 +294,9 @@ GetBackendTypeDesc(BackendType backendType) case B_LOGGER: backendDesc = "logger"; break; + case B_SLOTSYNC_WORKER: + backendDesc = "slotsyncworker"; + break; case B_STANDALONE_BACKEND: backendDesc = "standalone backend"; break; @@ -835,9 +839,10 @@ InitializeSessionUserIdStandalone(void) { /* * This function should only be called in single-user mode, in autovacuum - * workers, and in background workers. + * workers, in slot sync worker and in background workers. */ - Assert(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || IsBackgroundWorker); + Assert(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || + IsLogicalSlotSyncWorker() || IsBackgroundWorker); /* call only once */ Assert(!OidIsValid(AuthenticatedUserId)); diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 7797876d00..5ffe9bdd98 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -43,6 +43,7 @@ #include "postmaster/autovacuum.h" #include "postmaster/postmaster.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/walsender.h" #include "storage/bufmgr.h" #include "storage/fd.h" @@ -876,10 +877,11 @@ InitPostgres(const char *in_dbname, Oid dboid, * Perform client authentication if necessary, then figure out our * postgres user ID, and see if we are a superuser. * - * In standalone mode and in autovacuum worker processes, we use a fixed - * ID, otherwise we figure it out from the authenticated user name. + * In standalone mode, autovacuum worker processes and slot sync worker + * process, we use a fixed ID, otherwise we figure it out from the + * authenticated user name. */ - if (bootstrap || IsAutoVacuumWorkerProcess()) + if (bootstrap || IsAutoVacuumWorkerProcess() || IsLogicalSlotSyncWorker()) { InitializeSessionUserIdStandalone(); am_superuser = true; diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 70652f0a3f..37be0669bb 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -67,6 +67,7 @@ #include "postmaster/walwriter.h" #include "replication/logicallauncher.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/syncrep.h" #include "storage/bufmgr.h" #include "storage/large_object.h" @@ -2054,6 +2055,15 @@ struct config_bool ConfigureNamesBool[] = NULL, NULL, NULL }, + { + {"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY, + gettext_noop("Enables a physical standby to synchronize logical failover slots from the primary server."), + }, + &sync_replication_slots, + false, + NULL, NULL, NULL + }, + /* End-of-list marker */ { {NULL, 0, 0, NULL, NULL}, NULL, false, NULL, NULL, NULL diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index e10755972a..c97f9a25f0 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -361,6 +361,7 @@ #wal_retrieve_retry_interval = 5s # time to wait before retrying to # retrieve WAL after a failed attempt #recovery_min_apply_delay = 0 # minimum delay for applying changes during recovery +#sync_replication_slots = off # enables slot synchronization on the physical standby from the primary # - Subscribers - diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index 0445fbf61d..612fb5f42e 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -333,6 +333,7 @@ typedef enum BackendType B_BG_WRITER, B_CHECKPOINTER, B_LOGGER, + B_SLOTSYNC_WORKER, B_STANDALONE_BACKEND, B_STARTUP, B_WAL_RECEIVER, diff --git a/src/include/replication/slotsync.h b/src/include/replication/slotsync.h index e86d8a47b8..aa688a2f1b 100644 --- a/src/include/replication/slotsync.h +++ b/src/include/replication/slotsync.h @@ -14,10 +14,28 @@ #include "replication/walreceiver.h" -extern void ValidateSlotSyncParams(void); +extern PGDLLIMPORT bool sync_replication_slots; + +/* + * GUCs needed by slot sync worker to connect to the primary + * server and carry on with slots synchronization. + */ +extern PGDLLIMPORT char *PrimaryConnInfo; +extern PGDLLIMPORT char *PrimarySlotName; + +extern bool ValidateSlotSyncParams(int elevel); + +#ifdef EXEC_BACKEND +extern void ReplSlotSyncWorkerMain(int argc, char *argv[]) pg_attribute_noreturn(); +#endif +extern int StartSlotSyncWorker(void); + +extern void ShutDownSlotSync(void); +extern bool SlotSyncWorkerCanRestart(void); extern bool IsSyncingReplicationSlots(void); -extern Size SlotSyncShmemSize(void); -extern void SlotSyncShmemInit(void); +extern bool IsLogicalSlotSyncWorker(void); +extern Size SlotSyncWorkerShmemSize(void); +extern void SlotSyncWorkerShmemInit(void); extern void SyncReplicationSlots(WalReceiverConn *wrconn); #endif /* SLOTSYNC_H */ diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index 2755c3fc84..950f6b3738 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -22,7 +22,8 @@ $publisher->append_conf('postgresql.conf', 'autovacuum = off'); $publisher->start; $publisher->safe_psql('postgres', - "CREATE PUBLICATION regress_mypub FOR ALL TABLES;"); + "CREATE PUBLICATION regress_mypub FOR ALL TABLES;" +); my $publisher_connstr = $publisher->connstr . ' dbname=postgres'; @@ -322,6 +323,10 @@ ok( $stderr =~ /HINT: 'dbname' must be specified in "primary_conninfo"/, "cannot sync slots if dbname is not specified in primary_conninfo"); +# Add the dbname back to the primary_conninfo for further tests +$standby1->append_conf('postgresql.conf', "primary_conninfo = '$connstr_1 dbname=postgres'"); +$standby1->reload; + ################################################## # Test that we cannot synchronize slots to a cascading standby server. ################################################## @@ -355,4 +360,110 @@ ok( $stderr =~ /ERROR: cannot synchronize replication slots from a standby server/, "cannot sync slots to a cascading standby server"); +$cascading_standby->stop; + +################################################## +# Test to confirm that the slot sync worker waits until it gets valid values +# for all the slot sync related GUCs +################################################## + +$log_offset = -s $standby1->logfile; + +# Enable slot sync worker but disable another GUC required for slot sync +$standby1->append_conf( + 'postgresql.conf', qq( +sync_replication_slots = on +hot_standby_feedback = off +)); +$standby1->reload; + +# Confirm that the slot sync worker logs bad configuration msg +$standby1->wait_for_log(qr/LOG: bad configuration for slot synchronization/, + $log_offset); + +$log_offset = -s $standby1->logfile; + +$standby1->append_conf('postgresql.conf', "hot_standby_feedback = on"); +$standby1->reload; + +# Confirm that the slot sync worker proceeds once the configuration is correct. +$standby1->wait_for_log(qr/LOG: parameters validation done for slot synchronization/, + $log_offset); + +################################################## +# Test to confirm that restart_lsn and confirmed_flush_lsn of the logical slot +# on the primary is synced to the standby via the slot sync worker. +################################################## + +# Insert data on the primary +$primary->safe_psql( + 'postgres', qq[ + CREATE TABLE tab_int (a int PRIMARY KEY); + INSERT INTO tab_int SELECT generate_series(1, 10); +]); + +# Subscribe to the new table data and wait for it to arrive +$subscriber1->safe_psql( + 'postgres', qq[ + CREATE TABLE tab_int (a int PRIMARY KEY); + ALTER SUBSCRIPTION regress_mysub1 ENABLE; + ALTER SUBSCRIPTION regress_mysub1 REFRESH PUBLICATION; +]); + +$subscriber1->wait_for_subscription_sync; + +# Do not allow any further advancement of the restart_lsn and +# confirmed_flush_lsn for the lsub1_slot. +$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); + +# Wait for the replication slot to become inactive on the publisher +$primary->poll_query_until( + 'postgres', + "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active='f'", + 1); + +# Get the restart_lsn for the logical slot lsub1_slot on the primary +my $primary_restart_lsn = $primary->safe_psql('postgres', + "SELECT restart_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot';"); + +# Get the confirmed_flush_lsn for the logical slot lsub1_slot on the primary +my $primary_flush_lsn = $primary->safe_psql('postgres', + "SELECT confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot';"); + +# Confirm that restart_lsn and of confirmed_flush_lsn lsub1_slot slot are synced +# to the standby +ok( $standby1->poll_query_until( + 'postgres', + "SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot';"), + 'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby'); + +################################################## +# Promote the standby1 to primary. Confirm that: +# a) the slot 'lsub1_slot' is retained on the new primary +# b) logical replication for regress_mysub1 is resumed successfully after failover +################################################## +$standby1->promote; + +# Update subscription with the new primary's connection info +my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo'; + ALTER SUBSCRIPTION regress_mysub1 ENABLE; "); + +# Confirm the synced slot 'lsub1_slot' is retained on the new primary +is($standby1->safe_psql('postgres', + q{SELECT slot_name FROM pg_replication_slots WHERE slot_name = 'lsub1_slot';}), + 'lsub1_slot', + 'synced slot retained on the new primary'); + +# Insert data on the new primary +$standby1->safe_psql('postgres', + "INSERT INTO tab_int SELECT generate_series(11, 20);"); +$standby1->wait_for_catchup('regress_mysub1'); + +# Confirm that data in tab_int replicated on the subscriber +is( $subscriber1->safe_psql('postgres', q{SELECT count(*) FROM tab_int;}), + "20", + 'data replicated from the new primary'); + done_testing(); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index d808aad8b0..a35e399f0c 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2585,7 +2585,7 @@ SlabBlock SlabContext SlabSlot SlotNumber -SlotSyncCtxStruct +SlotSyncWorkerCtxStruct SlruCtl SlruCtlData SlruErrorCause -- 2.34.1 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-17 04:40 Amit Kapila <[email protected]> parent: shveta malik <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: Amit Kapila @ 2024-02-17 04:40 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> On Fri, Feb 16, 2024 at 4:10 PM shveta malik <[email protected]> wrote: > > On Thu, Feb 15, 2024 at 10:48 PM Bertrand Drouvot > <[email protected]> wrote: > > > 5 === > > > > + if (SlotSyncWorker->syncing) > > { > > - SpinLockRelease(&SlotSyncCtx->mutex); > > + SpinLockRelease(&SlotSyncWorker->mutex); > > ereport(ERROR, > > errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), > > errmsg("cannot synchronize replication slots concurrently")); > > } > > > > worth to add a test in 040_standby_failover_slots_sync.pl for it? > > It will be very difficult to stabilize this test as we have to make > sure that the concurrent users (SQL function(s) and/or worker(s)) are > in that target function at the same time to hit it. > Yeah, I also think would be tricky to write a stable test, maybe one can explore using a new injection point facility but I don't think it is worth for this error check as this appears straightforward to be broken in the future by other changes. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 46+ messages in thread
* RE: Synchronizing slots from primary to standby @ 2024-02-18 14:09 Zhijie Hou (Fujitsu) <[email protected]> parent: shveta malik <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: Zhijie Hou (Fujitsu) @ 2024-02-18 14:09 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>; Bertrand Drouvot <[email protected]> On Friday, February 16, 2024 6:41 PM shveta malik <[email protected]> wrote: > > On Thu, Feb 15, 2024 at 10:48 PM Bertrand Drouvot > <[email protected]> wrote: > > > > Looking at v88_0001, random comments: > > Thanks for the feedback. > > > > > 1 === > > > > Commit message "Be enabling slot synchronization" > > > > Typo? s:Be/By > > Modified. > > > 2 === > > > > + It enables a physical standby to synchronize logical failover slots > > + from the primary server so that logical subscribers are not blocked > > + after failover. > > > > Not sure "not blocked" is the right wording. > > "can be resumed from the new primary" maybe? (was discussed in [1]) > > Modified. > > > 3 === > > > > +#define SlotSyncWorkerAllowed() \ > > + (sync_replication_slots && pmState == PM_HOT_STANDBY && \ > > + SlotSyncWorkerCanRestart()) > > > > Maybe add a comment above the macro explaining the logic? > > Done. > > > 4 === > > > > +#include "replication/walreceiver.h" > > #include "replication/slotsync.h" > > > > should be reverse order? > > Removed walreceiver.h inclusion as it was not needed. > > > 5 === > > > > + if (SlotSyncWorker->syncing) > > { > > - SpinLockRelease(&SlotSyncCtx->mutex); > > + SpinLockRelease(&SlotSyncWorker->mutex); > > ereport(ERROR, > > > errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), > > errmsg("cannot synchronize replication > slots concurrently")); > > } > > > > worth to add a test in 040_standby_failover_slots_sync.pl for it? > > It will be very difficult to stabilize this test as we have to make sure that the > concurrent users (SQL function(s) and/or worker(s)) are in that target function at > the same time to hit it. > > > > > 6 === > > > > +static void > > +slotsync_reread_config(bool restart) > > +{ > > > > worth to add test(s) in 040_standby_failover_slots_sync.pl for it? > > Added test. > > Please find v89 patch set. The other changes are: Thanks for the patch. Here are few comments: 1. +static char * +get_dbname_from_conninfo(const char *conninfo) +{ + static char *dbname; + + if (dbname) + return dbname; + else + dbname = walrcv_get_dbname_from_conninfo(conninfo); + + return dbname; +} I think it's not necessary to have a static variable here, because the guc check doesn't seem performance sensitive. Additionaly, it does not work well with slotsync SQL functions, because the dbname will be allocated in temp memory context when reaching here via SQL function, so it's not safe to access this static variable in next function call. 2. +static bool +validate_remote_info(WalReceiverConn *wrconn, int elevel) ... + + return (!remote_in_recovery && primary_slot_valid); The primary_slot_valid could be uninitialized in this function. Best Regards, Hou zj ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-19 03:38 shveta malik <[email protected]> parent: Zhijie Hou (Fujitsu) <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: shveta malik @ 2024-02-19 03:38 UTC (permalink / raw) To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]> On Sun, Feb 18, 2024 at 7:40 PM Zhijie Hou (Fujitsu) <[email protected]> wrote: > > On Friday, February 16, 2024 6:41 PM shveta malik <[email protected]> wrote: > Thanks for the patch. Here are few comments: Thanks for the comments. > > 2. > > +static bool > +validate_remote_info(WalReceiverConn *wrconn, int elevel) > ... > + > + return (!remote_in_recovery && primary_slot_valid); > > The primary_slot_valid could be uninitialized in this function. return (!remote_in_recovery && primary_slot_valid); Here if remote_in_recovery is true, it will not even read primary_slot_valid. It will read primary_slot_valid only if remote_in_recovery is false and in such a case primary_slot_valid will always be initialized in the else block above, let me know if you still feel we shall initialize this to some default? thanks Shveta ^ permalink raw reply [nested|flat] 46+ messages in thread
* RE: Synchronizing slots from primary to standby @ 2024-02-19 04:02 Zhijie Hou (Fujitsu) <[email protected]> parent: shveta malik <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Zhijie Hou (Fujitsu) @ 2024-02-19 04:02 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>; Bertrand Drouvot <[email protected]> On Monday, February 19, 2024 11:39 AM shveta malik <[email protected]> wrote: > > On Sun, Feb 18, 2024 at 7:40 PM Zhijie Hou (Fujitsu) <[email protected]> > wrote: > > > > On Friday, February 16, 2024 6:41 PM shveta malik <[email protected]> > wrote: > > Thanks for the patch. Here are few comments: > > Thanks for the comments. > > > > > 2. > > > > +static bool > > +validate_remote_info(WalReceiverConn *wrconn, int elevel) > > ... > > + > > + return (!remote_in_recovery && primary_slot_valid); > > > > The primary_slot_valid could be uninitialized in this function. > > return (!remote_in_recovery && primary_slot_valid); > > Here if remote_in_recovery is true, it will not even read primary_slot_valid. It > will read primary_slot_valid only if remote_in_recovery is false and in such a > case primary_slot_valid will always be initialized in the else block above, let me > know if you still feel we shall initialize this to some default? I understand that it will not be used, but some complier could report WARNING for the un-initialized variable. The cfbot[1] seems complain about this as well. [1] https://cirrus-ci.com/task/5416851522453504 Best Regards, Hou zj ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-19 04:16 shveta malik <[email protected]> parent: Zhijie Hou (Fujitsu) <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: shveta malik @ 2024-02-19 04:16 UTC (permalink / raw) To: Zhijie Hou (Fujitsu) <[email protected]>; +Cc: Amit Kapila <[email protected]>; Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]> On Mon, Feb 19, 2024 at 9:32 AM Zhijie Hou (Fujitsu) <[email protected]> wrote: > > > I understand that it will not be used, but some complier could report WARNING > for the un-initialized variable. The cfbot[1] seems complain about this as well. > > [1] https://cirrus-ci.com/task/5416851522453504 Okay I see. Thanks for pointing it out. Here are the patches addressing your comments. Changes are in patch001, rest are rebased. thanks Shveta Attachments: [application/octet-stream] v90-0003-Document-the-steps-to-check-if-the-standby-is-re.patch (7.0K, ../../CAJpy0uBdktD4Kj9_LiaHyVHhpk959TQuQVVTbbQAGni=CzusVw@mail.gmail.com/2-v90-0003-Document-the-steps-to-check-if-the-standby-is-re.patch) download | inline diff: From 5f60959360d354c40890f3758c6f6d69b9288674 Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Fri, 19 Jan 2024 11:04:16 +0530 Subject: [PATCH v90 3/3] Document the steps to check if the standby is ready for failover --- doc/src/sgml/high-availability.sgml | 9 ++ doc/src/sgml/logical-replication.sgml | 136 ++++++++++++++++++++++++++ 2 files changed, 145 insertions(+) diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml index 236c0af65f..36215aa68c 100644 --- a/doc/src/sgml/high-availability.sgml +++ b/doc/src/sgml/high-availability.sgml @@ -1487,6 +1487,15 @@ synchronous_standby_names = 'ANY 2 (s1, s2, s3)' Written administration procedures are advised. </para> + <para> + If you have opted for synchronization of logical slots (see + <xref linkend="logicaldecoding-replication-slots-synchronization"/>), + then before switching to the standby server, it is recommended to check + if the logical slots synchronized on the standby server are ready + for failover. This can be done by following the steps described in + <xref linkend="logical-replication-failover"/>. + </para> + <para> To trigger failover of a log-shipping standby server, run <command>pg_ctl promote</command> or call <function>pg_promote()</function>. diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index ec2130669e..be59d306a1 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -687,6 +687,142 @@ ALTER SUBSCRIPTION </sect1> + <sect1 id="logical-replication-failover"> + <title>Logical Replication Failover</title> + + <para> + When the publisher server is the primary server of a streaming replication, + the logical slots on that primary server can be synchronized to the standby + server by specifying <literal>failover = true</literal> when creating + subscriptions for those publications. Enabling failover ensures a seamless + transition of those subscriptions after the standby is promoted. They can + continue subscribing to publications now on the new primary server without + any data loss. + </para> + + <para> + Because the slot synchronization logic copies asynchronously, it is + necessary to confirm that replication slots have been synced to the standby + server before the failover happens. Furthermore, to ensure a successful + failover, the standby server must not be lagging behind the subscriber. It + is highly recommended to use + <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link> + to prevent the subscriber from consuming changes faster than the hot standby. + To confirm that the standby server is indeed ready for failover, follow + these 2 steps: + </para> + + <procedure> + <step performance="required"> + <para> + Confirm that all the necessary logical replication slots have been synced to + the standby server. + </para> + <substeps> + <step performance="required"> + <para> + Firstly, on the subscriber node, use the following SQL to identify + which slots should be synced to the standby that we plan to promote. +<programlisting> +test_sub=# SELECT + array_agg(slotname) AS slots + FROM + (( + SELECT r.srsubid AS subid, CONCAT('pg_', srsubid, '_sync_', srrelid, '_', ctl.system_identifier) AS slotname + FROM pg_control_system() ctl, pg_subscription_rel r, pg_subscription s + WHERE r.srsubstate = 'f' AND s.oid = r.srsubid AND s.subfailover + ) UNION ( + SELECT s.oid AS subid, s.subslotname as slotname + FROM pg_subscription s + WHERE s.subfailover + )); + slots +------- + {sub1,sub2,sub3} +(1 row) +</programlisting></para> + </step> + <step performance="required"> + <para> + Next, check that the logical replication slots identified above exist on + the standby server and are ready for failover. +<programlisting> +test_standby=# SELECT slot_name, (synced AND NOT temporary AND conflict_reason IS NULL) AS failover_ready + FROM pg_replication_slots + WHERE slot_name IN ('sub1','sub2','sub3'); + slot_name | failover_ready +-------------+---------------- + sub1 | t + sub2 | t + sub3 | t +(3 rows) +</programlisting></para> + </step> + </substeps> + </step> + + <step performance="required"> + <para> + Confirm that the standby server is not lagging behind the subscribers. + This step can be skipped if + <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link> + has been correctly configured. If standby_slot_names is not configured + correctly, it is highly recommended to run this step after the primary + server is down, otherwise the results of the query may vary at different + points of time due to the ongoing replication on the logical subscribers + from the primary server. + </para> + <substeps> + <step performance="required"> + <para> + Firstly, on the subscriber node check the last replayed WAL. + This step needs to be run on the database(s) that includes the failover + enabled subscription(s), to find the last replayed WAL on each database. +<programlisting> +test_sub=# SELECT + MAX(remote_lsn) AS remote_lsn_on_subscriber + FROM + (( + SELECT (CASE WHEN r.srsubstate = 'f' THEN pg_replication_origin_progress(CONCAT('pg_', r.srsubid, '_', r.srrelid), false) + WHEN r.srsubstate IN ('s', 'r') THEN r.srsublsn END) AS remote_lsn + FROM pg_subscription_rel r, pg_subscription s + WHERE r.srsubstate IN ('f', 's', 'r') AND s.oid = r.srsubid AND s.subfailover + ) UNION ( + SELECT pg_replication_origin_progress(CONCAT('pg_', s.oid), false) AS remote_lsn + FROM pg_subscription s + WHERE s.subfailover + )); + remote_lsn_on_subscriber +-------------------------- + 0/3000388 +</programlisting></para> + </step> + <step performance="required"> + <para> + Next, on the standby server check that the last-received WAL location + is ahead of the replayed WAL location(s) on the subscriber identified + above. If the above SQL result was NULL, it means the subscriber has not + yet replayed any WAL, so the standby server must be ahead of the + subscriber, and this step can be skipped. +<programlisting> +test_standby=# SELECT pg_last_wal_receive_lsn() >= '0/3000388'::pg_lsn AS failover_ready; + failover_ready +---------------- + t +(1 row) +</programlisting></para> + </step> + </substeps> + </step> + </procedure> + + <para> + If the result (<literal>failover_ready</literal>) of both above steps is + true, existing subscriptions will be able to continue without data loss. + </para> + + </sect1> + <sect1 id="logical-replication-row-filter"> <title>Row Filters</title> -- 2.34.1 [application/octet-stream] v90-0001-Add-a-new-slotsync-worker.patch (62.5K, ../../CAJpy0uBdktD4Kj9_LiaHyVHhpk959TQuQVVTbbQAGni=CzusVw@mail.gmail.com/3-v90-0001-Add-a-new-slotsync-worker.patch) download | inline diff: From 5c23d45349186611a45c2543d47d7461766df83a Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Fri, 16 Feb 2024 12:44:58 +0530 Subject: [PATCH v90 1/3] Add a new slotsync worker By enabling slot synchronization, all the failover logical replication slots on the primary (assuming configurations are appropriate) are automatically created on the physical standbys and are synced periodically. Slot-sync worker on the standby server ping the primary server at regular intervals to get the necessary failover logical slots information and create/update the slots locally. The slots that no longer require synchronization are automatically dropped by the worker. The nap time of the worker is tuned according to the activity on the primary. The worker waits for a period of time before the next synchronization, with the duration varying based on whether any slots were updated during the last cycle. A new parameter sync_replication_slots enables or disables this new process. On promotion, the slot sync worker is shut down by the startup process to drop any temporary slots acquired by the slot sync worker and to prevent the worker from keep trying to fetch the failover slots. --- doc/src/sgml/config.sgml | 18 + doc/src/sgml/logicaldecoding.sgml | 5 +- src/backend/access/transam/xlogrecovery.c | 15 + src/backend/postmaster/postmaster.c | 88 +- .../libpqwalreceiver/libpqwalreceiver.c | 3 + src/backend/replication/logical/slotsync.c | 792 ++++++++++++++++-- src/backend/replication/slot.c | 14 + src/backend/replication/slotfuncs.c | 2 +- src/backend/replication/walsender.c | 2 +- src/backend/storage/ipc/ipci.c | 4 +- src/backend/storage/lmgr/proc.c | 13 +- src/backend/utils/activity/pgstat_io.c | 1 + .../utils/activity/wait_event_names.txt | 2 + src/backend/utils/init/miscinit.c | 9 +- src/backend/utils/init/postinit.c | 8 +- src/backend/utils/misc/guc_tables.c | 10 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/miscadmin.h | 1 + src/include/replication/slotsync.h | 24 +- .../t/040_standby_failover_slots_sync.pl | 113 ++- src/tools/pgindent/typedefs.list | 2 +- 21 files changed, 1027 insertions(+), 100 deletions(-) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index ffd711b7f2..ff184003fe 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4943,6 +4943,24 @@ ANY <replaceable class="parameter">num_sync</replaceable> ( <replaceable class=" </listitem> </varlistentry> + <varlistentry id="guc-sync-replication-slots" xreflabel="sync_replication_slots"> + <term><varname>sync_replication_slots</varname> (<type>boolean</type>) + <indexterm> + <primary><varname>sync_replication_slots</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + It enables a physical standby to synchronize logical failover slots + from the primary server so that logical subscribers can resume + replication from the new primary server after failover. + </para> + <para> + It is disabled by default. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server command line. + </para> + </listitem> + </varlistentry> </variablelist> </sect2> diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index eceaaaa273..930c0fa8a6 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -373,7 +373,10 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU <command>CREATE SUBSCRIPTION</command> during slot creation, and then calling <link linkend="pg-sync-replication-slots"> <function>pg_sync_replication_slots</function></link> - on the standby. For the synchronization to work, it is mandatory to + on the standby. By setting <link linkend="guc-sync-replication-slots"> + <varname>sync_replication_slots</varname></link> + on the standby, the failover slots can be synchronized periodically in + the slotsync worker. For the synchronization to work, it is mandatory to have a physical replication slot between the primary and the standby aka <link linkend="guc-primary-slot-name"><varname>primary_slot_name</varname></link> should be configured on the standby, and diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index 0bb472da27..d73a49b3e8 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -49,6 +49,7 @@ #include "postmaster/bgwriter.h" #include "postmaster/startup.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/walreceiver.h" #include "storage/fd.h" #include "storage/ipc.h" @@ -1467,6 +1468,20 @@ FinishWalRecovery(void) */ XLogShutdownWalRcv(); + /* + * Shutdown the slot sync worker to drop any temporary slots acquired by + * it and to prevent it from keep trying to fetch the failover slots. + * + * We do not update the 'synced' column from true to false here, as any + * failed update could leave 'synced' column false for some slots. This + * could cause issues during slot sync after restarting the server as a + * standby. While updating the 'synced' column after switching to the new + * timeline is an option, it does not simplify the handling for the + * 'synced' column. Therefore, we retain the 'synced' column as true after + * promotion as it may provide useful information about the slot origin. + */ + ShutDownSlotSync(); + /* * We are now done reading the xlog from stream. Turn off streaming * recovery to force fetching the files (which would be required at end of diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index df945a5ac4..3080d4aa53 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -115,6 +115,7 @@ #include "postmaster/syslogger.h" #include "postmaster/walsummarizer.h" #include "replication/logicallauncher.h" +#include "replication/slotsync.h" #include "replication/walsender.h" #include "storage/fd.h" #include "storage/ipc.h" @@ -167,11 +168,11 @@ * they will never become live backends. dead_end children are not assigned a * PMChildSlot. dead_end children have bkend_type NORMAL. * - * "Special" children such as the startup, bgwriter and autovacuum launcher - * tasks are not in this list. They are tracked via StartupPID and other - * pid_t variables below. (Thus, there can't be more than one of any given - * "special" child process type. We use BackendList entries for any child - * process there can be more than one of.) + * "Special" children such as the startup, bgwriter, autovacuum launcher, and + * slot sync worker tasks are not in this list. They are tracked via StartupPID + * and other pid_t variables below. (Thus, there can't be more than one of any + * given "special" child process type. We use BackendList entries for any + * child process there can be more than one of.) */ typedef struct bkend { @@ -254,7 +255,8 @@ static pid_t StartupPID = 0, WalSummarizerPID = 0, AutoVacPID = 0, PgArchPID = 0, - SysLoggerPID = 0; + SysLoggerPID = 0, + SlotSyncWorkerPID = 0; /* Startup process's status */ typedef enum @@ -458,6 +460,17 @@ static void InitPostmasterDeathWatchHandle(void); (pmState == PM_RECOVERY || pmState == PM_HOT_STANDBY))) && \ PgArchCanRestart()) +/* + * Slot sync worker allowed to start up? + * + * If we are on a hot standby, slot sync parameter is enabled, and it is + * the first time of worker's launch, or enough time has passed since the + * worker was launched last, then it is allowed to be started. + */ +#define SlotSyncWorkerAllowed() \ + (sync_replication_slots && pmState == PM_HOT_STANDBY && \ + SlotSyncWorkerCanRestart()) + #ifdef EXEC_BACKEND #ifdef WIN32 @@ -1822,6 +1835,10 @@ ServerLoop(void) if (PgArchPID == 0 && PgArchStartupAllowed()) PgArchPID = StartChildProcess(ArchiverProcess); + /* If we need to start a slot sync worker, try to do that now */ + if (SlotSyncWorkerPID == 0 && SlotSyncWorkerAllowed()) + SlotSyncWorkerPID = StartSlotSyncWorker(); + /* If we need to signal the autovacuum launcher, do so now */ if (avlauncher_needs_signal) { @@ -2661,6 +2678,8 @@ process_pm_reload_request(void) signal_child(PgArchPID, SIGHUP); if (SysLoggerPID != 0) signal_child(SysLoggerPID, SIGHUP); + if (SlotSyncWorkerPID != 0) + signal_child(SlotSyncWorkerPID, SIGHUP); /* Reload authentication config files too */ if (!load_hba()) @@ -3011,6 +3030,9 @@ process_pm_child_exit(void) if (PgArchStartupAllowed() && PgArchPID == 0) PgArchPID = StartChildProcess(ArchiverProcess); + if (SlotSyncWorkerAllowed() && SlotSyncWorkerPID == 0) + SlotSyncWorkerPID = StartSlotSyncWorker(); + /* workers may be scheduled to start now */ maybe_start_bgworkers(); @@ -3180,6 +3202,22 @@ process_pm_child_exit(void) continue; } + /* + * Was it the slot sync worker? Normal exit or FATAL exit can be + * ignored (FATAL can be caused by libpqwalreceiver on receiving + * shutdown request by the startup process during promotion); we'll + * start a new one at the next iteration of the postmaster's main + * loop, if necessary. Any other exit condition is treated as a crash. + */ + if (pid == SlotSyncWorkerPID) + { + SlotSyncWorkerPID = 0; + if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus)) + HandleChildCrash(pid, exitstatus, + _("slot sync worker process")); + continue; + } + /* Was it one of our background workers? */ if (CleanupBackgroundWorker(pid, exitstatus)) { @@ -3384,7 +3422,7 @@ CleanupBackend(int pid, /* * HandleChildCrash -- cleanup after failed backend, bgwriter, checkpointer, - * walwriter, autovacuum, archiver or background worker. + * walwriter, autovacuum, archiver, slot sync worker, or background worker. * * The objectives here are to clean up our local state about the child * process, and to signal all other remaining children to quickdie. @@ -3546,6 +3584,12 @@ HandleChildCrash(int pid, int exitstatus, const char *procname) else if (PgArchPID != 0 && take_action) sigquit_child(PgArchPID); + /* Take care of the slot sync worker too */ + if (pid == SlotSyncWorkerPID) + SlotSyncWorkerPID = 0; + else if (SlotSyncWorkerPID != 0 && take_action) + sigquit_child(SlotSyncWorkerPID); + /* We do NOT restart the syslogger */ if (Shutdown != ImmediateShutdown) @@ -3686,6 +3730,8 @@ PostmasterStateMachine(void) signal_child(WalReceiverPID, SIGTERM); if (WalSummarizerPID != 0) signal_child(WalSummarizerPID, SIGTERM); + if (SlotSyncWorkerPID != 0) + signal_child(SlotSyncWorkerPID, SIGTERM); /* checkpointer, archiver, stats, and syslogger may continue for now */ /* Now transition to PM_WAIT_BACKENDS state to wait for them to die */ @@ -3701,13 +3747,13 @@ PostmasterStateMachine(void) /* * PM_WAIT_BACKENDS state ends when we have no regular backends * (including autovac workers), no bgworkers (including unconnected - * ones), and no walwriter, autovac launcher or bgwriter. If we are - * doing crash recovery or an immediate shutdown then we expect the - * checkpointer to exit as well, otherwise not. The stats and - * syslogger processes are disregarded since they are not connected to - * shared memory; we also disregard dead_end children here. Walsenders - * and archiver are also disregarded, they will be terminated later - * after writing the checkpoint record. + * ones), and no walwriter, autovac launcher, bgwriter or slot sync + * worker. If we are doing crash recovery or an immediate shutdown + * then we expect the checkpointer to exit as well, otherwise not. The + * stats and syslogger processes are disregarded since they are not + * connected to shared memory; we also disregard dead_end children + * here. Walsenders and archiver are also disregarded, they will be + * terminated later after writing the checkpoint record. */ if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 && StartupPID == 0 && @@ -3717,7 +3763,8 @@ PostmasterStateMachine(void) (CheckpointerPID == 0 || (!FatalError && Shutdown < ImmediateShutdown)) && WalWriterPID == 0 && - AutoVacPID == 0) + AutoVacPID == 0 && + SlotSyncWorkerPID == 0) { if (Shutdown >= ImmediateShutdown || FatalError) { @@ -3815,6 +3862,7 @@ PostmasterStateMachine(void) Assert(CheckpointerPID == 0); Assert(WalWriterPID == 0); Assert(AutoVacPID == 0); + Assert(SlotSyncWorkerPID == 0); /* syslogger is not considered here */ pmState = PM_NO_CHILDREN; } @@ -4038,6 +4086,8 @@ TerminateChildren(int signal) signal_child(AutoVacPID, signal); if (PgArchPID != 0) signal_child(PgArchPID, signal); + if (SlotSyncWorkerPID != 0) + signal_child(SlotSyncWorkerPID, signal); } /* @@ -4850,6 +4900,7 @@ SubPostmasterMain(int argc, char *argv[]) */ if (strcmp(argv[1], "--forkbackend") == 0 || strcmp(argv[1], "--forkavlauncher") == 0 || + strcmp(argv[1], "--forkssworker") == 0 || strcmp(argv[1], "--forkavworker") == 0 || strcmp(argv[1], "--forkaux") == 0 || strcmp(argv[1], "--forkbgworker") == 0) @@ -4953,6 +5004,13 @@ SubPostmasterMain(int argc, char *argv[]) AutoVacWorkerMain(argc - 2, argv + 2); /* does not return */ } + if (strcmp(argv[1], "--forkssworker") == 0) + { + /* Restore basic shared memory pointers */ + InitShmemAccess(UsedShmemSegAddr); + + ReplSlotSyncWorkerMain(argc - 2, argv + 2); /* does not return */ + } if (strcmp(argv[1], "--forkbgworker") == 0) { /* do this as early as possible; in particular, before InitProcess() */ diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index 9270d7b855..04271ee703 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -6,6 +6,9 @@ * loaded as a dynamic module to avoid linking the main server binary with * libpq. * + * Apart from walreceiver, the libpq-specific routines are now being used by + * logical replication workers and slot synchronization. + * * Portions Copyright (c) 2010-2024, PostgreSQL Global Development Group * * diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 4cab7b7101..40ab87bce1 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -10,18 +10,25 @@ * * This file contains the code for slot synchronization on a physical standby * to fetch logical failover slots information from the primary server, create - * the slots on the standby and synchronize them. This is done by a call to SQL - * function pg_sync_replication_slots. + * the slots on the standby and synchronize them periodically. * - * If on physical standby, the WAL corresponding to the remote's restart_lsn - * is not available or the remote's catalog_xmin precedes the oldest xid for which - * it is guaranteed that rows wouldn't have been removed then we cannot create - * the local standby slot because that would mean moving the local slot + * Slot synchronization can be performed either automatically by enabling slot + * sync worker or manually by calling SQL function pg_sync_replication_slots(). + * + * If the WAL corresponding to the remote's restart_lsn is not available on the + * physical standby or the remote's catalog_xmin precedes the oldest xid for + * which it is guaranteed that rows wouldn't have been removed then we cannot + * create the local standby slot because that would mean moving the local slot * backward and decoding won't be possible via such a slot. In this case, the * slot will be marked as RS_TEMPORARY. Once the primary server catches up, * the slot will be marked as RS_PERSISTENT (which means sync-ready) after - * which we can call pg_sync_replication_slots() periodically to perform - * syncs. + * which slot sync worker can perform the sync periodically or user can call + * pg_sync_replication_slots() periodically to perform the syncs. + * + * The slot sync worker waits for some time before the next synchronization, + * with the duration varying based on whether any slots were updated during + * the last cycle. Refer to the comments above wait_for_slot_activity() for + * more details. * * Any standby synchronized slots will be dropped if they no longer need * to be synchronized. See comment atop drop_local_obsolete_slots() for more @@ -31,31 +38,80 @@ #include "postgres.h" +#include <time.h> + #include "access/xlog_internal.h" #include "access/xlogrecovery.h" #include "catalog/pg_database.h" #include "commands/dbcommands.h" +#include "libpq/pqsignal.h" +#include "pgstat.h" +#include "postmaster/fork_process.h" +#include "postmaster/interrupt.h" +#include "postmaster/postmaster.h" #include "replication/logical.h" #include "replication/slotsync.h" #include "storage/ipc.h" #include "storage/lmgr.h" +#include "storage/proc.h" #include "storage/procarray.h" +#include "tcop/tcopprot.h" #include "utils/builtins.h" #include "utils/pg_lsn.h" +#include "utils/ps_status.h" +#include "utils/timeout.h" -/* Struct for sharing information to control slot synchronization. */ -typedef struct SlotSyncCtxStruct +/* + * Struct for sharing information to control slot synchronization. + * + * Slot sync worker's pid is needed by the startup process in order to shut it + * down during promotion. Startup process shuts down the slot sync worker and + * also sets stopSignaled=true to handle the race condition when postmaster has + * not noticed the promotion yet and thus may end up restarting slot sync + * worker. If stopSignaled is set, the worker will exit in such a case. + * + * The 'syncing' flag is needed to prevent concurrent slot syncs to avoid slot + * overwrites. + * + * The last_start_time is needed by postmaster to start the slot sync worker + * once per SLOTSYNC_RESTART_INTERVAL_SEC. In cases where a immediate restart + * is expected (e.g., slot sync GUCs change), slot sync worker will reset + * last_start_time before exiting, so that postmaster can start the worker + * without waiting for SLOTSYNC_RESTART_INTERVAL_SEC. + */ +typedef struct SlotSyncWorkerCtxStruct { - /* prevents concurrent slot syncs to avoid slot overwrites */ + pid_t pid; + bool stopSignaled; bool syncing; + time_t last_start_time; slock_t mutex; -} SlotSyncCtxStruct; +} SlotSyncWorkerCtxStruct; + +SlotSyncWorkerCtxStruct *SlotSyncWorker = NULL; -SlotSyncCtxStruct *SlotSyncCtx = NULL; +/* GUC variable */ +bool sync_replication_slots = false; + +/* + * The sleep time (ms) between slot-sync cycles varies dynamically + * (within a MIN/MAX range) according to slot activity. See + * wait_for_slot_activity() for details. + */ +#define MIN_WORKER_NAPTIME_MS 200 +#define MAX_WORKER_NAPTIME_MS 30000 /* 30s */ + +static long sleep_ms = MIN_WORKER_NAPTIME_MS; + +/* The restart interval for slot sync work used by postmaster */ +#define SLOTSYNC_RESTART_INTERVAL_SEC 10 + +/* Flag to tell if we are in a slot sync worker process */ +static bool am_slotsync_worker = false; /* * Flag to tell if we are syncing replication slots. Unlike the 'syncing' flag - * in SlotSyncCtxStruct, this flag is true only if the current process is + * in SlotSyncWorkerCtxStruct, this flag is true only if the current process is * performing slot synchronization. */ static bool syncing_slots = false; @@ -79,6 +135,15 @@ typedef struct RemoteSlot ReplicationSlotInvalidationCause invalidated; } RemoteSlot; +static void ProcessSlotSyncInterrupts(WalReceiverConn *wrconn, bool restart); + +#ifdef EXEC_BACKEND +static pid_t slotsyncworker_forkexec(void); +#endif +NON_EXEC_STATIC void ReplSlotSyncWorkerMain(int argc, char *argv[]) pg_attribute_noreturn(); + +static void slotsync_failure_callback(int code, Datum arg); + /* * If necessary, update the local synced slot's metadata based on the data * from the remote slot. @@ -343,8 +408,11 @@ reserve_wal_for_local_slot(XLogRecPtr restart_lsn) * If the remote restart_lsn and catalog_xmin have caught up with the * local ones, then update the LSNs and persist the local synced slot for * future synchronization; otherwise, do nothing. + * + * Return true if the slot is marked as RS_PERSISTENT (sync-ready), otherwise + * false. */ -static void +static bool update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) { ReplicationSlot *slot = MyReplicationSlot; @@ -376,7 +444,7 @@ update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) LSN_FORMAT_ARGS(slot->data.restart_lsn), slot->data.catalog_xmin)); - return; + return false; } /* First time slot update, the function must return true */ @@ -388,6 +456,8 @@ update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) ereport(LOG, errmsg("newly created slot \"%s\" is sync-ready now", remote_slot->name)); + + return true; } /* @@ -400,12 +470,15 @@ update_and_persist_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) * the remote_slot catches up with locally reserved position and local slot is * updated. The slot is then persisted and is considered as sync-ready for * periodic syncs. + * + * Returns TRUE if the local slot is updated. */ -static void +static bool synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) { ReplicationSlot *slot; XLogRecPtr latestFlushPtr; + bool slot_updated = false; /* * Make sure that concerned WAL is received and flushed before syncing @@ -413,13 +486,17 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) */ latestFlushPtr = GetStandbyFlushRecPtr(NULL); if (remote_slot->confirmed_lsn > latestFlushPtr) - elog(ERROR, + { + elog(am_slotsync_worker ? LOG : ERROR, "skipping slot synchronization as the received slot sync" " LSN %X/%X for slot \"%s\" is ahead of the standby position %X/%X", LSN_FORMAT_ARGS(remote_slot->confirmed_lsn), remote_slot->name, LSN_FORMAT_ARGS(latestFlushPtr)); + return false; + } + /* Search for the named slot */ if ((slot = SearchNamedReplicationSlot(remote_slot->name, true))) { @@ -466,19 +543,22 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) /* Make sure the invalidated state persists across server restart */ ReplicationSlotMarkDirty(); ReplicationSlotSave(); + + slot_updated = true; } /* Skip the sync of an invalidated slot */ if (slot->data.invalidated != RS_INVAL_NONE) { ReplicationSlotRelease(); - return; + return slot_updated; } /* Slot not ready yet, let's attempt to make it sync-ready now. */ if (slot->data.persistency == RS_TEMPORARY) { - update_and_persist_local_synced_slot(remote_slot, remote_dbid); + slot_updated = update_and_persist_local_synced_slot(remote_slot, + remote_dbid); } /* Slot ready for sync, so sync it. */ @@ -501,6 +581,8 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) { ReplicationSlotMarkDirty(); ReplicationSlotSave(); + + slot_updated = true; } } } @@ -512,7 +594,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) /* Skip creating the local slot if remote_slot is invalidated already */ if (remote_slot->invalidated != RS_INVAL_NONE) - return; + return false; /* * We create temporary slots instead of ephemeral slots here because @@ -549,9 +631,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) LWLockRelease(ProcArrayLock); update_and_persist_local_synced_slot(remote_slot, remote_dbid); + + slot_updated = true; } ReplicationSlotRelease(); + + return slot_updated; } /* @@ -559,8 +645,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) * * Gets the failover logical slots info from the primary server and updates * the slots locally. Creates the slots if not present on the standby. + * + * Returns TRUE if any of the slots gets updated in this sync-cycle. */ -static void +static bool synchronize_slots(WalReceiverConn *wrconn) { #define SLOTSYNC_COLUMN_COUNT 9 @@ -571,21 +659,30 @@ synchronize_slots(WalReceiverConn *wrconn) TupleTableSlot *tupslot; StringInfoData s; List *remote_slot_list = NIL; + bool some_slot_updated = false; + bool started_tx = false; - SpinLockAcquire(&SlotSyncCtx->mutex); - if (SlotSyncCtx->syncing) + SpinLockAcquire(&SlotSyncWorker->mutex); + if (SlotSyncWorker->syncing) { - SpinLockRelease(&SlotSyncCtx->mutex); + SpinLockRelease(&SlotSyncWorker->mutex); ereport(ERROR, errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), errmsg("cannot synchronize replication slots concurrently")); } - SlotSyncCtx->syncing = true; - SpinLockRelease(&SlotSyncCtx->mutex); + SlotSyncWorker->syncing = true; + SpinLockRelease(&SlotSyncWorker->mutex); syncing_slots = true; + /* The syscache access in walrcv_exec() needs a transaction env. */ + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + initStringInfo(&s); /* Construct query to fetch slots with failover enabled. */ @@ -694,7 +791,7 @@ synchronize_slots(WalReceiverConn *wrconn) */ LockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); - synchronize_one_slot(remote_slot, remote_dbid); + some_slot_updated |= synchronize_one_slot(remote_slot, remote_dbid); UnlockSharedObject(DatabaseRelationId, remote_dbid, 0, AccessShareLock); } @@ -704,21 +801,26 @@ synchronize_slots(WalReceiverConn *wrconn) walrcv_clear_result(res); - SpinLockAcquire(&SlotSyncCtx->mutex); - SlotSyncCtx->syncing = false; - SpinLockRelease(&SlotSyncCtx->mutex); + if (started_tx) + CommitTransactionCommand(); + + SpinLockAcquire(&SlotSyncWorker->mutex); + SlotSyncWorker->syncing = false; + SpinLockRelease(&SlotSyncWorker->mutex); syncing_slots = false; + + return some_slot_updated; } /* * Checks the remote server info. * - * We ensure that the 'primary_slot_name' exists on the remote server and the - * remote server is not a standby node. + * Check whether we are a cascading standby. For non-cascading standbys, it + * also ensures that the 'primary_slot_name' exists on the remote server. */ -static void -validate_remote_info(WalReceiverConn *wrconn) +static bool +validate_remote_info(WalReceiverConn *wrconn, int elevel) { #define PRIMARY_INFO_OUTPUT_COL_COUNT 2 WalRcvExecResult *res; @@ -727,7 +829,9 @@ validate_remote_info(WalReceiverConn *wrconn) bool isnull; TupleTableSlot *tupslot; bool remote_in_recovery; - bool primary_slot_valid; + bool primary_slot_valid = false; /* initialize to keep compiler + * quiet */ + bool started_tx = false; initStringInfo(&cmd); appendStringInfo(&cmd, @@ -736,6 +840,13 @@ validate_remote_info(WalReceiverConn *wrconn) " WHERE slot_type='physical' AND slot_name=%s", quote_literal_cstr(PrimarySlotName)); + /* The syscache access in walrcv_exec() needs a transaction env. */ + if (!IsTransactionState()) + { + StartTransactionCommand(); + started_tx = true; + } + res = walrcv_exec(wrconn, cmd.data, PRIMARY_INFO_OUTPUT_COL_COUNT, slotRow); pfree(cmd.data); @@ -754,31 +865,71 @@ validate_remote_info(WalReceiverConn *wrconn) Assert(!isnull); if (remote_in_recovery) - ereport(ERROR, + { + /* + * If we are a cascading standby, no need to check further for + * 'primary_slot_name'. + */ + ereport(elevel, errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot synchronize replication slots from a standby server")); + } + else + { + primary_slot_valid = DatumGetBool(slot_getattr(tupslot, 2, &isnull)); + Assert(!isnull); - primary_slot_valid = DatumGetBool(slot_getattr(tupslot, 2, &isnull)); - Assert(!isnull); - - if (!primary_slot_valid) - ereport(ERROR, - errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("bad configuration for slot synchronization"), - /* translator: second %s is a GUC variable name */ - errdetail("The replication slot \"%s\" specified by \"%s\" does not exist on the primary server.", - PrimarySlotName, "primary_slot_name")); + if (!primary_slot_valid) + ereport(elevel, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("bad configuration for slot synchronization"), + /* translator: second %s is a GUC variable name */ + errdetail("The replication slot \"%s\" specified by \"%s\" does not exist on the primary server.", + PrimarySlotName, "primary_slot_name")); + } ExecClearTuple(tupslot); walrcv_clear_result(res); + + if (started_tx) + CommitTransactionCommand(); + + return (!remote_in_recovery && primary_slot_valid); } /* - * Check all necessary GUCs for slot synchronization are set - * appropriately, otherwise, raise ERROR. + * Check that we are not a cascading standby and 'primary_slot_name' exists + * on the remote server. If not, sleep for MAX_WORKER_NAPTIME_MS and check + * again. The idea is to become no-op until we get valid remote info. */ -void -ValidateSlotSyncParams(void) +static void +ensure_valid_remote_info(WalReceiverConn *wrconn) +{ + int rc; + + for (;;) + { + if (validate_remote_info(wrconn, LOG)) + break; + + rc = WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + MAX_WORKER_NAPTIME_MS, + WAIT_EVENT_REPL_SLOTSYNC_MAIN); + + if (rc & WL_LATCH_SET) + ResetLatch(MyLatch); + + ProcessSlotSyncInterrupts(NULL, true); + } +} + +/* + * Return true if all necessary GUCs for slot synchronization are set + * appropriately, otherwise, return false. + */ +bool +ValidateSlotSyncParams(int elevel) { char *dbname; @@ -789,11 +940,14 @@ ValidateSlotSyncParams(void) * be invalidated. */ if (PrimarySlotName == NULL || *PrimarySlotName == '\0') - ereport(ERROR, + { + ereport(elevel, /* translator: %s is a GUC variable name */ errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("\"%s\" must be defined.", "primary_slot_name")); + return false; + } /* * hot_standby_feedback must be enabled to cooperate with the physical @@ -801,29 +955,38 @@ ValidateSlotSyncParams(void) * catalog_xmin values on the standby. */ if (!hot_standby_feedback) - ereport(ERROR, + { + ereport(elevel, /* translator: %s is a GUC variable name */ errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("\"%s\" must be enabled.", "hot_standby_feedback")); + return false; + } /* Logical slot sync/creation requires wal_level >= logical. */ if (wal_level < WAL_LEVEL_LOGICAL) - ereport(ERROR, + { + ereport(elevel, errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("\"wal_level\" must be >= logical.")); + return false; + } /* * The primary_conninfo is required to make connection to primary for * getting slots information. */ if (PrimaryConnInfo == NULL || *PrimaryConnInfo == '\0') - ereport(ERROR, + { + ereport(elevel, /* translator: %s is a GUC variable name */ errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("\"%s\" must be defined.", "primary_conninfo")); + return false; + } /* * The slot synchronization needs a database connection for walrcv_exec to @@ -831,7 +994,8 @@ ValidateSlotSyncParams(void) */ dbname = walrcv_get_dbname_from_conninfo(PrimaryConnInfo); if (dbname == NULL) - ereport(ERROR, + { + ereport(elevel, /* * translator: 'dbname' is a specific option; %s is a GUC variable @@ -840,10 +1004,491 @@ ValidateSlotSyncParams(void) errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("bad configuration for slot synchronization"), errhint("'dbname' must be specified in \"%s\".", "primary_conninfo")); + return false; + } + + return true; +} + +/* + * Check that all necessary GUCs for slot synchronization are set + * appropriately. If not, sleep for MAX_WORKER_NAPTIME_MS and check again. + * The idea is to become no-op until we get valid GUCs values. + */ +static void +ensure_valid_slotsync_params(void) +{ + int rc; + + /* Sanity check. */ + Assert(sync_replication_slots); + + for (;;) + { + if (ValidateSlotSyncParams(LOG)) + { + ereport(LOG, errmsg("parameters validation done for slot synchronization")); + break; + } + + ereport(LOG, errmsg("skipping slot synchronization")); + + rc = WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + MAX_WORKER_NAPTIME_MS, + WAIT_EVENT_REPL_SLOTSYNC_MAIN); + + if (rc & WL_LATCH_SET) + ResetLatch(MyLatch); + + /* + * Process the interrupts. Pass 'restart' as false as we do not want + * the worker to be restarted on GUC change. + */ + ProcessSlotSyncInterrupts(NULL, false); + } +} + +/* + * Re-read the config file. + * + * If any of the slot sync GUCs have changed, exit the worker and + * let it get restarted by the postmaster. The worker to be exited for + * restart purpose only if the caller passed restart as true. + */ +static void +slotsync_reread_config(bool restart) +{ + char *old_primary_conninfo = pstrdup(PrimaryConnInfo); + char *old_primary_slotname = pstrdup(PrimarySlotName); + bool old_sync_replication_slots = sync_replication_slots; + bool old_hot_standby_feedback = hot_standby_feedback; + bool conninfo_changed; + bool primary_slotname_changed; + + Assert(sync_replication_slots); + + ConfigReloadPending = false; + ProcessConfigFile(PGC_SIGHUP); + + conninfo_changed = strcmp(old_primary_conninfo, PrimaryConnInfo) != 0; + primary_slotname_changed = strcmp(old_primary_slotname, PrimarySlotName) != 0; + pfree(old_primary_conninfo); + pfree(old_primary_slotname); + + if (old_sync_replication_slots != sync_replication_slots) + { + ereport(LOG, + /* translator: %s is a GUC variable name */ + errmsg("slot sync worker will shutdown because %s is disabled", "sync_replication_slots")); + proc_exit(0); + } + + /* The caller instructed to skip restart */ + if (!restart) + return; + + if (conninfo_changed || + primary_slotname_changed || + (old_hot_standby_feedback != hot_standby_feedback)) + { + ereport(LOG, + errmsg("slot sync worker will restart because of a parameter change")); + + /* + * Reset the last-start time for this worker so that the postmaster + * can restart it without waiting for SLOTSYNC_RESTART_INTERVAL_SEC. + */ + SlotSyncWorker->last_start_time = 0; + + proc_exit(0); + } + +} + +/* + * Interrupt handler for main loop of slot sync worker. + */ +static void +ProcessSlotSyncInterrupts(WalReceiverConn *wrconn, bool restart) +{ + CHECK_FOR_INTERRUPTS(); + + if (ShutdownRequestPending) + { + ereport(LOG, + errmsg("replication slot sync worker is shutting down on receiving SIGINT")); + + proc_exit(0); + } + + if (ConfigReloadPending) + slotsync_reread_config(restart); } /* - * Is current process syncing replication slots ? + * Cleanup function for slotsync worker. + * + * Called on slotsync worker exit. + */ +static void +slotsync_worker_onexit(int code, Datum arg) +{ + SpinLockAcquire(&SlotSyncWorker->mutex); + SlotSyncWorker->pid = InvalidPid; + SpinLockRelease(&SlotSyncWorker->mutex); +} + +/* + * Sleep for long enough that we believe it's likely that the slots on primary + * get updated. + * + * If there is no slot activity the wait time between sync-cycles will double + * (to a maximum of 30s). If there is some slot activity the wait time between + * sync-cycles is reset to the minimum (200ms). + */ +static void +wait_for_slot_activity(bool some_slot_updated) +{ + int rc; + + if (!some_slot_updated) + { + /* + * No slots were updated, so double the sleep time, but not beyond the + * maximum allowable value. + */ + sleep_ms = Min(sleep_ms * 2, MAX_WORKER_NAPTIME_MS); + } + else + { + /* + * Some slots were updated since the last sleep, so reset the sleep + * time. + */ + sleep_ms = MIN_WORKER_NAPTIME_MS; + } + + rc = WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + sleep_ms, + WAIT_EVENT_REPL_SLOTSYNC_MAIN); + + if (rc & WL_LATCH_SET) + ResetLatch(MyLatch); +} + +/* + * The main loop of our worker process. + * + * It connects to the primary server, fetches logical failover slots + * information periodically in order to create and sync the slots. + */ +NON_EXEC_STATIC void +ReplSlotSyncWorkerMain(int argc, char *argv[]) +{ + WalReceiverConn *wrconn = NULL; + char *dbname; + char *err; + sigjmp_buf local_sigjmp_buf; + StringInfoData app_name; + + am_slotsync_worker = true; + + MyBackendType = B_SLOTSYNC_WORKER; + + init_ps_display(NULL); + + SetProcessingMode(InitProcessing); + + /* + * Create a per-backend PGPROC struct in shared memory. We must do this + * before we access any shared memory. + */ + InitProcess(); + + /* + * Early initialization. + */ + BaseInit(); + + Assert(SlotSyncWorker != NULL); + + SpinLockAcquire(&SlotSyncWorker->mutex); + Assert(SlotSyncWorker->pid == InvalidPid); + + /* + * Startup process signaled the slot sync worker to stop, so if meanwhile + * postmaster ended up starting the worker again, exit. + */ + if (SlotSyncWorker->stopSignaled) + { + SpinLockRelease(&SlotSyncWorker->mutex); + proc_exit(0); + } + + /* Advertise our PID so that the startup process can kill us on promotion */ + SlotSyncWorker->pid = MyProcPid; + SpinLockRelease(&SlotSyncWorker->mutex); + + ereport(LOG, errmsg("replication slot sync worker started")); + + on_shmem_exit(slotsync_worker_onexit, (Datum) 0); + + /* Setup signal handling */ + pqsignal(SIGHUP, SignalHandlerForConfigReload); + pqsignal(SIGINT, SignalHandlerForShutdownRequest); + pqsignal(SIGTERM, die); + pqsignal(SIGFPE, FloatExceptionHandler); + pqsignal(SIGUSR1, procsignal_sigusr1_handler); + pqsignal(SIGUSR2, SIG_IGN); + pqsignal(SIGPIPE, SIG_IGN); + pqsignal(SIGCHLD, SIG_DFL); + + /* + * Establishes SIGALRM handler and initialize timeout module. It is needed + * by InitPostgres to register different timeouts. + */ + InitializeTimeouts(); + + /* Load the libpq-specific functions */ + load_file("libpqwalreceiver", false); + + /* + * If an exception is encountered, processing resumes here. + * + * We just need to clean up, report the error, and go away. + * + * If we do not have this handling here, then since this worker process + * operates at the bottom of the exception stack, ERRORs turn into FATALs. + * Therefore, we create our own exception handler to catch ERRORs. + */ + if (sigsetjmp(local_sigjmp_buf, 1) != 0) + { + /* since not using PG_TRY, must reset error stack by hand */ + error_context_stack = NULL; + + /* Prevents interrupts while cleaning up */ + HOLD_INTERRUPTS(); + + /* Report the error to the server log */ + EmitErrorReport(); + + /* + * We can now go away. Note that because we called InitProcess, a + * callback was registered to do ProcKill, which will clean up + * necessary state. + */ + proc_exit(0); + } + + /* We can now handle ereport(ERROR) */ + PG_exception_stack = &local_sigjmp_buf; + + /* + * Unblock signals (they were blocked when the postmaster forked us) + */ + sigprocmask(SIG_SETMASK, &UnBlockSig, NULL); + + ensure_valid_slotsync_params(); + + dbname = walrcv_get_dbname_from_conninfo(PrimaryConnInfo); + + /* + * Connect to the database specified by user in primary_conninfo. We need + * a database connection for walrcv_exec to work. Please see comments atop + * libpqrcv_exec. + */ + InitPostgres(dbname, InvalidOid, NULL, InvalidOid, 0, NULL); + + SetProcessingMode(NormalProcessing); + + initStringInfo(&app_name); + if (cluster_name[0]) + appendStringInfo(&app_name, "%s_%s", cluster_name, "slotsyncworker"); + else + appendStringInfo(&app_name, "%s", "slotsyncworker"); + + /* + * Establish the connection to the primary server for slots + * synchronization. + */ + wrconn = walrcv_connect(PrimaryConnInfo, false, false, false, + app_name.data, + &err); + pfree(app_name.data); + + if (!wrconn) + ereport(ERROR, + errcode(ERRCODE_CONNECTION_FAILURE), + errmsg("could not connect to the primary server: %s", err)); + + before_shmem_exit(slotsync_failure_callback, PointerGetDatum(wrconn)); + + /* + * Using the specified primary server connection, ensure that we are not a + * cascading standby and slot configured in 'primary_slot_name' exists on + * the primary server. + */ + ensure_valid_remote_info(wrconn); + + /* Main wait loop */ + for (;;) + { + bool some_slot_updated = false; + + ProcessSlotSyncInterrupts(wrconn, true); + + some_slot_updated = synchronize_slots(wrconn); + + wait_for_slot_activity(some_slot_updated); + } + + /* + * The slot sync worker can not get here because it will only stop when it + * receives a SIGINT from the startup process, or when there is an error. + */ + Assert(false); +} + +/* + * Main entry point for slot sync worker process, to be called from the + * postmaster. + */ +int +StartSlotSyncWorker(void) +{ + pid_t pid; + +#ifdef EXEC_BACKEND + switch ((pid = slotsyncworker_forkexec())) + { +#else + switch ((pid = fork_process())) + { + case 0: + /* in postmaster child ... */ + InitPostmasterChild(); + + /* Close the postmaster's sockets */ + ClosePostmasterPorts(false); + + ReplSlotSyncWorkerMain(0, NULL); + break; +#endif + case -1: + ereport(LOG, + (errmsg("could not fork slot sync worker process: %m"))); + return 0; + + default: + return (int) pid; + } + + /* shouldn't get here */ + return 0; +} + +#ifdef EXEC_BACKEND +/* + * The forkexec routine for the slot sync worker process. + * + * Format up the arglist, then fork and exec. + */ +static pid_t +slotsyncworker_forkexec(void) +{ + char *av[10]; + int ac = 0; + + av[ac++] = "postgres"; + av[ac++] = "--forkssworker"; + av[ac++] = NULL; /* filled in by postmaster_forkexec */ + av[ac] = NULL; + + Assert(ac < lengthof(av)); + + return postmaster_forkexec(ac, av); +} +#endif + +/* + * Shut down the slot sync worker. + */ +void +ShutDownSlotSync(void) +{ + SpinLockAcquire(&SlotSyncWorker->mutex); + + SlotSyncWorker->stopSignaled = true; + + if (SlotSyncWorker->pid == InvalidPid) + { + SpinLockRelease(&SlotSyncWorker->mutex); + return; + } + SpinLockRelease(&SlotSyncWorker->mutex); + + kill(SlotSyncWorker->pid, SIGINT); + + /* Wait for it to die */ + for (;;) + { + int rc; + + /* Wait a bit, we don't expect to have to wait long */ + rc = WaitLatch(MyLatch, + WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH, + 10L, WAIT_EVENT_REPL_SLOTSYNC_SHUTDOWN); + + if (rc & WL_LATCH_SET) + { + ResetLatch(MyLatch); + CHECK_FOR_INTERRUPTS(); + } + + SpinLockAcquire(&SlotSyncWorker->mutex); + + /* Is it gone? */ + if (SlotSyncWorker->pid == InvalidPid) + break; + + SpinLockRelease(&SlotSyncWorker->mutex); + } + + SpinLockRelease(&SlotSyncWorker->mutex); +} + +/* + * SlotSyncWorkerCanRestart + * + * Returns true if enough time has passed (SLOTSYNC_RESTART_INTERVAL_SEC) + * since it was launched last. Otherwise returns false. + * + * This is a safety valve to protect against continuous respawn attempts if the + * worker is dying immediately at launch. Note that since we will retry to + * launch the worker from the postmaster main loop, we will get another + * chance later. + */ +bool +SlotSyncWorkerCanRestart(void) +{ + time_t curtime = time(NULL); + + /* Return false if too soon since last start. */ + if ((unsigned int) (curtime - SlotSyncWorker->last_start_time) < + (unsigned int) SLOTSYNC_RESTART_INTERVAL_SEC) + return false; + + SlotSyncWorker->last_start_time = curtime; + + return true; +} + +/* + * Is current process syncing replication slots? + * + * Could be either backend executing SQL function or slot sync worker. */ bool IsSyncingReplicationSlots(void) @@ -851,30 +1496,41 @@ IsSyncingReplicationSlots(void) return syncing_slots; } +/* + * Is current process a slot sync worker? + */ +bool +IsLogicalSlotSyncWorker(void) +{ + return am_slotsync_worker; +} + /* * Amount of shared memory required for slot synchronization. */ Size -SlotSyncShmemSize(void) +SlotSyncWorkerShmemSize(void) { - return sizeof(SlotSyncCtxStruct); + return sizeof(SlotSyncWorkerCtxStruct); } /* * Allocate and initialize the shared memory of slot synchronization. */ void -SlotSyncShmemInit(void) +SlotSyncWorkerShmemInit(void) { + Size size = SlotSyncWorkerShmemSize(); bool found; - SlotSyncCtx = (SlotSyncCtxStruct *) - ShmemInitStruct("Slot Sync Data", SlotSyncShmemSize(), &found); + SlotSyncWorker = (SlotSyncWorkerCtxStruct *) + ShmemInitStruct("Slot Sync Worker Data", size, &found); if (!found) { - SlotSyncCtx->syncing = false; - SpinLockInit(&SlotSyncCtx->mutex); + memset(SlotSyncWorker, 0, size); + SlotSyncWorker->pid = InvalidPid; + SpinLockInit(&SlotSyncWorker->mutex); } } @@ -893,9 +1549,9 @@ slotsync_failure_callback(int code, Datum arg) * without resetting the flag. So, we need to clean up shared memory * and reset the flag here. */ - SpinLockAcquire(&SlotSyncCtx->mutex); - SlotSyncCtx->syncing = false; - SpinLockRelease(&SlotSyncCtx->mutex); + SpinLockAcquire(&SlotSyncWorker->mutex); + SlotSyncWorker->syncing = false; + SpinLockRelease(&SlotSyncWorker->mutex); syncing_slots = false; } @@ -912,7 +1568,7 @@ SyncReplicationSlots(WalReceiverConn *wrconn) { PG_ENSURE_ERROR_CLEANUP(slotsync_failure_callback, PointerGetDatum(wrconn)); { - validate_remote_info(wrconn); + validate_remote_info(wrconn, ERROR); synchronize_slots(wrconn); } diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index 2180a38063..dd4ea0388a 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -1236,6 +1236,20 @@ restart: * concurrently being dropped by a backend connected to another DB. * * That's fairly unlikely in practice, so we'll just bail out. + * + * The slot sync worker holds a shared lock on the database before + * operating on synced logical slots to avoid conflict with the drop + * happening here. The persistent synced slots are thus safe but there + * is a possibility that the slot sync worker has created a temporary + * slot (which stays active even on release) and we are trying to drop + * the same here. In practice, the chances of hitting this scenario is + * very less as during slot synchronization, the temporary slot is + * immediately converted to persistent and thus is safe due to the + * shared lock taken on the database. So for the time being, we'll + * just bail out in such a scenario. + * + * XXX: If needed, we can consider shutting down slot sync worker + * before trying to drop synced temporary slots here. */ if (active_pid) ereport(ERROR, diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index d2fa5e669a..c4a0bf99ee 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -975,7 +975,7 @@ pg_sync_replication_slots(PG_FUNCTION_ARGS) /* Load the libpq-specific functions */ load_file("libpqwalreceiver", false); - ValidateSlotSyncParams(); + ValidateSlotSyncParams(ERROR); initStringInfo(&app_name); if (cluster_name[0]) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index e5477c1de1..b33044e10f 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -3389,7 +3389,7 @@ WalSndDone(WalSndSendDataCallback send_data) * This should only be called when in recovery. * * This is called either by cascading walsender to find WAL postion to be sent - * to a cascaded standby or by slot synchronization function to validate remote + * to a cascaded standby or by slot synchronization operation to validate remote * slot's lsn before syncing it locally. * * As a side-effect, *tli is updated to the TLI of the last diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c index 7e7941d625..8ecd89f8ea 100644 --- a/src/backend/storage/ipc/ipci.c +++ b/src/backend/storage/ipc/ipci.c @@ -154,7 +154,7 @@ CalculateShmemSize(int *num_semaphores) size = add_size(size, StatsShmemSize()); size = add_size(size, WaitEventExtensionShmemSize()); size = add_size(size, InjectionPointShmemSize()); - size = add_size(size, SlotSyncShmemSize()); + size = add_size(size, SlotSyncWorkerShmemSize()); #ifdef EXEC_BACKEND size = add_size(size, ShmemBackendArraySize()); #endif @@ -349,7 +349,7 @@ CreateOrAttachShmemStructs(void) WalSummarizerShmemInit(); PgArchShmemInit(); ApplyLauncherShmemInit(); - SlotSyncShmemInit(); + SlotSyncWorkerShmemInit(); /* * Set up other modules that need some shared memory space diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c index 1afcbfc052..ac83331bbc 100644 --- a/src/backend/storage/lmgr/proc.c +++ b/src/backend/storage/lmgr/proc.c @@ -40,6 +40,7 @@ #include "pgstat.h" #include "postmaster/autovacuum.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/syncrep.h" #include "replication/walsender.h" #include "storage/condition_variable.h" @@ -365,8 +366,12 @@ InitProcess(void) * child; this is so that the postmaster can detect it if we exit without * cleaning up. (XXX autovac launcher currently doesn't participate in * this; it probably should.) + * + * Slot sync worker also does not participate in it, see comments atop + * 'struct bkend' in postmaster.c. */ - if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess()) + if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && + !IsLogicalSlotSyncWorker()) MarkPostmasterChildActive(); /* @@ -935,8 +940,12 @@ ProcKill(int code, Datum arg) * This process is no longer present in shared memory in any meaningful * way, so tell the postmaster we've cleaned up acceptably well. (XXX * autovac launcher should be included here someday) + * + * Slot sync worker is also not a postmaster child, so skip this shared + * memory related processing here. */ - if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess()) + if (IsUnderPostmaster && !IsAutoVacuumLauncherProcess() && + !IsLogicalSlotSyncWorker()) MarkPostmasterChildInactive(); /* wake autovac launcher if needed -- see comments in FreeWorkerInfo */ diff --git a/src/backend/utils/activity/pgstat_io.c b/src/backend/utils/activity/pgstat_io.c index 43c393d6fe..9d6e067382 100644 --- a/src/backend/utils/activity/pgstat_io.c +++ b/src/backend/utils/activity/pgstat_io.c @@ -338,6 +338,7 @@ pgstat_tracks_io_bktype(BackendType bktype) case B_BG_WORKER: case B_BG_WRITER: case B_CHECKPOINTER: + case B_SLOTSYNC_WORKER: case B_STANDALONE_BACKEND: case B_STARTUP: case B_WAL_SENDER: diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index 6464386b77..b52afd4eac 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -53,6 +53,8 @@ LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process." LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process." LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process." RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery." +REPL_SLOTSYNC_MAIN "Waiting in main loop of slot sync worker." +REPL_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down." SYSLOGGER_MAIN "Waiting in main loop of syslogger process." WAL_RECEIVER_MAIN "Waiting in main loop of WAL receiver process." WAL_SENDER_MAIN "Waiting in main loop of WAL sender process." diff --git a/src/backend/utils/init/miscinit.c b/src/backend/utils/init/miscinit.c index 23f77a59e5..26aaf292c5 100644 --- a/src/backend/utils/init/miscinit.c +++ b/src/backend/utils/init/miscinit.c @@ -40,6 +40,7 @@ #include "postmaster/interrupt.h" #include "postmaster/pgarch.h" #include "postmaster/postmaster.h" +#include "replication/slotsync.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/latch.h" @@ -293,6 +294,9 @@ GetBackendTypeDesc(BackendType backendType) case B_LOGGER: backendDesc = "logger"; break; + case B_SLOTSYNC_WORKER: + backendDesc = "slotsyncworker"; + break; case B_STANDALONE_BACKEND: backendDesc = "standalone backend"; break; @@ -835,9 +839,10 @@ InitializeSessionUserIdStandalone(void) { /* * This function should only be called in single-user mode, in autovacuum - * workers, and in background workers. + * workers, in slot sync worker and in background workers. */ - Assert(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || IsBackgroundWorker); + Assert(!IsUnderPostmaster || IsAutoVacuumWorkerProcess() || + IsLogicalSlotSyncWorker() || IsBackgroundWorker); /* call only once */ Assert(!OidIsValid(AuthenticatedUserId)); diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index 7797876d00..5ffe9bdd98 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -43,6 +43,7 @@ #include "postmaster/autovacuum.h" #include "postmaster/postmaster.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/walsender.h" #include "storage/bufmgr.h" #include "storage/fd.h" @@ -876,10 +877,11 @@ InitPostgres(const char *in_dbname, Oid dboid, * Perform client authentication if necessary, then figure out our * postgres user ID, and see if we are a superuser. * - * In standalone mode and in autovacuum worker processes, we use a fixed - * ID, otherwise we figure it out from the authenticated user name. + * In standalone mode, autovacuum worker processes and slot sync worker + * process, we use a fixed ID, otherwise we figure it out from the + * authenticated user name. */ - if (bootstrap || IsAutoVacuumWorkerProcess()) + if (bootstrap || IsAutoVacuumWorkerProcess() || IsLogicalSlotSyncWorker()) { InitializeSessionUserIdStandalone(); am_superuser = true; diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 70652f0a3f..37be0669bb 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -67,6 +67,7 @@ #include "postmaster/walwriter.h" #include "replication/logicallauncher.h" #include "replication/slot.h" +#include "replication/slotsync.h" #include "replication/syncrep.h" #include "storage/bufmgr.h" #include "storage/large_object.h" @@ -2054,6 +2055,15 @@ struct config_bool ConfigureNamesBool[] = NULL, NULL, NULL }, + { + {"sync_replication_slots", PGC_SIGHUP, REPLICATION_STANDBY, + gettext_noop("Enables a physical standby to synchronize logical failover slots from the primary server."), + }, + &sync_replication_slots, + false, + NULL, NULL, NULL + }, + /* End-of-list marker */ { {NULL, 0, 0, NULL, NULL}, NULL, false, NULL, NULL, NULL diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index e10755972a..c97f9a25f0 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -361,6 +361,7 @@ #wal_retrieve_retry_interval = 5s # time to wait before retrying to # retrieve WAL after a failed attempt #recovery_min_apply_delay = 0 # minimum delay for applying changes during recovery +#sync_replication_slots = off # enables slot synchronization on the physical standby from the primary # - Subscribers - diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h index 0445fbf61d..612fb5f42e 100644 --- a/src/include/miscadmin.h +++ b/src/include/miscadmin.h @@ -333,6 +333,7 @@ typedef enum BackendType B_BG_WRITER, B_CHECKPOINTER, B_LOGGER, + B_SLOTSYNC_WORKER, B_STANDALONE_BACKEND, B_STARTUP, B_WAL_RECEIVER, diff --git a/src/include/replication/slotsync.h b/src/include/replication/slotsync.h index e86d8a47b8..aa688a2f1b 100644 --- a/src/include/replication/slotsync.h +++ b/src/include/replication/slotsync.h @@ -14,10 +14,28 @@ #include "replication/walreceiver.h" -extern void ValidateSlotSyncParams(void); +extern PGDLLIMPORT bool sync_replication_slots; + +/* + * GUCs needed by slot sync worker to connect to the primary + * server and carry on with slots synchronization. + */ +extern PGDLLIMPORT char *PrimaryConnInfo; +extern PGDLLIMPORT char *PrimarySlotName; + +extern bool ValidateSlotSyncParams(int elevel); + +#ifdef EXEC_BACKEND +extern void ReplSlotSyncWorkerMain(int argc, char *argv[]) pg_attribute_noreturn(); +#endif +extern int StartSlotSyncWorker(void); + +extern void ShutDownSlotSync(void); +extern bool SlotSyncWorkerCanRestart(void); extern bool IsSyncingReplicationSlots(void); -extern Size SlotSyncShmemSize(void); -extern void SlotSyncShmemInit(void); +extern bool IsLogicalSlotSyncWorker(void); +extern Size SlotSyncWorkerShmemSize(void); +extern void SlotSyncWorkerShmemInit(void); extern void SyncReplicationSlots(WalReceiverConn *wrconn); #endif /* SLOTSYNC_H */ diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index 2755c3fc84..950f6b3738 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -22,7 +22,8 @@ $publisher->append_conf('postgresql.conf', 'autovacuum = off'); $publisher->start; $publisher->safe_psql('postgres', - "CREATE PUBLICATION regress_mypub FOR ALL TABLES;"); + "CREATE PUBLICATION regress_mypub FOR ALL TABLES;" +); my $publisher_connstr = $publisher->connstr . ' dbname=postgres'; @@ -322,6 +323,10 @@ ok( $stderr =~ /HINT: 'dbname' must be specified in "primary_conninfo"/, "cannot sync slots if dbname is not specified in primary_conninfo"); +# Add the dbname back to the primary_conninfo for further tests +$standby1->append_conf('postgresql.conf', "primary_conninfo = '$connstr_1 dbname=postgres'"); +$standby1->reload; + ################################################## # Test that we cannot synchronize slots to a cascading standby server. ################################################## @@ -355,4 +360,110 @@ ok( $stderr =~ /ERROR: cannot synchronize replication slots from a standby server/, "cannot sync slots to a cascading standby server"); +$cascading_standby->stop; + +################################################## +# Test to confirm that the slot sync worker waits until it gets valid values +# for all the slot sync related GUCs +################################################## + +$log_offset = -s $standby1->logfile; + +# Enable slot sync worker but disable another GUC required for slot sync +$standby1->append_conf( + 'postgresql.conf', qq( +sync_replication_slots = on +hot_standby_feedback = off +)); +$standby1->reload; + +# Confirm that the slot sync worker logs bad configuration msg +$standby1->wait_for_log(qr/LOG: bad configuration for slot synchronization/, + $log_offset); + +$log_offset = -s $standby1->logfile; + +$standby1->append_conf('postgresql.conf', "hot_standby_feedback = on"); +$standby1->reload; + +# Confirm that the slot sync worker proceeds once the configuration is correct. +$standby1->wait_for_log(qr/LOG: parameters validation done for slot synchronization/, + $log_offset); + +################################################## +# Test to confirm that restart_lsn and confirmed_flush_lsn of the logical slot +# on the primary is synced to the standby via the slot sync worker. +################################################## + +# Insert data on the primary +$primary->safe_psql( + 'postgres', qq[ + CREATE TABLE tab_int (a int PRIMARY KEY); + INSERT INTO tab_int SELECT generate_series(1, 10); +]); + +# Subscribe to the new table data and wait for it to arrive +$subscriber1->safe_psql( + 'postgres', qq[ + CREATE TABLE tab_int (a int PRIMARY KEY); + ALTER SUBSCRIPTION regress_mysub1 ENABLE; + ALTER SUBSCRIPTION regress_mysub1 REFRESH PUBLICATION; +]); + +$subscriber1->wait_for_subscription_sync; + +# Do not allow any further advancement of the restart_lsn and +# confirmed_flush_lsn for the lsub1_slot. +$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); + +# Wait for the replication slot to become inactive on the publisher +$primary->poll_query_until( + 'postgres', + "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active='f'", + 1); + +# Get the restart_lsn for the logical slot lsub1_slot on the primary +my $primary_restart_lsn = $primary->safe_psql('postgres', + "SELECT restart_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot';"); + +# Get the confirmed_flush_lsn for the logical slot lsub1_slot on the primary +my $primary_flush_lsn = $primary->safe_psql('postgres', + "SELECT confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot';"); + +# Confirm that restart_lsn and of confirmed_flush_lsn lsub1_slot slot are synced +# to the standby +ok( $standby1->poll_query_until( + 'postgres', + "SELECT '$primary_restart_lsn' = restart_lsn AND '$primary_flush_lsn' = confirmed_flush_lsn from pg_replication_slots WHERE slot_name = 'lsub1_slot';"), + 'restart_lsn and confirmed_flush_lsn of slot lsub1_slot synced to standby'); + +################################################## +# Promote the standby1 to primary. Confirm that: +# a) the slot 'lsub1_slot' is retained on the new primary +# b) logical replication for regress_mysub1 is resumed successfully after failover +################################################## +$standby1->promote; + +# Update subscription with the new primary's connection info +my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; +$subscriber1->safe_psql('postgres', + "ALTER SUBSCRIPTION regress_mysub1 CONNECTION '$standby1_conninfo'; + ALTER SUBSCRIPTION regress_mysub1 ENABLE; "); + +# Confirm the synced slot 'lsub1_slot' is retained on the new primary +is($standby1->safe_psql('postgres', + q{SELECT slot_name FROM pg_replication_slots WHERE slot_name = 'lsub1_slot';}), + 'lsub1_slot', + 'synced slot retained on the new primary'); + +# Insert data on the new primary +$standby1->safe_psql('postgres', + "INSERT INTO tab_int SELECT generate_series(11, 20);"); +$standby1->wait_for_catchup('regress_mysub1'); + +# Confirm that data in tab_int replicated on the subscriber +is( $subscriber1->safe_psql('postgres', q{SELECT count(*) FROM tab_int;}), + "20", + 'data replicated from the new primary'); + done_testing(); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index d808aad8b0..a35e399f0c 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2585,7 +2585,7 @@ SlabBlock SlabContext SlabSlot SlotNumber -SlotSyncCtxStruct +SlotSyncWorkerCtxStruct SlruCtl SlruCtlData SlruErrorCause -- 2.34.1 [application/octet-stream] v90-0002-Allow-logical-walsenders-to-wait-for-the-physica.patch (43.5K, ../../CAJpy0uBdktD4Kj9_LiaHyVHhpk959TQuQVVTbbQAGni=CzusVw@mail.gmail.com/4-v90-0002-Allow-logical-walsenders-to-wait-for-the-physica.patch) download | inline diff: From 039c0d3020e71b13302cfbdf620cb48651513707 Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Fri, 16 Feb 2024 15:21:48 +0530 Subject: [PATCH v90 2/3] Allow logical walsenders to wait for the physical This patch introduces a mechanism to ensure that physical standby servers, which are potential failover candidates, have received and flushed changes before making them visible to subscribers. By doing so, it guarantees that the promoted standby server is not lagging behind the subscribers when a failover is necessary. A new parameter named standby_slot_names is introduced. The logical walsender now guarantees that all local changes are sent and flushed to the standby servers corresponding to the replication slots specified in standby_slot_names before sending those changes to the subscriber. Additionally, The SQL functions pg_logical_slot_get_changes and pg_replication_slot_advance are modified to wait for the replication slots mentioned in standby_slot_names to catch up before returning the changes to the user. --- doc/src/sgml/config.sgml | 24 ++ doc/src/sgml/logicaldecoding.sgml | 8 + .../replication/logical/logicalfuncs.c | 13 + src/backend/replication/logical/slotsync.c | 13 + src/backend/replication/slot.c | 342 +++++++++++++++++- src/backend/replication/slotfuncs.c | 9 + src/backend/replication/walsender.c | 110 +++++- .../utils/activity/wait_event_names.txt | 1 + src/backend/utils/misc/guc_tables.c | 14 + src/backend/utils/misc/postgresql.conf.sample | 2 + src/include/replication/slot.h | 7 + src/include/replication/walsender.h | 1 + src/include/replication/walsender_private.h | 7 + src/include/utils/guc_hooks.h | 3 + src/test/recovery/t/006_logical_decoding.pl | 3 +- .../t/040_standby_failover_slots_sync.pl | 261 +++++++++++-- 16 files changed, 761 insertions(+), 57 deletions(-) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index ff184003fe..1c84d35c96 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4420,6 +4420,30 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows </listitem> </varlistentry> + <varlistentry id="guc-standby-slot-names" xreflabel="standby_slot_names"> + <term><varname>standby_slot_names</varname> (<type>string</type>) + <indexterm> + <primary><varname>standby_slot_names</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + List of physical slots guarantees that logical replication slots with + failover enabled do not consume changes until those changes are received + and flushed to corresponding physical standbys. If a logical replication + connection is meant to switch to a physical standby after the standby is + promoted, the physical replication slot for the standby should be listed + here. + </para> + <para> + The standbys corresponding to the physical replication slots in + <varname>standby_slot_names</varname> must configure + <literal>sync_replication_slots = true</literal> so they can receive + failover logical slots changes from the primary. + </para> + </listitem> + </varlistentry> + </variablelist> </sect2> diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index 930c0fa8a6..73b2abeda5 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -384,6 +384,14 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU must be enabled on the standby. It is also necessary to specify a valid <literal>dbname</literal> in the <link linkend="guc-primary-conninfo"><varname>primary_conninfo</varname></link>. + It's also highly recommended that the said physical replication slot + is named in + <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link> + list on the primary, to prevent the subscriber from consuming changes + faster than the hot standby. But once we configure it, then certain latency + is expected in sending changes to logical subscribers due to wait on + physical replication slots in + <link linkend="guc-standby-slot-names"><varname>standby_slot_names</varname></link> </para> <para> diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c index b0081d3ce5..5ff761dd65 100644 --- a/src/backend/replication/logical/logicalfuncs.c +++ b/src/backend/replication/logical/logicalfuncs.c @@ -30,6 +30,7 @@ #include "replication/decode.h" #include "replication/logical.h" #include "replication/message.h" +#include "replication/walsender.h" #include "storage/fd.h" #include "utils/array.h" #include "utils/builtins.h" @@ -109,6 +110,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin MemoryContext per_query_ctx; MemoryContext oldcontext; XLogRecPtr end_of_wal; + XLogRecPtr wait_for_wal_lsn; LogicalDecodingContext *ctx; ResourceOwner old_resowner = CurrentResourceOwner; ArrayType *arr; @@ -228,6 +230,17 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin NameStr(MyReplicationSlot->data.plugin), format_procedure(fcinfo->flinfo->fn_oid)))); + if (XLogRecPtrIsInvalid(upto_lsn)) + wait_for_wal_lsn = end_of_wal; + else + wait_for_wal_lsn = Min(upto_lsn, end_of_wal); + + /* + * Wait for specified streaming replication standby servers (if any) + * to confirm receipt of WAL up to wait_for_wal_lsn. + */ + WaitForStandbyConfirmation(wait_for_wal_lsn); + ctx->output_writer_private = p; /* diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 40ab87bce1..9c5bdb9764 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -487,6 +487,10 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) latestFlushPtr = GetStandbyFlushRecPtr(NULL); if (remote_slot->confirmed_lsn > latestFlushPtr) { + /* + * Can get here only if GUC 'standby_slot_names' on the primary server + * was not configured correctly. + */ elog(am_slotsync_worker ? LOG : ERROR, "skipping slot synchronization as the received slot sync" " LSN %X/%X for slot \"%s\" is ahead of the standby position %X/%X", @@ -864,6 +868,15 @@ validate_remote_info(WalReceiverConn *wrconn, int elevel) remote_in_recovery = DatumGetBool(slot_getattr(tupslot, 1, &isnull)); Assert(!isnull); + /* + * Slot sync is currently not supported on the cascading standby. This is + * because if we allow it, the primary server needs to wait for all the + * cascading standbys, otherwise, logical subscribers can still be ahead + * of one of the cascading standbys which we plan to promote. Thus, to + * avoid this additional complexity, we restrict it for the time being. + * + * XXX: If needed, this can be attempted in future. + */ if (remote_in_recovery) { /* diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index dd4ea0388a..1ab0f758ec 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -46,13 +46,18 @@ #include "common/string.h" #include "miscadmin.h" #include "pgstat.h" +#include "postmaster/interrupt.h" #include "replication/slotsync.h" #include "replication/slot.h" +#include "replication/walsender_private.h" #include "storage/fd.h" #include "storage/ipc.h" #include "storage/proc.h" #include "storage/procarray.h" #include "utils/builtins.h" +#include "utils/guc_hooks.h" +#include "utils/memutils.h" +#include "utils/varlena.h" /* * Replication slot on-disk data structure. @@ -99,10 +104,19 @@ ReplicationSlotCtlData *ReplicationSlotCtl = NULL; /* My backend's replication slot in the shared memory array */ ReplicationSlot *MyReplicationSlot = NULL; -/* GUC variable */ +/* GUC variables */ int max_replication_slots = 10; /* the maximum number of replication * slots */ +/* + * This GUC lists streaming replication standby server slot names that + * logical WAL sender processes will wait for. + */ +char *standby_slot_names; + +/* This is parsed and cached list for raw standby_slot_names. */ +static List *standby_slot_names_list = NIL; + static void ReplicationSlotShmemExit(int code, Datum arg); static void ReplicationSlotDropPtr(ReplicationSlot *slot); @@ -2297,3 +2311,329 @@ GetSlotInvalidationCause(char *conflict_reason) /* Keep compiler quiet */ return RS_INVAL_NONE; } + +/* + * A helper function to validate slots specified in GUC standby_slot_names. + */ +static bool +validate_standby_slots(char **newval) +{ + char *rawname; + List *elemlist; + ListCell *lc; + bool ok; + + /* Need a modifiable copy of string */ + rawname = pstrdup(*newval); + + /* Verify syntax and parse string into a list of identifiers */ + ok = SplitIdentifierString(rawname, ',', &elemlist); + + if (!ok) + GUC_check_errdetail("List syntax is invalid."); + + /* + * If there is a syntax error in the name or if the replication slots' + * data is not initialized yet (i.e., we are in the startup process), skip + * the slot verification. + */ + if (!ok || !ReplicationSlotCtl) + { + pfree(rawname); + list_free(elemlist); + return ok; + } + + foreach(lc, elemlist) + { + char *name = lfirst(lc); + ReplicationSlot *slot; + + slot = SearchNamedReplicationSlot(name, true); + + if (!slot) + { + GUC_check_errdetail("replication slot \"%s\" does not exist", + name); + ok = false; + break; + } + + if (!SlotIsPhysical(slot)) + { + GUC_check_errdetail("\"%s\" is not a physical replication slot", + name); + ok = false; + break; + } + } + + pfree(rawname); + list_free(elemlist); + return ok; +} + +/* + * GUC check_hook for standby_slot_names + */ +bool +check_standby_slot_names(char **newval, void **extra, GucSource source) +{ + if (strcmp(*newval, "") == 0) + return true; + + /* + * "*" is not accepted as in that case primary will not be able to know + * for which all standbys to wait for. Even if we have physical-slots + * info, there is no way to confirm whether there is any standby + * configured for the known physical slots. + */ + if (strcmp(*newval, "*") == 0) + { + GUC_check_errdetail("\"%s\" is not accepted for standby_slot_names", + *newval); + return false; + } + + /* Now verify if the specified slots really exist and have correct type */ + if (!validate_standby_slots(newval)) + return false; + + *extra = guc_strdup(ERROR, *newval); + + return true; +} + +/* + * GUC assign_hook for standby_slot_names + */ +void +assign_standby_slot_names(const char *newval, void *extra) +{ + List *standby_slots; + MemoryContext oldcxt; + char *standby_slot_names_cpy = extra; + + list_free(standby_slot_names_list); + standby_slot_names_list = NIL; + + /* No value is specified for standby_slot_names. */ + if (standby_slot_names_cpy == NULL) + return; + + if (!SplitIdentifierString(standby_slot_names_cpy, ',', &standby_slots)) + { + /* This should not happen if GUC checked check_standby_slot_names. */ + elog(ERROR, "invalid list syntax"); + } + + /* + * Switch to the same memory context under which GUC variables are + * allocated (GUCMemoryContext). + */ + oldcxt = MemoryContextSwitchTo(GetMemoryChunkContext(standby_slot_names_cpy)); + standby_slot_names_list = list_copy(standby_slots); + MemoryContextSwitchTo(oldcxt); +} + +/* + * Return a copy of standby_slot_names_list if the copy flag is set to true, + * otherwise return the original list. + */ +List * +GetStandbySlotList(bool copy) +{ + /* + * Since we do not support syncing slots to cascading standbys, we return + * NIL here if we are running in a standby to indicate that no standby + * slots need to be waited for. + */ + if (RecoveryInProgress()) + return NIL; + + if (copy) + return list_copy(standby_slot_names_list); + else + return standby_slot_names_list; +} + +/* + * Reload the config file and reinitialize the standby slot list if the GUC + * standby_slot_names has changed. + */ +void +RereadConfigAndReInitSlotList(List **standby_slots) +{ + char *pre_standby_slot_names; + + /* + * If we are running on a standby, there is no need to reload + * standby_slot_names since we do not support syncing slots to cascading + * standbys. + */ + if (RecoveryInProgress()) + { + ProcessConfigFile(PGC_SIGHUP); + return; + } + + pre_standby_slot_names = pstrdup(standby_slot_names); + + ProcessConfigFile(PGC_SIGHUP); + + if (strcmp(pre_standby_slot_names, standby_slot_names) != 0) + { + list_free(*standby_slots); + *standby_slots = GetStandbySlotList(true); + } + + pfree(pre_standby_slot_names); +} + +/* + * Filter the standby slots based on the specified log sequence number + * (wait_for_lsn). + * + * This function updates the passed standby_slots list, removing any slots that + * have already caught up to or surpassed the given wait_for_lsn. Additionally, + * it removes slots that have been invalidated, dropped, or converted to + * logical slots. + */ +void +FilterStandbySlots(XLogRecPtr wait_for_lsn, List **standby_slots) +{ + ListCell *lc; + List *standby_slots_cpy = *standby_slots; + + foreach(lc, standby_slots_cpy) + { + char *name = lfirst(lc); + char *warningfmt = NULL; + ReplicationSlot *slot; + + slot = SearchNamedReplicationSlot(name, true); + + if (!slot) + { + /* + * It may happen that the slot specified in standby_slot_names GUC + * value is dropped, so let's skip over it. + */ + warningfmt = _("replication slot \"%s\" specified in parameter \"%s\" does not exist, ignoring"); + } + else if (SlotIsLogical(slot)) + { + /* + * If a logical slot name is provided in standby_slot_names, issue + * a WARNING and skip it. Although logical slots are disallowed in + * the GUC check_hook(validate_standby_slots), it is still + * possible for a user to drop an existing physical slot and + * recreate a logical slot with the same name. Since it is + * harmless, a WARNING should be enough, no need to error-out. + */ + warningfmt = _("cannot have logical replication slot \"%s\" in parameter \"%s\", ignoring"); + } + else + { + SpinLockAcquire(&slot->mutex); + + if (slot->data.invalidated != RS_INVAL_NONE) + { + /* + * Specified physical slot have been invalidated, so no point + * in waiting for it. + */ + warningfmt = _("physical slot \"%s\" specified in parameter \"%s\" has been invalidated, ignoring"); + } + else if (XLogRecPtrIsInvalid(slot->data.restart_lsn) || + slot->data.restart_lsn < wait_for_lsn) + { + bool inactive = (slot->active_pid == 0); + + SpinLockRelease(&slot->mutex); + + /* Log warning if no active_pid for this physical slot */ + if (inactive) + ereport(WARNING, + errmsg("replication slot \"%s\" specified in parameter \"%s\" does not have active_pid", + name, "standby_slot_names"), + errdetail("Logical replication is waiting on the " + "standby associated with \"%s\".", name), + errhint("Consider starting standby associated with " + "\"%s\" or amend standby_slot_names.", name)); + + /* Continue if the current slot hasn't caught up. */ + continue; + } + else + { + Assert(slot->data.restart_lsn >= wait_for_lsn); + } + + SpinLockRelease(&slot->mutex); + } + + /* + * Reaching here indicates that either the slot has passed the + * wait_for_lsn or there is an issue with the slot that requires a + * warning to be reported. + */ + if (warningfmt) + ereport(WARNING, errmsg(warningfmt, name, "standby_slot_names")); + + standby_slots_cpy = foreach_delete_current(standby_slots_cpy, lc); + } + + *standby_slots = standby_slots_cpy; +} + +/* + * Wait for physical standby to confirm receiving the given lsn. + * + * Used by logical decoding SQL functions that acquired slot with failover + * enabled. It waits for physical standbys corresponding to the physical slots + * specified in the standby_slot_names GUC. + */ +void +WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn) +{ + List *standby_slots; + + if (!MyReplicationSlot->data.failover) + return; + + standby_slots = GetStandbySlotList(true); + + if (standby_slots == NIL) + return; + + ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv); + + for (;;) + { + CHECK_FOR_INTERRUPTS(); + + if (ConfigReloadPending) + { + ConfigReloadPending = false; + RereadConfigAndReInitSlotList(&standby_slots); + } + + FilterStandbySlots(wait_for_lsn, &standby_slots); + + /* Exit if done waiting for every slot. */ + if (standby_slots == NIL) + break; + + /* + * We wait for the slots in the standby_slot_names to catch up, but we + * use a timeout so we can also check the if the standby_slot_names + * has been changed. + */ + ConditionVariableTimedSleep(&WalSndCtl->wal_confirm_rcv_cv, 1000, + WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION); + } + + ConditionVariableCancelSleep(); + list_free(standby_slots); +} diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index c4a0bf99ee..4f0a1fddc2 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -22,6 +22,7 @@ #include "replication/logical.h" #include "replication/slot.h" #include "replication/slotsync.h" +#include "replication/walsender.h" #include "utils/builtins.h" #include "utils/guc.h" #include "utils/inval.h" @@ -476,6 +477,8 @@ pg_physical_replication_slot_advance(XLogRecPtr moveto) * crash, but this makes the data consistent after a clean shutdown. */ ReplicationSlotMarkDirty(); + + PhysicalWakeupLogicalWalSnd(); } return retlsn; @@ -516,6 +519,12 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto) .segment_close = wal_segment_close), NULL, NULL, NULL); + /* + * Wait for specified streaming replication standby servers (if any) + * to confirm receipt of WAL up to moveto lsn. + */ + WaitForStandbyConfirmation(moveto); + /* * Start reading at the slot's restart_lsn, which we know to point to * a valid record. diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index b33044e10f..54dbad7158 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1728,27 +1728,78 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId ProcessPendingWrites(); } +/* + * Wake up the logical walsender processes with failover-enabled slots if the + * currently acquired physical slot is specified in standby_slot_names + * GUC. + */ +void +PhysicalWakeupLogicalWalSnd(void) +{ + ListCell *lc; + List *standby_slots; + + Assert(MyReplicationSlot && SlotIsPhysical(MyReplicationSlot)); + + standby_slots = GetStandbySlotList(false); + + foreach(lc, standby_slots) + { + char *name = lfirst(lc); + + if (strcmp(name, NameStr(MyReplicationSlot->data.name)) == 0) + { + ConditionVariableBroadcast(&WalSndCtl->wal_confirm_rcv_cv); + return; + } + } +} + /* * Wait till WAL < loc is flushed to disk so it can be safely sent to client. * - * Returns end LSN of flushed WAL. Normally this will be >= loc, but - * if we detect a shutdown request (either from postmaster or client) - * we will return early, so caller must always check. + * If the walsender holds a logical slot that has enabled failover, we also + * wait for all the specified streaming replication standby servers to + * confirm receipt of WAL up to RecentFlushPtr. + * + * Returns end LSN of flushed WAL. Normally this will be >= loc, but if we + * detect a shutdown request (either from postmaster or client) we will return + * early, so caller must always check. */ static XLogRecPtr WalSndWaitForWal(XLogRecPtr loc) { int wakeEvents; + bool wait_for_standby = false; + uint32 wait_event; + List *standby_slots = NIL; static XLogRecPtr RecentFlushPtr = InvalidXLogRecPtr; + if (MyReplicationSlot->data.failover && replication_active) + standby_slots = GetStandbySlotList(true); + /* - * Fast path to avoid acquiring the spinlock in case we already know we - * have enough WAL available. This is particularly interesting if we're - * far behind. + * Check if all the standby servers have confirmed receipt of WAL up to + * RecentFlushPtr even when we already know we have enough WAL available. + * + * Note that we cannot directly return without checking the status of + * standby servers because the standby_slot_names may have changed, which + * means there could be new standby slots in the list that have not yet + * caught up to the RecentFlushPtr. */ - if (RecentFlushPtr != InvalidXLogRecPtr && - loc <= RecentFlushPtr) - return RecentFlushPtr; + if (!XLogRecPtrIsInvalid(RecentFlushPtr) && loc <= RecentFlushPtr) + { + FilterStandbySlots(RecentFlushPtr, &standby_slots); + + /* + * Fast path to avoid acquiring the spinlock in case we already know + * we have enough WAL available and all the standby servers have + * confirmed receipt of WAL up to RecentFlushPtr. This is particularly + * interesting if we're far behind. + */ + if (standby_slots == NIL) + return RecentFlushPtr; + } /* Get a more recent flush pointer. */ if (!RecoveryInProgress()) @@ -1769,7 +1820,7 @@ WalSndWaitForWal(XLogRecPtr loc) if (ConfigReloadPending) { ConfigReloadPending = false; - ProcessConfigFile(PGC_SIGHUP); + RereadConfigAndReInitSlotList(&standby_slots); SyncRepInitConfig(); } @@ -1784,8 +1835,18 @@ WalSndWaitForWal(XLogRecPtr loc) if (got_STOPPING) XLogBackgroundFlush(); + /* + * Update the standby slots that have not yet caught up to the flushed + * position. It is good to wait up to RecentFlushPtr and then let it + * send the changes to logical subscribers one by one which are + * already covered in RecentFlushPtr without needing to wait on every + * change for standby confirmation. + */ + if (wait_for_standby) + FilterStandbySlots(RecentFlushPtr, &standby_slots); + /* Update our idea of the currently flushed position. */ - if (!RecoveryInProgress()) + else if (!RecoveryInProgress()) RecentFlushPtr = GetFlushRecPtr(NULL); else RecentFlushPtr = GetXLogReplayRecPtr(NULL); @@ -1813,9 +1874,18 @@ WalSndWaitForWal(XLogRecPtr loc) !waiting_for_ping_response) WalSndKeepalive(false, InvalidXLogRecPtr); - /* check whether we're done */ - if (loc <= RecentFlushPtr) + if (loc > RecentFlushPtr) + wait_event = WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL; + else if (standby_slots) + { + wait_event = WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION; + wait_for_standby = true; + } + else + { + /* Already caught up and doesn't need to wait for standby_slots. */ break; + } /* Waiting for new WAL. Since we need to wait, we're now caught up. */ WalSndCaughtUp = true; @@ -1855,9 +1925,11 @@ WalSndWaitForWal(XLogRecPtr loc) if (pq_is_send_pending()) wakeEvents |= WL_SOCKET_WRITEABLE; - WalSndWait(wakeEvents, sleeptime, WAIT_EVENT_WAL_SENDER_WAIT_FOR_WAL); + WalSndWait(wakeEvents, sleeptime, wait_event); } + list_free(standby_slots); + /* reactivate latch so WalSndLoop knows to continue */ SetLatch(MyLatch); return RecentFlushPtr; @@ -2265,6 +2337,7 @@ PhysicalConfirmReceivedLocation(XLogRecPtr lsn) { ReplicationSlotMarkDirty(); ReplicationSlotsComputeRequiredLSN(); + PhysicalWakeupLogicalWalSnd(); } /* @@ -3538,6 +3611,7 @@ WalSndShmemInit(void) ConditionVariableInit(&WalSndCtl->wal_flush_cv); ConditionVariableInit(&WalSndCtl->wal_replay_cv); + ConditionVariableInit(&WalSndCtl->wal_confirm_rcv_cv); } } @@ -3607,8 +3681,14 @@ WalSndWait(uint32 socket_events, long timeout, uint32 wait_event) * * And, we use separate shared memory CVs for physical and logical * walsenders for selective wake ups, see WalSndWakeup() for more details. + * + * If the wait event is WAIT_FOR_STANDBY_CONFIRMATION, wait on another CV + * until awakened by physical walsenders after the walreceiver confirms + * the receipt of the LSN. */ - if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL) + if (wait_event == WAIT_EVENT_WAIT_FOR_STANDBY_CONFIRMATION) + ConditionVariablePrepareToSleep(&WalSndCtl->wal_confirm_rcv_cv); + else if (MyWalSnd->kind == REPLICATION_KIND_PHYSICAL) ConditionVariablePrepareToSleep(&WalSndCtl->wal_flush_cv); else if (MyWalSnd->kind == REPLICATION_KIND_LOGICAL) ConditionVariablePrepareToSleep(&WalSndCtl->wal_replay_cv); diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt index b52afd4eac..2f99649581 100644 --- a/src/backend/utils/activity/wait_event_names.txt +++ b/src/backend/utils/activity/wait_event_names.txt @@ -78,6 +78,7 @@ GSS_OPEN_SERVER "Waiting to read data from the client while establishing a GSSAP LIBPQWALRECEIVER_CONNECT "Waiting in WAL receiver to establish connection to remote server." LIBPQWALRECEIVER_RECEIVE "Waiting in WAL receiver to receive data from remote server." SSL_OPEN_SERVER "Waiting for SSL while attempting connection." +WAIT_FOR_STANDBY_CONFIRMATION "Waiting for the WAL to be received by physical standby." WAL_SENDER_WAIT_FOR_WAL "Waiting for WAL to be flushed in WAL sender process." WAL_SENDER_WRITE_DATA "Waiting for any activity when processing replies from WAL receiver in WAL sender process." diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 37be0669bb..d3bbdbe94e 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -4639,6 +4639,20 @@ struct config_string ConfigureNamesString[] = check_debug_io_direct, assign_debug_io_direct, NULL }, + { + {"standby_slot_names", PGC_SIGHUP, REPLICATION_PRIMARY, + gettext_noop("Lists streaming replication standby server slot " + "names that logical WAL sender processes will wait for."), + gettext_noop("Decoded changes are sent out to plugins by logical " + "WAL sender processes only after specified " + "replication slots confirm receiving WAL."), + GUC_LIST_INPUT | GUC_LIST_QUOTE + }, + &standby_slot_names, + "", + check_standby_slot_names, assign_standby_slot_names, NULL + }, + /* End-of-list marker */ { {NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index c97f9a25f0..cadfe10958 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -334,6 +334,8 @@ # method to choose sync standbys, number of sync standbys, # and comma-separated list of application_name # from standby(s); '*' = all +#standby_slot_names = '' # streaming replication standby server slot names that + # logical walsender processes will wait for # - Standby Servers - diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index e706ca834c..7b754a1f48 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -229,6 +229,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot; /* GUCs */ extern PGDLLIMPORT int max_replication_slots; +extern PGDLLIMPORT char *standby_slot_names; /* shmem initialization functions */ extern Size ReplicationSlotsShmemSize(void); @@ -277,4 +278,10 @@ extern void CheckSlotPermissions(void); extern ReplicationSlotInvalidationCause GetSlotInvalidationCause(char *conflict_reason); +extern List *GetStandbySlotList(bool copy); +extern void WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn); +extern void FilterStandbySlots(XLogRecPtr wait_for_lsn, + List **standby_slots); +extern void RereadConfigAndReInitSlotList(List **standby_slots); + #endif /* SLOT_H */ diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h index 0c3996e926..f2d8297f01 100644 --- a/src/include/replication/walsender.h +++ b/src/include/replication/walsender.h @@ -39,6 +39,7 @@ extern void InitWalSender(void); extern bool exec_replication_command(const char *cmd_string); extern void WalSndErrorCleanup(void); extern void WalSndResourceCleanup(bool isCommit); +extern void PhysicalWakeupLogicalWalSnd(void); extern XLogRecPtr GetStandbyFlushRecPtr(TimeLineID *tli); extern void WalSndSignals(void); extern Size WalSndShmemSize(void); diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h index 3113e9ea47..0f962b0c72 100644 --- a/src/include/replication/walsender_private.h +++ b/src/include/replication/walsender_private.h @@ -113,6 +113,13 @@ typedef struct ConditionVariable wal_flush_cv; ConditionVariable wal_replay_cv; + /* + * Used by physical walsenders holding slots specified in + * standby_slot_names to wake up logical walsenders holding + * failover-enabled slots when a walreceiver confirms the receipt of LSN. + */ + ConditionVariable wal_confirm_rcv_cv; + WalSnd walsnds[FLEXIBLE_ARRAY_MEMBER]; } WalSndCtlData; diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h index 339c490300..dcf9a040d1 100644 --- a/src/include/utils/guc_hooks.h +++ b/src/include/utils/guc_hooks.h @@ -163,5 +163,8 @@ extern bool check_wal_consistency_checking(char **newval, void **extra, extern void assign_wal_consistency_checking(const char *newval, void *extra); extern bool check_wal_segment_size(int *newval, void **extra, GucSource source); extern void assign_wal_sync_method(int new_wal_sync_method, void *extra); +extern bool check_standby_slot_names(char **newval, void **extra, + GucSource source); +extern void assign_standby_slot_names(const char *newval, void *extra); #endif /* GUC_HOOKS_H */ diff --git a/src/test/recovery/t/006_logical_decoding.pl b/src/test/recovery/t/006_logical_decoding.pl index 5c7b4ca5e3..85f019774c 100644 --- a/src/test/recovery/t/006_logical_decoding.pl +++ b/src/test/recovery/t/006_logical_decoding.pl @@ -172,9 +172,10 @@ is($node_primary->slot('otherdb_slot')->{'slot_name'}, undef, 'logical slot was actually dropped with DB'); # Test logical slot advancing and its durability. +# Pass failover=true (last-arg), it should not have any impact on advancing. my $logical_slot = 'logical_slot'; $node_primary->safe_psql('postgres', - "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false);" + "SELECT pg_create_logical_replication_slot('$logical_slot', 'test_decoding', false, false, true);" ); $node_primary->psql( 'postgres', " diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index 950f6b3738..632d99dae4 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -113,19 +113,30 @@ ok( $stderr =~ "cannot sync slots on a non-standby server"); ################################################## -# Test logical failover slots on the standby -# Configure standby1 to replicate and synchronize logical slots configured -# for failover on the primary +# Test primary disallowing specified logical replication slots getting ahead of +# specified physical replication slots. It uses the following set up: # -# failover slot lsub1_slot ->| ----> subscriber1 (connected via logical replication) -# failover slot lsub2_slot | inactive -# primary ---> | -# physical slot sb1_slot --->| ----> standby1 (connected via streaming replication) -# | lsub1_slot, lsub2_slot (synced_slot) +# | ----> standby1 (primary_slot_name = sb1_slot) +# | ----> standby2 (primary_slot_name = sb2_slot) +# primary ----- | +# | ----> subscriber1 (failover = true) +# | ----> subscriber2 (failover = false) +# +# standby_slot_names = 'sb1_slot' +# +# Set up is configured in such a way that the logical slot of subscriber1 is +# enabled failover, thus it will wait for the physical slot of +# standby1(sb1_slot) to catch up before sending decoded changes to subscriber1. ################################################## my $primary = $publisher; my $backup_name = 'backup'; + +$primary->psql('postgres', + q{SELECT pg_create_physical_replication_slot('sb1_slot');}); +$primary->psql('postgres', + q{SELECT pg_create_physical_replication_slot('sb2_slot');}); + $primary->backup($backup_name); # Create a standby @@ -135,13 +146,206 @@ $standby1->init_from_backup( has_streaming => 1, has_restoring => 1); +$standby1->append_conf( + 'postgresql.conf', qq( +primary_slot_name = 'sb1_slot' +)); +$standby1->start; +$primary->wait_for_replay_catchup($standby1); + +# Speed up the slot creation +$primary->safe_psql('postgres', "SELECT pg_log_standby_snapshot()"); + +# Create a slot to save the old xmin info, which speeds up the persistence of +# the newly created slot by obtaining an existing old xmin value. +$standby1->psql('postgres', + "SELECT pg_create_logical_replication_slot('save_xmin', 'test_decoding');"); + +# Create another standby +my $standby2 = PostgreSQL::Test::Cluster->new('standby2'); +$standby2->init_from_backup( + $primary, $backup_name, + has_streaming => 1, + has_restoring => 1); +$standby2->append_conf( + 'postgresql.conf', qq( +primary_slot_name = 'sb2_slot' +)); +$standby2->start; +$primary->wait_for_replay_catchup($standby2); + +# Configure primary to disallow any logical slots that enabled failover from +# getting ahead of specified physical replication slot (sb1_slot). +$primary->append_conf( + 'postgresql.conf', qq( +standby_slot_names = 'sb1_slot' +)); +$primary->reload; + +$primary->safe_psql('postgres', "CREATE TABLE tab_int (a int PRIMARY KEY);"); + +# Create a table and refresh the publication +$subscriber1->safe_psql( + 'postgres', qq[ + CREATE TABLE tab_int (a int PRIMARY KEY); + ALTER SUBSCRIPTION regress_mysub1 REFRESH PUBLICATION WITH (copy_data = false); +]); + +# Create another subscriber node without enabling failover, wait for sync to +# complete +my $subscriber2 = PostgreSQL::Test::Cluster->new('subscriber2'); +$subscriber2->init; +$subscriber2->start; +$subscriber2->safe_psql( + 'postgres', qq[ + CREATE TABLE tab_int (a int PRIMARY KEY); + CREATE SUBSCRIPTION regress_mysub2 CONNECTION '$publisher_connstr' PUBLICATION regress_mypub WITH (slot_name = lsub2_slot, copy_data = false); +]); + +# Stop the standby associated with the specified physical replication slot so +# that the logical replication slot won't receive changes until the standby +# comes up. +$standby1->stop; + +# Create some data on the primary +my $primary_row_count = 10; +$primary->safe_psql('postgres', + "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);"); + +# Wait for the standby that's up and running gets the data from primary +$primary->wait_for_replay_catchup($standby2); +$result = $standby2->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', "standby2 gets data from primary"); + +# Wait for the subscription that's up and running and is not enabled for failover. +# It gets the data from primary without waiting for any standbys. +$publisher->wait_for_catchup('regress_mysub2'); +$result = $subscriber2->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', "subscriber2 gets data from primary"); + +# The subscription that's up and running and is enabled for failover +# doesn't get the data from primary and keeps waiting for the +# standby specified in standby_slot_names. +$result = + $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;"); +is($result, 't', + "subscriber1 doesn't get data from primary until standby1 acknowledges changes" +); + +# Start the standby specified in standby_slot_names and wait for it to catch +# up with the primary. +$standby1->start; +$primary->wait_for_replay_catchup($standby1); +$result = $standby1->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', "standby1 gets data from primary"); + +# Now that the standby specified in standby_slot_names is up and running, +# primary must send the decoded changes to subscription enabled for failover +# While the standby was down, this subscriber didn't receive any data from +# primary i.e. the primary didn't allow it to go ahead of standby. +$publisher->wait_for_catchup('regress_mysub1'); +$result = $subscriber1->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', + "subscriber1 gets data from primary after standby1 acknowledges changes"); + +# Stop the standby associated with the specified physical replication slot so +# that the logical replication slot won't receive changes until the standby +# slot's restart_lsn is advanced or the slot is removed from the +# standby_slot_names list. +$publisher->safe_psql('postgres', "TRUNCATE tab_int;"); +$publisher->wait_for_catchup('regress_mysub1'); +$standby1->stop; + +################################################## +# Verify that when using pg_logical_slot_get_changes to consume changes from a +# logical slot with failover enabled, it will also wait for the slots specified +# in standby_slot_names to catch up. +################################################## + +# Create a logical 'test_decoding' replication slot with failover enabled +$publisher->safe_psql('postgres', + "SELECT pg_create_logical_replication_slot('test_slot', 'test_decoding', false, false, true);" +); + +my $back_q = $primary->background_psql('postgres', on_error_stop => 0); +my $pid = $back_q->query('SELECT pg_backend_pid()'); + +# Try and get changes from the logical slot with failover enabled. +my $offset = -s $primary->logfile; +$back_q->query_until(qr//, + "SELECT pg_logical_slot_get_changes('test_slot', NULL, NULL);\n"); + +# Wait until the primary server logs a warning indicating that it is waiting +# for the sb1_slot to catch up. +$primary->wait_for_log( + qr/WARNING: ( [A-Z0-9]+:)? replication slot \"sb1_slot\" specified in parameter \"standby_slot_names\" does not have active_pid/, + $offset); + +ok($primary->safe_psql('postgres', "SELECT pg_cancel_backend($pid)"), + "cancelling pg_logical_slot_get_changes command"); + +$back_q->quit; + +$publisher->safe_psql('postgres', + "SELECT pg_drop_replication_slot('test_slot');" +); + +################################################## +# Test that logical replication will wait for the user-created inactive +# physical slot to catch up until we remove the slot from standby_slot_names. +################################################## + +# Create some data on the primary +$primary_row_count = 10; +$primary->safe_psql('postgres', + "INSERT INTO tab_int SELECT generate_series(1, $primary_row_count);"); + +$result = + $subscriber1->safe_psql('postgres', "SELECT count(*) = 0 FROM tab_int;"); +is($result, 't', + "subscriber1 doesn't get data as the sb1_slot doesn't catch up"); + +# Remove the standby from the standby_slot_names list and reload the +# configuration. +$primary->adjust_conf('postgresql.conf', 'standby_slot_names', "''"); +$primary->reload; + +# Since there are no slots in standby_slot_names, the primary server should now +# send the decoded changes to the subscription. +$publisher->wait_for_catchup('regress_mysub1'); +$result = $subscriber1->safe_psql('postgres', + "SELECT count(*) = $primary_row_count FROM tab_int;"); +is($result, 't', + "subscriber1 gets data from primary after standby1 is removed from the standby_slot_names list" +); + +# Put the standby back on the primary_slot_name for the rest of the tests +$primary->adjust_conf('postgresql.conf', 'standby_slot_names', 'sb1_slot'); +$primary->reload; + +################################################## +# Test logical failover slots on the standby +# Configure standby1 to replicate and synchronize logical slots configured +# for failover on the primary +# +# failover slot lsub1_slot ->| ----> subscriber1 (connected via logical replication) +# failover slot lsub3_slot | inactive +# primary ---> | +# physical slot sb1_slot --->| ----> standby1 (connected via streaming replication) +# | lsub1_slot, lsub3_slot (synced_slot) +################################################## + +# Configure the standby # Increase the log_min_messages setting to DEBUG2 on both the standby and # primary to debug test failures, if any. my $connstr_1 = $primary->connstr; $standby1->append_conf( 'postgresql.conf', qq( hot_standby_feedback = on -primary_slot_name = 'sb1_slot' primary_conninfo = '$connstr_1 dbname=postgres' log_min_messages = 'debug2' )); @@ -150,29 +354,12 @@ $primary->append_conf('postgresql.conf', "log_min_messages = 'debug2'"); $primary->reload; $primary->psql('postgres', - q{SELECT pg_create_logical_replication_slot('lsub2_slot', 'test_decoding', false, false, true);} + q{SELECT pg_create_logical_replication_slot('lsub3_slot', 'test_decoding', false, false, true);} ); -$primary->psql('postgres', - q{SELECT pg_create_physical_replication_slot('sb1_slot');}); - # Start the standby so that slot syncing can begin $standby1->start; -$primary->wait_for_catchup('regress_mysub1'); - -# Do not allow any further advancement of the restart_lsn for the lsub1_slot. -$subscriber1->safe_psql('postgres', - "ALTER SUBSCRIPTION regress_mysub1 DISABLE"); - -# Wait for the replication slot to become inactive on the publisher -$primary->poll_query_until( - 'postgres', - "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", - 1); - -# Wait for the standby to catch up so that the standby is not lagging behind -# the subscriber. $primary->wait_for_replay_catchup($standby1); # Synchronize the primary server slots to the standby. @@ -182,7 +369,7 @@ $standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); # flagged as 'synced' is( $standby1->safe_psql( 'postgres', - q{SELECT count(*) = 2 FROM pg_replication_slots WHERE slot_name IN ('lsub1_slot', 'lsub2_slot') AND synced AND NOT temporary;} + q{SELECT count(*) = 2 FROM pg_replication_slots WHERE slot_name IN ('lsub1_slot', 'lsub3_slot') AND synced AND NOT temporary;} ), "t", 'logical slots have synced as true on standby'); @@ -192,13 +379,13 @@ is( $standby1->safe_psql( # slot on the primary server has been dropped. ################################################## -$primary->psql('postgres', "SELECT pg_drop_replication_slot('lsub2_slot');"); +$primary->psql('postgres', "SELECT pg_drop_replication_slot('lsub3_slot');"); $standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); is( $standby1->safe_psql( 'postgres', - q{SELECT count(*) = 0 FROM pg_replication_slots WHERE slot_name = 'lsub2_slot';} + q{SELECT count(*) = 0 FROM pg_replication_slots WHERE slot_name = 'lsub3_slot';} ), "t", 'synchronized slot has been dropped'); @@ -398,19 +585,13 @@ $standby1->wait_for_log(qr/LOG: parameters validation done for slot synchroniza # Insert data on the primary $primary->safe_psql( 'postgres', qq[ - CREATE TABLE tab_int (a int PRIMARY KEY); + TRUNCATE TABLE tab_int; INSERT INTO tab_int SELECT generate_series(1, 10); ]); -# Subscribe to the new table data and wait for it to arrive -$subscriber1->safe_psql( - 'postgres', qq[ - CREATE TABLE tab_int (a int PRIMARY KEY); - ALTER SUBSCRIPTION regress_mysub1 ENABLE; - ALTER SUBSCRIPTION regress_mysub1 REFRESH PUBLICATION; -]); - -$subscriber1->wait_for_subscription_sync; +# Enable the subscription to let it catch up to the latest wal position +$subscriber1->safe_psql('postgres', "ALTER SUBSCRIPTION regress_mysub1 ENABLE"); +$primary->wait_for_catchup('regress_mysub1'); # Do not allow any further advancement of the restart_lsn and # confirmed_flush_lsn for the lsub1_slot. -- 2.34.1 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-19 07:53 Bertrand Drouvot <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Bertrand Drouvot @ 2024-02-19 07:53 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: shveta malik <[email protected]>; Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]> Hi, On Sat, Feb 17, 2024 at 10:10:18AM +0530, Amit Kapila wrote: > On Fri, Feb 16, 2024 at 4:10 PM shveta malik <[email protected]> wrote: > > > > On Thu, Feb 15, 2024 at 10:48 PM Bertrand Drouvot > > <[email protected]> wrote: > > > > > 5 === > > > > > > + if (SlotSyncWorker->syncing) > > > { > > > - SpinLockRelease(&SlotSyncCtx->mutex); > > > + SpinLockRelease(&SlotSyncWorker->mutex); > > > ereport(ERROR, > > > errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), > > > errmsg("cannot synchronize replication slots concurrently")); > > > } > > > > > > worth to add a test in 040_standby_failover_slots_sync.pl for it? > > > > It will be very difficult to stabilize this test as we have to make > > sure that the concurrent users (SQL function(s) and/or worker(s)) are > > in that target function at the same time to hit it. > > > > Yeah, I also think would be tricky to write a stable test, maybe one > can explore using a new injection point facility but I don't think it > is worth for this error check as this appears straightforward to be > broken in the future by other changes. Yeah, injection point would probably be the way to go. Agree that's probably not worth adding such a test (we can change our mind later on if needed anyway). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Synchronizing slots from primary to standby @ 2024-02-19 12:01 Amit Kapila <[email protected]> parent: shveta malik <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Amit Kapila @ 2024-02-19 12:01 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Zhijie Hou (Fujitsu) <[email protected]>; Masahiko Sawada <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Dilip Kumar <[email protected]>; Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Eisentraut <[email protected]>; Bruce Momjian <[email protected]>; Ashutosh Sharma <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; Alvaro Herrera <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]> On Mon, Feb 19, 2024 at 9:46 AM shveta malik <[email protected]> wrote: > > Okay I see. Thanks for pointing it out. Here are the patches > addressing your comments. Changes are in patch001, rest are rebased. > Few comments on 0001 ==================== 1. I think it is better to error out when the valid GUC or option is not set in ensure_valid_slotsync_params() and ensure_valid_remote_info() instead of waiting. And we shouldn't start the worker in the first place if not all required GUCs are set. This will additionally simplify the code a bit. 2. +typedef struct SlotSyncWorkerCtxStruct { - /* prevents concurrent slot syncs to avoid slot overwrites */ + pid_t pid; + bool stopSignaled; bool syncing; + time_t last_start_time; slock_t mutex; -} SlotSyncCtxStruct; +} SlotSyncWorkerCtxStruct; I think we don't need to change the name of this struct as this can be used both by the worker and the backend. We can probably add the comment to indicate that all the fields except 'syncing' are used by slotsync worker. 3. Similar to above, function names like SlotSyncShmemInit() shouldn't be changed to SlotSyncWorkerShmemInit(). 4. +ReplSlotSyncWorkerMain(int argc, char *argv[]) { ... + on_shmem_exit(slotsync_worker_onexit, (Datum) 0); ... + before_shmem_exit(slotsync_failure_callback, PointerGetDatum(wrconn)); ... } Do we need two separate callbacks? Can't we have just one (say slotsync_failure_callback) that cleans additional things in case of slotsync worker? And, if we need both those callbacks then please add some comments for both and why one is before_shmem_exit() and the other is on_shmem_exit(). In addition to the above, I have made a few changes in the comments and code (cosmetic code changes). Please include those in the next version if you find them okay. You need to rename .txt file to remove .txt and apply atop v90-0001*. -- With Regards, Amit Kapila. diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index 3080d4aa53..790799c164 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -463,9 +463,9 @@ static void InitPostmasterDeathWatchHandle(void); /* * Slot sync worker allowed to start up? * - * If we are on a hot standby, slot sync parameter is enabled, and it is - * the first time of worker's launch, or enough time has passed since the - * worker was launched last, then it is allowed to be started. + * We allow to start the slot sync worker when we are on a hot standby, + * sync_replication_slots is enabled, and it is the first time of worker's + * launch, or enough time has passed since the worker was launched last. */ #define SlotSyncWorkerAllowed() \ (sync_replication_slots && pmState == PM_HOT_STANDBY && \ diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 40ab87bce1..0e0f3cdf76 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -1052,9 +1052,9 @@ ensure_valid_slotsync_params(void) /* * Re-read the config file. * - * If any of the slot sync GUCs have changed, exit the worker and - * let it get restarted by the postmaster. The worker to be exited for - * restart purpose only if the caller passed restart as true. + * Exit if any of the slot sync GUCs have changed. The postmaster will + * restart it. The worker to be exited for restart purpose only if the + * caller passed restart as true. */ static void slotsync_reread_config(bool restart) @@ -1295,9 +1295,9 @@ ReplSlotSyncWorkerMain(int argc, char *argv[]) dbname = walrcv_get_dbname_from_conninfo(PrimaryConnInfo); /* - * Connect to the database specified by user in primary_conninfo. We need - * a database connection for walrcv_exec to work. Please see comments atop - * libpqrcv_exec. + * Connect to the database specified by the user in primary_conninfo. We + * need a database connection for walrcv_exec to work which we use to fetch + * slot information from the remote node. See comments atop libpqrcv_exec. */ InitPostgres(dbname, InvalidOid, NULL, InvalidOid, 0, NULL); @@ -1310,12 +1310,11 @@ ReplSlotSyncWorkerMain(int argc, char *argv[]) appendStringInfo(&app_name, "%s", "slotsyncworker"); /* - * Establish the connection to the primary server for slots + * Establish the connection to the primary server for slot * synchronization. */ wrconn = walrcv_connect(PrimaryConnInfo, false, false, false, - app_name.data, - &err); + app_name.data, &err); pfree(app_name.data); if (!wrconn) @@ -1332,7 +1331,7 @@ ReplSlotSyncWorkerMain(int argc, char *argv[]) */ ensure_valid_remote_info(wrconn); - /* Main wait loop */ + /* Main loop to synchronize slots */ for (;;) { bool some_slot_updated = false; @@ -1345,7 +1344,7 @@ ReplSlotSyncWorkerMain(int argc, char *argv[]) } /* - * The slot sync worker can not get here because it will only stop when it + * The slot sync worker can't get here because it will only stop when it * receives a SIGINT from the startup process, or when there is an error. */ Assert(false); Attachments: [text/plain] v90_0001_amit.patch.txt (3.2K, ../../CAA4eK1L=P0zopPPiue_b4dkRAPFpyEJCY7hMiC2iN9yzx2UMqw@mail.gmail.com/2-v90_0001_amit.patch.txt) download | inline diff: diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index 3080d4aa53..790799c164 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -463,9 +463,9 @@ static void InitPostmasterDeathWatchHandle(void); /* * Slot sync worker allowed to start up? * - * If we are on a hot standby, slot sync parameter is enabled, and it is - * the first time of worker's launch, or enough time has passed since the - * worker was launched last, then it is allowed to be started. + * We allow to start the slot sync worker when we are on a hot standby, + * sync_replication_slots is enabled, and it is the first time of worker's + * launch, or enough time has passed since the worker was launched last. */ #define SlotSyncWorkerAllowed() \ (sync_replication_slots && pmState == PM_HOT_STANDBY && \ diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 40ab87bce1..0e0f3cdf76 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -1052,9 +1052,9 @@ ensure_valid_slotsync_params(void) /* * Re-read the config file. * - * If any of the slot sync GUCs have changed, exit the worker and - * let it get restarted by the postmaster. The worker to be exited for - * restart purpose only if the caller passed restart as true. + * Exit if any of the slot sync GUCs have changed. The postmaster will + * restart it. The worker to be exited for restart purpose only if the + * caller passed restart as true. */ static void slotsync_reread_config(bool restart) @@ -1295,9 +1295,9 @@ ReplSlotSyncWorkerMain(int argc, char *argv[]) dbname = walrcv_get_dbname_from_conninfo(PrimaryConnInfo); /* - * Connect to the database specified by user in primary_conninfo. We need - * a database connection for walrcv_exec to work. Please see comments atop - * libpqrcv_exec. + * Connect to the database specified by the user in primary_conninfo. We + * need a database connection for walrcv_exec to work which we use to fetch + * slot information from the remote node. See comments atop libpqrcv_exec. */ InitPostgres(dbname, InvalidOid, NULL, InvalidOid, 0, NULL); @@ -1310,12 +1310,11 @@ ReplSlotSyncWorkerMain(int argc, char *argv[]) appendStringInfo(&app_name, "%s", "slotsyncworker"); /* - * Establish the connection to the primary server for slots + * Establish the connection to the primary server for slot * synchronization. */ wrconn = walrcv_connect(PrimaryConnInfo, false, false, false, - app_name.data, - &err); + app_name.data, &err); pfree(app_name.data); if (!wrconn) @@ -1332,7 +1331,7 @@ ReplSlotSyncWorkerMain(int argc, char *argv[]) */ ensure_valid_remote_info(wrconn); - /* Main wait loop */ + /* Main loop to synchronize slots */ for (;;) { bool some_slot_updated = false; @@ -1345,7 +1344,7 @@ ReplSlotSyncWorkerMain(int argc, char *argv[]) } /* - * The slot sync worker can not get here because it will only stop when it + * The slot sync worker can't get here because it will only stop when it * receives a SIGINT from the startup process, or when there is an error. */ Assert(false); ^ permalink raw reply [nested|flat] 46+ messages in thread
end of thread, other threads:[~2024-02-19 12:01 UTC | newest] Thread overview: 46+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2019-06-29 00:10 Re: [bug fix] Produce a crash dump before main() on Windows Amit Kapila <[email protected]> 2023-02-15 22:48 [PATCH v16 4/5] restructure archive modules docs in preparation for restore modules Nathan Bossart <[email protected]> 2023-02-15 22:48 [PATCH v19 4/5] restructure archive modules docs in preparation for restore modules Nathan Bossart <[email protected]> 2023-02-15 22:48 [PATCH v18 4/5] restructure archive modules docs in preparation for restore modules Nathan Bossart <[email protected]> 2023-02-15 22:48 [PATCH v18 4/5] restructure archive modules docs in preparation for restore modules Nathan Bossart <[email protected]> 2023-02-15 22:48 [PATCH v15 6/7] restructure archive modules docs in preparation for restore modules Nathan Bossart <[email protected]> 2023-02-15 22:48 [PATCH v19 4/5] restructure archive modules docs in preparation for restore modules Nathan Bossart <[email protected]> 2023-02-15 22:48 [PATCH v20 4/5] restructure archive modules docs in preparation for restore modules Nathan Bossart <[email protected]> 2023-02-15 22:48 [PATCH v13 6/7] restructure archive modules docs in preparation for restore modules Nathan Bossart <[email protected]> 2023-02-15 22:48 [PATCH v18 4/5] restructure archive modules docs in preparation for restore modules Nathan Bossart <[email protected]> 2023-02-15 22:48 [PATCH v14 6/7] restructure archive modules docs in preparation for restore modules Nathan Bossart <[email protected]> 2023-02-15 22:48 [PATCH v12 5/6] restructure archive modules docs in preparation for restore modules Nathan Bossart <[email protected]> 2023-02-15 22:48 [PATCH v20 4/5] restructure archive modules docs in preparation for restore modules Nathan Bossart <[email protected]> 2023-02-15 22:48 [PATCH v19 4/5] restructure archive modules docs in preparation for restore modules Nathan Bossart <[email protected]> 2024-02-10 12:00 Re: Synchronizing slots from primary to standby Masahiko Sawada <[email protected]> 2024-02-10 13:10 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]> 2024-02-11 13:23 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]> 2024-02-12 09:40 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]> 2024-02-13 01:15 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]> 2024-02-13 03:38 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]> 2024-02-13 04:08 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]> 2024-02-13 06:50 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]> 2024-02-13 12:35 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]> 2024-02-13 11:29 ` Re: Synchronizing slots from primary to standby Bertrand Drouvot <[email protected]> 2024-02-13 11:50 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]> 2024-02-13 15:55 ` Re: Synchronizing slots from primary to standby Bertrand Drouvot <[email protected]> 2024-02-14 02:39 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]> 2024-02-14 04:03 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]> 2024-02-14 08:44 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]> 2024-02-14 10:05 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]> 2024-02-13 12:39 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]> 2024-02-15 06:36 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]> 2024-02-15 12:43 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]> 2024-02-15 17:18 ` Re: Synchronizing slots from primary to standby Bertrand Drouvot <[email protected]> 2024-02-16 10:40 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]> 2024-02-17 04:40 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]> 2024-02-19 07:53 ` Re: Synchronizing slots from primary to standby Bertrand Drouvot <[email protected]> 2024-02-18 14:09 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]> 2024-02-19 03:38 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]> 2024-02-19 04:02 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]> 2024-02-19 04:16 ` Re: Synchronizing slots from primary to standby shveta malik <[email protected]> 2024-02-19 12:01 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]> 2024-02-12 10:03 ` Re: Synchronizing slots from primary to standby Bertrand Drouvot <[email protected]> 2024-02-12 10:49 ` Re: Synchronizing slots from primary to standby Amit Kapila <[email protected]> 2024-02-12 14:06 ` Re: Synchronizing slots from primary to standby Bertrand Drouvot <[email protected]> 2024-02-13 01:15 ` RE: Synchronizing slots from primary to standby Zhijie Hou (Fujitsu) <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox