public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v7 1/2] Allow file inclusion in pg_hba and pg_ident files.
33+ messages / 8 participants
[nested] [flat]
* [PATCH v7 1/2] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-02-21 07:45 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw)
Catversion is bumped.
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/catalogs.sgml | 48 +++-
doc/src/sgml/client-auth.sgml | 48 +++-
src/backend/libpq/hba.c | 343 +++++++++++++++++++++----
src/backend/libpq/pg_hba.conf.sample | 10 +-
src/backend/libpq/pg_ident.conf.sample | 12 +-
src/backend/utils/adt/hbafuncs.c | 51 +++-
src/include/catalog/pg_proc.dat | 11 +-
src/include/libpq/hba.h | 2 +
src/test/regress/expected/rules.out | 12 +-
9 files changed, 453 insertions(+), 84 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index a533a2153e..b44e62d388 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -10554,12 +10554,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>rule_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this rule if the rule is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule in the given file_name
</para></entry>
</row>
@@ -10694,6 +10713,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>mapping_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this mapping if the mapping is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this mapping
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>line_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Line number of this mapping in the given file_name
+ </para></entry>
+ </row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index b2a459fb0d..f7d871e660 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,21 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> or <literal>include_dir</literal> directive and
+ the file or directory to be included. The file or directory can be a
+ relative of absolute path, and can be double quoted if needed. For the
+ <literal>include_dir</literal> form, all files not starting with a
+ <literal>.</literal> and ending with <literal>.conf</literal> will be
+ included.
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,6 +116,8 @@
<para>
A record can have several formats:
<synopsis>
+include <replaceable>file</replaceable>
+include_dir <replaceable>directory</replaceable>
local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
@@ -118,6 +133,26 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_dir</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of all the files found in
+ the directory, if they don't start with a <literal>.</literal> and end
+ with <literal>.conf</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -835,8 +870,10 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
+<replaceable>include_dir</replaceable> <replaceable>directory</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -847,6 +884,11 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ As for <filename>pg_hba.conf</filename>, the lines in this file can either
+ be inclusion directives or an authentication rule, and follow the same
+ rules.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index 327a4b42af..56b6cec9d5 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -21,6 +21,7 @@
#include <fcntl.h>
#include <sys/param.h>
#include <sys/socket.h>
+#include <sys/stat.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
@@ -68,6 +69,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -112,10 +119,19 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
const char *inc_filename, int elevel, char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
+static char *process_included_authfile(const char *inc_filename,
+ const char *outer_filename, int elevel,
+ MemoryContext linecxt, List **tok_lines);
/*
@@ -355,36 +371,11 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
- {
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
- }
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename,
+ elevel, err_msg, &inc_fullname);
- inc_file = AllocateFile(inc_fullname, "r");
if (inc_file == NULL)
- {
- int save_errno = errno;
-
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
return tokens;
- }
/* There is possible recursion here if the file contains @ */
linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
@@ -425,11 +416,36 @@ tokenize_inc_file(List *tokens,
/*
* tokenize_auth_file
- * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a decicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel);
+
+ return linecxt;
+}
+
+/*
+ * Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see the struct definition
* in libpq/hba.h.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -438,30 +454,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -522,29 +530,151 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ AuthToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
+
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, filename,
+ elevel, linecxt, tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
+ else if (strcmp(first->string, "include_dir") == 0)
+ {
+ char *dir_name;
+ DIR *d;
+ struct dirent *de;
+ StringInfoData err_buf;
+
+ dir_name = second->string;
+ d = AllocateDir(dir_name);
+ if (d == NULL)
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not open directory \"%s\": %m",
+ dir_name)));
+ err_msg = psprintf("coud not open directory \"%s\": %m",
+ dir_name);
+ goto process_line;
+ }
+
+ initStringInfo(&err_buf);
+ while ((de = ReadDir(d, dir_name)) != NULL)
+ {
+ struct stat st;
+ char inc_filename[MAXPGPATH];
+
+ /*
+ * Only parse files with names ending in ".conf".
+ * Explicitly reject files starting with ".". This
+ * excludes things like "." and "..", as well as typical
+ * hidden files, backup files, and editor debris.
+ */
+ if (strlen(de->d_name) < 6)
+ continue;
+ if (de->d_name[0] == '.')
+ continue;
+ if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
+ continue;
+
+ join_path_components(inc_filename, dir_name, de->d_name);
+ canonicalize_path(inc_filename);
+ if (stat(inc_filename, &st) == 0)
+ {
+ /* Ignore directories. */
+ if (S_ISDIR(st.st_mode))
+ continue;
+
+ /*
+ * err_msg is used here as a temp buffer, it will be
+ * overwritten at the end of the loop with the
+ * cumulated errors, if any.
+ */
+ err_msg = process_included_authfile(inc_filename,
+ filename, elevel, linecxt,
+ tok_lines);
+
+ /* Cumulate errors if any. */
+ if (err_msg)
+ {
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfoString(&err_buf, err_msg);
+ }
+ }
+ else
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not stat file \"%s\": %m",
+ filename)));
+
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfo(&err_buf,
+ "could not stat file \"%s\": %m",
+ inc_filename);
+ }
+ }
+ FreeDir(d);
+
+ /*
+ * If there were no errors, the line is fully processed, bypass
+ * the general TokenizedAuthLine processing.
+ */
+ if (err_buf.len == 0)
+ goto next_line;
+
+ /* Otherwise, process the cumulated errors, if any. */
+ err_msg = err_buf.data;
+ }
}
+process_line:
+ /*
+ * General processing: report the error if any and emit line to the
+ * TokenizedAuthLine
+ */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
+
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -859,7 +989,7 @@ do { \
errmsg("authentication option \"%s\" is only valid for authentication methods %s", \
optname, _(validmethods)), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \
optname, validmethods); \
return false; \
@@ -879,7 +1009,7 @@ do { \
errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname); \
return NULL; \
@@ -901,7 +1031,7 @@ do { \
ereport(elevel, \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("missing entry in file \"%s\" at end of line %d", \
- IdentFileName, line_num))); \
+ tok_line->file_name, line_num))); \
*err_msg = psprintf("missing entry at end of line"); \
return NULL; \
} \
@@ -914,7 +1044,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("multiple values in ident field"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, tok_line->file_name))); \
*err_msg = psprintf("multiple values in ident field"); \
return NULL; \
} \
@@ -937,6 +1067,7 @@ HbaLine *
parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
{
int line_num = tok_line->line_num;
+ char *file_name = tok_line->file_name;
char **err_msg = &tok_line->err_msg;
char *str;
struct addrinfo *gai_result;
@@ -951,6 +1082,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -1675,6 +1807,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg)
{
int line_num = hbaline->linenumber;
+ char *file_name = hbaline->sourcefile;
#ifdef USE_LDAP
hbaline->ldapscope = LDAP_SCOPE_SUBTREE;
@@ -2296,6 +2429,102 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
+
+/*
+ * Try to open an included file, and tokenize it using the given context.
+ */
+static char *
+process_included_authfile(const char *inc_filename, const char *outer_filename,
+ int elevel, MemoryContext linecxt, List **tok_lines)
+{
+ char *inc_fullname;
+ FILE *inc_file;
+ char *err_msg = NULL;
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename, outer_filename,
+ elevel, &err_msg, &inc_fullname);
+
+ if (inc_file == NULL)
+ {
+ /* open_inc_file should have reported an error. */
+ Assert(err_msg != NULL);
+ return err_msg;
+ }
+ else
+ {
+ /* No error message should have been reported. */
+ Assert(err_msg == NULL);
+ }
+
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ return NULL;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..0050e24186 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,6 +9,8 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
+# include FILE
+# include_dir DIRECTORY
# local DATABASE USER METHOD [OPTIONS]
# host DATABASE USER ADDRESS METHOD [OPTIONS]
# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
@@ -18,7 +20,13 @@
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include" or "include_dir", it's not a mapping record but a directive to
+# include records from other file(s), as specified in the field. FILE is the
+# file to include, and DIR is the directory containing the file(s) to include
+# It can be specified with a relative or absolute path, and can be double
+# quoted if it contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..9b700d9989 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,20 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
-# MAPNAME SYSTEM-USERNAME PG-USERNAME
+# include FILE
+# include_dir FILE
+# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include" or "include_dir", it's not an authentication
+# record but a directive to include records from other file(s), specified in
+# the field. FILE is the file to include and DIR is the directory containing
+# the file(s) to include. It can be specified with a relative or absolute
+# path, and can be double quoted if it contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index 9fe7b62c9a..4d902927e0 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,9 +26,11 @@
static ArrayType *get_hba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
+ int mapping_number, const char *filename,
int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
@@ -157,7 +159,7 @@ get_hba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 9
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line
@@ -174,7 +176,8 @@ get_hba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg)
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
+ const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
bool nulls[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -193,6 +196,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* rule_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -336,7 +346,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -359,6 +369,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *hba_lines = NIL;
ListCell *line;
+ int rule_number = 0;
MemoryContext linecxt;
MemoryContext hbacxt;
MemoryContext oldcxt;
@@ -393,8 +404,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
hbaline = parse_hba_line(tok_line, DEBUG3);
- fill_hba_line(tuple_store, tupdesc, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ rule_number++;
+
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -430,8 +445,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
PG_RETURN_NULL();
}
-/* Number of columns in pg_ident_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5
+/* Number of columns in pg_hba_file_mappings view */
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -448,7 +463,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, IdentLine *ident, const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -461,6 +477,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* mapping_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -473,7 +496,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -495,6 +518,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *ident_lines = NIL;
ListCell *line;
+ int mapping_number = 0;
MemoryContext linecxt;
MemoryContext identcxt;
MemoryContext oldcxt;
@@ -529,7 +553,12 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
identline = parse_ident_line(tok_line, DEBUG3);
- fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline,
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ mapping_number++;
+
+ fill_ident_line(tuple_store, tupdesc, mapping_number,
+ tok_line->file_name, tok_line->line_num, identline,
tok_line->err_msg);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index babe16f00a..65bfd32753 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6128,15 +6128,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '6250', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,text,text,text}', proargmodes => '{o,o,o,o,o}',
- proargnames => '{line_number,map_name,sys_name,pg_username,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index 90036f7bcd..59f6faf9f8 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -155,6 +156,7 @@ typedef struct AuthToken
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of AuthTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 21effe8315..11ffe1743b 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname,
WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
-pg_hba_file_rules| SELECT a.line_number,
+pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
+ a.line_number,
a.type,
a.database,
a.user_name,
@@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error);
-pg_ident_file_mappings| SELECT a.line_number,
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
+ a.line_number,
a.map_name,
a.sys_name,
a.pg_username,
a.error
- FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_username, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.33.1
--o45jk7yo2ulu7ifa
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v4 2/3] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-02-21 07:45 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw)
Catversion is bumped.
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/catalogs.sgml | 48 +++++-
doc/src/sgml/client-auth.sgml | 34 +++-
src/backend/libpq/hba.c | 229 +++++++++++++++++++------
src/backend/libpq/pg_hba.conf.sample | 8 +-
src/backend/libpq/pg_ident.conf.sample | 8 +-
src/backend/utils/adt/hbafuncs.c | 53 ++++--
src/include/catalog/pg_proc.dat | 12 +-
src/include/libpq/hba.h | 2 +
src/test/regress/expected/rules.out | 12 +-
9 files changed, 321 insertions(+), 85 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 75fedfa07e..bd1c9a8d21 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -10496,12 +10496,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>rule_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this rule if the rule is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule in the given file_name
</para></entry>
</row>
@@ -10637,6 +10656,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>mapping_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this mapping if the mapping is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this mapping
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>line_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Line number of this mapping in the given file_name
+ </para></entry>
+ </row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index 142b0affcb..e1d0e103b3 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,17 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> directive and the file to be included. The file
+ can be a relative of absolute path, and can be double quoted if needed.
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,6 +112,7 @@
<para>
A record can have several formats:
<synopsis>
+include <replaceable>file</replaceable>
local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
@@ -118,6 +128,15 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -835,8 +854,9 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -847,6 +867,14 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ The lines can record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> directive and the file to be included. The file
+ can be a relative of absolute path, and can be double quoted if needed.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index 556f473b41..1551b34c53 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -68,6 +68,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -112,10 +118,16 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
const char *inc_filename, int elevel, char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
/*
@@ -355,36 +367,11 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
- {
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
- }
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename,
+ elevel, err_msg, &inc_fullname);
- inc_file = AllocateFile(inc_fullname, "r");
if (inc_file == NULL)
- {
- int save_errno = errno;
-
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
return tokens;
- }
/* There is possible recursion here if the file contains @ */
linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
@@ -425,11 +412,36 @@ tokenize_inc_file(List *tokens,
/*
* tokenize_auth_file
- * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a decicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel);
+
+ return linecxt;
+}
+
+/*
+ * Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see the struct definition
* in libpq/hba.h.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -438,30 +450,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -522,29 +526,76 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ AuthToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
+ char *inc_fullname;
+ FILE *inc_file;
+
+ inc_filename = second->string;
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename,
+ filename, elevel, &err_msg,
+ &inc_fullname);
+
+ /*
+ * The included file could be open, now recursively process it.
+ * Errors will be reported in the general TokenizedAuthLine
+ * processing.
+ */
+ if (inc_file != NULL)
+ {
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ else
+ {
+ /* We should got an error */
+ Assert(err_msg != NULL);
+ }
+ }
}
+ /* General processing: emit line to the TokenizedAuthLine */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
+
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -859,7 +910,7 @@ do { \
errmsg("authentication option \"%s\" is only valid for authentication methods %s", \
optname, _(validmethods)), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \
optname, validmethods); \
return false; \
@@ -879,7 +930,7 @@ do { \
errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname); \
return NULL; \
@@ -901,7 +952,7 @@ do { \
ereport(elevel, \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("missing entry in file \"%s\" at end of line %d", \
- IdentFileName, line_num))); \
+ tok_line->file_name, line_num))); \
*err_msg = psprintf("missing entry at end of line"); \
return NULL; \
} \
@@ -914,7 +965,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("multiple values in ident field"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, tok_line->file_name))); \
*err_msg = psprintf("multiple values in ident field"); \
return NULL; \
} \
@@ -937,6 +988,7 @@ HbaLine *
parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
{
int line_num = tok_line->line_num;
+ char *file_name = tok_line->file_name;
char **err_msg = &tok_line->err_msg;
char *str;
struct addrinfo *gai_result;
@@ -951,6 +1003,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -1677,6 +1730,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg)
{
int line_num = hbaline->linenumber;
+ char *file_name = hbaline->sourcefile;
#ifdef USE_LDAP
hbaline->ldapscope = LDAP_SCOPE_SUBTREE;
@@ -2299,6 +2353,67 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..0b6589a7b9 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,6 +9,7 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
+# include FILE
# local DATABASE USER METHOD [OPTIONS]
# host DATABASE USER ADDRESS METHOD [OPTIONS]
# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
@@ -18,7 +19,12 @@
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include", it's not a mapping record but a directive to
+# include records from another file, specified in the field. FILE is the file
+# to include. It can be specified with a relative or absolute path, and can be
+# double quoted if it contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..138359cf03 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,18 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
+# include FILE
# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include", it's not an authentication record but a
+# directive to include records from another file, specified in the field. FILE
+# is the file to include. It can be specified with a relative or absolute
+# path, and can be double quoted if it contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index 1970b4c497..33cf5fb954 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,9 +26,11 @@
static ArrayType *get_hba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
+ int mapping_number, const char *filename,
int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
@@ -157,7 +159,7 @@ get_hba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 9
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line
@@ -174,7 +176,8 @@ get_hba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg)
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
+ const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
bool nulls[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -193,6 +196,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* rule_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -336,7 +346,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -359,6 +369,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *hba_lines = NIL;
ListCell *line;
+ int rule_number = 0;
MemoryContext linecxt;
MemoryContext hbacxt;
MemoryContext oldcxt;
@@ -393,8 +404,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
hbaline = parse_hba_line(tok_line, DEBUG3);
- fill_hba_line(tuple_store, tupdesc, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ rule_number++;
+
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -430,8 +445,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
PG_RETURN_NULL();
}
-/* Number of columns in pg_ident_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5
+/* Number of columns in pg_hba_file_mappings view */
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -448,7 +463,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, IdentLine *ident, const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -461,6 +477,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* mapping_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -473,7 +496,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -495,6 +518,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *ident_lines = NIL;
ListCell *line;
+ int mapping_number = 0;
MemoryContext linecxt;
MemoryContext identcxt;
MemoryContext oldcxt;
@@ -529,8 +553,13 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
identline = parse_ident_line(tok_line, DEBUG3);
- fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline,
- tok_line->err_msg);
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ mapping_number++;
+
+ fill_ident_line(tuple_store, tupdesc, mapping_number,
+ tok_line->file_name, tok_line->line_num, identline,
+ tok_line->err_msg);
}
/* Free tokenizer memory */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index abc9a83223..87842d5ce2 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6111,16 +6111,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '9556', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,text,text,text}',
- proargmodes => '{o,o,o,o,o}',
- proargnames => '{line_number,map_name,sys_name,pg_username,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index 90036f7bcd..59f6faf9f8 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -155,6 +156,7 @@ typedef struct AuthToken
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of AuthTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 5a20e5a7e1..e2c6aa8ffb 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname,
WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
-pg_hba_file_rules| SELECT a.line_number,
+pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
+ a.line_number,
a.type,
a.database,
a.user_name,
@@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error);
-pg_ident_file_mappings| SELECT a.line_number,
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
+ a.line_number,
a.map_name,
a.sys_name,
a.pg_username,
a.error
- FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_username, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.33.1
--4xmgvl4uv4ppkth6
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v4-0003-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v3 3/4] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-02-21 07:45 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw)
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/catalogs.sgml | 48 +++++-
doc/src/sgml/client-auth.sgml | 34 +++-
src/backend/libpq/hba.c | 218 +++++++++++++++++++------
src/backend/libpq/pg_hba.conf.sample | 8 +-
src/backend/libpq/pg_ident.conf.sample | 8 +-
src/backend/utils/adt/hbafuncs.c | 51 ++++--
src/include/catalog/pg_proc.dat | 12 +-
src/include/libpq/hba.h | 2 +
src/test/regress/expected/rules.out | 12 +-
9 files changed, 314 insertions(+), 79 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index e87f0543d6..c6c5e7d08e 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -10440,12 +10440,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>rule_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this rule if the rule is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule in the given file_name
</para></entry>
</row>
@@ -10581,6 +10600,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>mapping_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this mapping if the mapping is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this mapping
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>line_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Line number of this mapping in the given file_name
+ </para></entry>
+ </row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index 142b0affcb..e1d0e103b3 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,17 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> directive and the file to be included. The file
+ can be a relative of absolute path, and can be double quoted if needed.
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,6 +112,7 @@
<para>
A record can have several formats:
<synopsis>
+include <replaceable>file</replaceable>
local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
@@ -118,6 +128,15 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -835,8 +854,9 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -847,6 +867,14 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ The lines can record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> directive and the file to be included. The file
+ can be a relative of absolute path, and can be double quoted if needed.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index 68136f6244..1bfcaef025 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -68,6 +68,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -112,10 +118,16 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
const char *inc_filename, int elevel, char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
/*
@@ -355,36 +367,11 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
- {
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
- }
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename,
+ elevel, err_msg, &inc_fullname);
- inc_file = AllocateFile(inc_fullname, "r");
if (inc_file == NULL)
- {
- int save_errno = errno;
-
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
return tokens;
- }
/* There is possible recursion here if the file contains @ */
linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
@@ -423,12 +410,38 @@ tokenize_inc_file(List *tokens,
return tokens;
}
+/*
+ * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a decicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel);
+
+ return linecxt;
+}
+
/*
* Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see struct definition
* above.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -437,30 +450,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -521,29 +526,76 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ HbaToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
+ char *inc_fullname;
+ FILE *inc_file;
+
+ inc_filename = second->string;
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename,
+ filename, elevel, &err_msg,
+ &inc_fullname);
+
+ /*
+ * The included file could be open, now recursively process it.
+ * Errors will be reported in the general TokenizedAuthLine
+ * processing.
+ */
+ if (inc_file != NULL)
+ {
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ else
+ {
+ /* We should got an error */
+ Assert(err_msg != NULL);
+ }
+ }
}
+ /* General processing: emit line to the TokenizedAuthLine */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
+
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -950,6 +1002,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(tok_line->file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -2298,6 +2351,67 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..0b6589a7b9 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,6 +9,7 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
+# include FILE
# local DATABASE USER METHOD [OPTIONS]
# host DATABASE USER ADDRESS METHOD [OPTIONS]
# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
@@ -18,7 +19,12 @@
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include", it's not a mapping record but a directive to
+# include records from another file, specified in the field. FILE is the file
+# to include. It can be specified with a relative or absolute path, and can be
+# double quoted if it contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..138359cf03 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,18 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
+# include FILE
# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include", it's not an authentication record but a
+# directive to include records from another file, specified in the field. FILE
+# is the file to include. It can be specified with a relative or absolute
+# path, and can be double quoted if it contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index 75e69383c2..ee12dc1893 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,9 +26,11 @@
static ArrayType *gethba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
+ int mapping_number, const char *filename,
int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
@@ -157,7 +159,7 @@ gethba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 9
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line: build one row of pg_hba_file_rules view, add it to tuplestore
@@ -173,7 +175,8 @@ gethba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg)
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
+ const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
bool nulls[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -192,6 +195,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* rule_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -335,7 +345,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -357,6 +367,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *hba_lines = NIL;
ListCell *line;
+ int rule_number = 0;
MemoryContext linecxt;
MemoryContext hbacxt;
MemoryContext oldcxt;
@@ -391,8 +402,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
hbaline = parse_hba_line(tok_line, DEBUG3);
- fill_hba_line(tuple_store, tupdesc, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ rule_number++;
+
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -426,7 +441,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
}
/* Number of columns in pg_hba_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -443,7 +458,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, IdentLine *ident, const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -456,6 +472,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* mapping_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -468,7 +491,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -490,6 +513,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *ident_lines = NIL;
ListCell *line;
+ int mapping_number = 0;
MemoryContext linecxt;
MemoryContext identcxt;
MemoryContext oldcxt;
@@ -524,8 +548,13 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
identline = parse_ident_line(tok_line, DEBUG3);
- fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline,
- tok_line->err_msg);
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ mapping_number++;
+
+ fill_ident_line(tuple_store, tupdesc, mapping_number,
+ tok_line->file_name, tok_line->line_num, identline,
+ tok_line->err_msg);
}
/* Free tokenizer memory */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 6ccbc9af4c..0e8a589302 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6111,16 +6111,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '9556', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,text,text,text}',
- proargmodes => '{o,o,o,o,o}',
- proargnames => '{line_number,map_name,sys_name,pg_usernamee,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_usernamee,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index fce7db248b..551e961585 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -153,6 +154,7 @@ typedef struct IdentLine
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of HbaTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 76a209b717..5b7a4c01ee 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname,
WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
-pg_hba_file_rules| SELECT a.line_number,
+pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
+ a.line_number,
a.type,
a.database,
a.user_name,
@@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error);
-pg_ident_file_mappings| SELECT a.line_number,
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
+ a.line_number,
a.map_name,
a.sys_name,
a.pg_usernamee,
a.error
- FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_usernamee, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_usernamee, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.33.1
--45a4to7ul53bhaf6
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v3-0004-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v7 1/2] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-02-21 07:45 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw)
Catversion is bumped.
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/catalogs.sgml | 48 +++-
doc/src/sgml/client-auth.sgml | 48 +++-
src/backend/libpq/hba.c | 343 +++++++++++++++++++++----
src/backend/libpq/pg_hba.conf.sample | 10 +-
src/backend/libpq/pg_ident.conf.sample | 12 +-
src/backend/utils/adt/hbafuncs.c | 51 +++-
src/include/catalog/pg_proc.dat | 11 +-
src/include/libpq/hba.h | 2 +
src/test/regress/expected/rules.out | 12 +-
9 files changed, 453 insertions(+), 84 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index a533a2153e..b44e62d388 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -10554,12 +10554,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>rule_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this rule if the rule is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule in the given file_name
</para></entry>
</row>
@@ -10694,6 +10713,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>mapping_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this mapping if the mapping is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this mapping
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>line_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Line number of this mapping in the given file_name
+ </para></entry>
+ </row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index b2a459fb0d..f7d871e660 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,21 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> or <literal>include_dir</literal> directive and
+ the file or directory to be included. The file or directory can be a
+ relative of absolute path, and can be double quoted if needed. For the
+ <literal>include_dir</literal> form, all files not starting with a
+ <literal>.</literal> and ending with <literal>.conf</literal> will be
+ included.
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,6 +116,8 @@
<para>
A record can have several formats:
<synopsis>
+include <replaceable>file</replaceable>
+include_dir <replaceable>directory</replaceable>
local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
@@ -118,6 +133,26 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_dir</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of all the files found in
+ the directory, if they don't start with a <literal>.</literal> and end
+ with <literal>.conf</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -835,8 +870,10 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
+<replaceable>include_dir</replaceable> <replaceable>directory</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -847,6 +884,11 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ As for <filename>pg_hba.conf</filename>, the lines in this file can either
+ be inclusion directives or an authentication rule, and follow the same
+ rules.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index 327a4b42af..56b6cec9d5 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -21,6 +21,7 @@
#include <fcntl.h>
#include <sys/param.h>
#include <sys/socket.h>
+#include <sys/stat.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
@@ -68,6 +69,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -112,10 +119,19 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
const char *inc_filename, int elevel, char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
+static char *process_included_authfile(const char *inc_filename,
+ const char *outer_filename, int elevel,
+ MemoryContext linecxt, List **tok_lines);
/*
@@ -355,36 +371,11 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
- {
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
- }
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename,
+ elevel, err_msg, &inc_fullname);
- inc_file = AllocateFile(inc_fullname, "r");
if (inc_file == NULL)
- {
- int save_errno = errno;
-
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
return tokens;
- }
/* There is possible recursion here if the file contains @ */
linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
@@ -425,11 +416,36 @@ tokenize_inc_file(List *tokens,
/*
* tokenize_auth_file
- * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a decicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel);
+
+ return linecxt;
+}
+
+/*
+ * Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see the struct definition
* in libpq/hba.h.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -438,30 +454,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -522,29 +530,151 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ AuthToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
+
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, filename,
+ elevel, linecxt, tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
+ else if (strcmp(first->string, "include_dir") == 0)
+ {
+ char *dir_name;
+ DIR *d;
+ struct dirent *de;
+ StringInfoData err_buf;
+
+ dir_name = second->string;
+ d = AllocateDir(dir_name);
+ if (d == NULL)
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not open directory \"%s\": %m",
+ dir_name)));
+ err_msg = psprintf("coud not open directory \"%s\": %m",
+ dir_name);
+ goto process_line;
+ }
+
+ initStringInfo(&err_buf);
+ while ((de = ReadDir(d, dir_name)) != NULL)
+ {
+ struct stat st;
+ char inc_filename[MAXPGPATH];
+
+ /*
+ * Only parse files with names ending in ".conf".
+ * Explicitly reject files starting with ".". This
+ * excludes things like "." and "..", as well as typical
+ * hidden files, backup files, and editor debris.
+ */
+ if (strlen(de->d_name) < 6)
+ continue;
+ if (de->d_name[0] == '.')
+ continue;
+ if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
+ continue;
+
+ join_path_components(inc_filename, dir_name, de->d_name);
+ canonicalize_path(inc_filename);
+ if (stat(inc_filename, &st) == 0)
+ {
+ /* Ignore directories. */
+ if (S_ISDIR(st.st_mode))
+ continue;
+
+ /*
+ * err_msg is used here as a temp buffer, it will be
+ * overwritten at the end of the loop with the
+ * cumulated errors, if any.
+ */
+ err_msg = process_included_authfile(inc_filename,
+ filename, elevel, linecxt,
+ tok_lines);
+
+ /* Cumulate errors if any. */
+ if (err_msg)
+ {
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfoString(&err_buf, err_msg);
+ }
+ }
+ else
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not stat file \"%s\": %m",
+ filename)));
+
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfo(&err_buf,
+ "could not stat file \"%s\": %m",
+ inc_filename);
+ }
+ }
+ FreeDir(d);
+
+ /*
+ * If there were no errors, the line is fully processed, bypass
+ * the general TokenizedAuthLine processing.
+ */
+ if (err_buf.len == 0)
+ goto next_line;
+
+ /* Otherwise, process the cumulated errors, if any. */
+ err_msg = err_buf.data;
+ }
}
+process_line:
+ /*
+ * General processing: report the error if any and emit line to the
+ * TokenizedAuthLine
+ */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
+
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -859,7 +989,7 @@ do { \
errmsg("authentication option \"%s\" is only valid for authentication methods %s", \
optname, _(validmethods)), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \
optname, validmethods); \
return false; \
@@ -879,7 +1009,7 @@ do { \
errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname); \
return NULL; \
@@ -901,7 +1031,7 @@ do { \
ereport(elevel, \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("missing entry in file \"%s\" at end of line %d", \
- IdentFileName, line_num))); \
+ tok_line->file_name, line_num))); \
*err_msg = psprintf("missing entry at end of line"); \
return NULL; \
} \
@@ -914,7 +1044,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("multiple values in ident field"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, tok_line->file_name))); \
*err_msg = psprintf("multiple values in ident field"); \
return NULL; \
} \
@@ -937,6 +1067,7 @@ HbaLine *
parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
{
int line_num = tok_line->line_num;
+ char *file_name = tok_line->file_name;
char **err_msg = &tok_line->err_msg;
char *str;
struct addrinfo *gai_result;
@@ -951,6 +1082,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -1675,6 +1807,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg)
{
int line_num = hbaline->linenumber;
+ char *file_name = hbaline->sourcefile;
#ifdef USE_LDAP
hbaline->ldapscope = LDAP_SCOPE_SUBTREE;
@@ -2296,6 +2429,102 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
+
+/*
+ * Try to open an included file, and tokenize it using the given context.
+ */
+static char *
+process_included_authfile(const char *inc_filename, const char *outer_filename,
+ int elevel, MemoryContext linecxt, List **tok_lines)
+{
+ char *inc_fullname;
+ FILE *inc_file;
+ char *err_msg = NULL;
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename, outer_filename,
+ elevel, &err_msg, &inc_fullname);
+
+ if (inc_file == NULL)
+ {
+ /* open_inc_file should have reported an error. */
+ Assert(err_msg != NULL);
+ return err_msg;
+ }
+ else
+ {
+ /* No error message should have been reported. */
+ Assert(err_msg == NULL);
+ }
+
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ return NULL;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..0050e24186 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,6 +9,8 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
+# include FILE
+# include_dir DIRECTORY
# local DATABASE USER METHOD [OPTIONS]
# host DATABASE USER ADDRESS METHOD [OPTIONS]
# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
@@ -18,7 +20,13 @@
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include" or "include_dir", it's not a mapping record but a directive to
+# include records from other file(s), as specified in the field. FILE is the
+# file to include, and DIR is the directory containing the file(s) to include
+# It can be specified with a relative or absolute path, and can be double
+# quoted if it contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..9b700d9989 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,20 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
-# MAPNAME SYSTEM-USERNAME PG-USERNAME
+# include FILE
+# include_dir FILE
+# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include" or "include_dir", it's not an authentication
+# record but a directive to include records from other file(s), specified in
+# the field. FILE is the file to include and DIR is the directory containing
+# the file(s) to include. It can be specified with a relative or absolute
+# path, and can be double quoted if it contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index 9fe7b62c9a..4d902927e0 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,9 +26,11 @@
static ArrayType *get_hba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
+ int mapping_number, const char *filename,
int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
@@ -157,7 +159,7 @@ get_hba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 9
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line
@@ -174,7 +176,8 @@ get_hba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg)
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
+ const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
bool nulls[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -193,6 +196,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* rule_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -336,7 +346,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -359,6 +369,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *hba_lines = NIL;
ListCell *line;
+ int rule_number = 0;
MemoryContext linecxt;
MemoryContext hbacxt;
MemoryContext oldcxt;
@@ -393,8 +404,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
hbaline = parse_hba_line(tok_line, DEBUG3);
- fill_hba_line(tuple_store, tupdesc, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ rule_number++;
+
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -430,8 +445,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
PG_RETURN_NULL();
}
-/* Number of columns in pg_ident_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5
+/* Number of columns in pg_hba_file_mappings view */
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -448,7 +463,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, IdentLine *ident, const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -461,6 +477,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* mapping_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -473,7 +496,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -495,6 +518,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *ident_lines = NIL;
ListCell *line;
+ int mapping_number = 0;
MemoryContext linecxt;
MemoryContext identcxt;
MemoryContext oldcxt;
@@ -529,7 +553,12 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
identline = parse_ident_line(tok_line, DEBUG3);
- fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline,
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ mapping_number++;
+
+ fill_ident_line(tuple_store, tupdesc, mapping_number,
+ tok_line->file_name, tok_line->line_num, identline,
tok_line->err_msg);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index babe16f00a..65bfd32753 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6128,15 +6128,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '6250', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,text,text,text}', proargmodes => '{o,o,o,o,o}',
- proargnames => '{line_number,map_name,sys_name,pg_username,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index 90036f7bcd..59f6faf9f8 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -155,6 +156,7 @@ typedef struct AuthToken
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of AuthTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 21effe8315..11ffe1743b 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname,
WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
-pg_hba_file_rules| SELECT a.line_number,
+pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
+ a.line_number,
a.type,
a.database,
a.user_name,
@@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error);
-pg_ident_file_mappings| SELECT a.line_number,
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
+ a.line_number,
a.map_name,
a.sys_name,
a.pg_username,
a.error
- FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_username, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.33.1
--o45jk7yo2ulu7ifa
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v2 3/4] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-02-21 07:45 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw)
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/catalogs.sgml | 48 +++-
doc/src/sgml/client-auth.sgml | 34 ++-
src/backend/libpq/hba.c | 373 ++++++++++++++++---------
src/backend/libpq/pg_hba.conf.sample | 8 +-
src/backend/libpq/pg_ident.conf.sample | 8 +-
src/backend/utils/adt/hbafuncs.c | 53 +++-
src/include/catalog/pg_proc.dat | 12 +-
src/include/libpq/hba.h | 2 +
src/test/regress/expected/rules.out | 12 +-
9 files changed, 395 insertions(+), 155 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 0de40a9626..07c6679a52 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -10430,12 +10430,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>rule_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this rule if the rule is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule in the given file_name
</para></entry>
</row>
@@ -10571,6 +10590,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>mapping_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this mapping if the mapping is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this mapping
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>line_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Line number of this mapping in the given file_name
+ </para></entry>
+ </row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index 142b0affcb..e1d0e103b3 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,17 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> directive and the file to be included. The file
+ can be a relative of absolute path, and can be double quoted if needed.
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,6 +112,7 @@
<para>
A record can have several formats:
<synopsis>
+include <replaceable>file</replaceable>
local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
@@ -118,6 +128,15 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -835,8 +854,9 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -847,6 +867,14 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ The lines can record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> directive and the file to be included. The file
+ can be a relative of absolute path, and can be double quoted if needed.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index e673ff5474..85988fdeda 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -68,6 +68,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -112,6 +118,12 @@ static const char *const UserAuthName[] =
};
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
+static void tokenize_auth_file_with_context(MemoryContext Linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
const char *inc_filename, int elevel, char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
@@ -327,6 +339,68 @@ next_field_expand(const char *filename, char **lineptr,
return tokens;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+*/
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
+
/*
* tokenize_inc_file
* Expand a file included from another file into an hba "field"
@@ -355,36 +429,11 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
- {
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
- }
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename,
+ elevel, err_msg, &inc_fullname);
- inc_file = AllocateFile(inc_fullname, "r");
if (inc_file == NULL)
- {
- int save_errno = errno;
-
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
return tokens;
- }
/* There is possible recursion here if the file contains @ */
linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
@@ -423,6 +472,169 @@ tokenize_inc_file(List *tokens,
return tokens;
}
+/*
+ * Tokenize the given file.
+ *
+ * The output is a list of TokenizedAuthLine structs; see struct definition
+ * above.
+ *
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
+ * filename: the absolute path to the target file
+ * file: the already-opened target file
+ * tok_lines: receives output list
+ * elevel: message logging level
+ *
+ * Errors are reported by logging messages at ereport level elevel and by
+ * adding TokenizedAuthLine structs containing non-null err_msg fields to the
+ * output list.
+ *
+ */
+static void
+tokenize_auth_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int elevel)
+{
+ StringInfoData buf;
+ int line_number = 1;
+ MemoryContext oldcxt;
+
+ oldcxt = MemoryContextSwitchTo(linecxt);
+
+ initStringInfo(&buf);
+
+ while (!feof(file) && !ferror(file))
+ {
+ TokenizedAuthLine *tok_line;
+ char *lineptr;
+ List *current_line = NIL;
+ char *err_msg = NULL;
+ int last_backslash_buflen = 0;
+ int continuations = 0;
+
+ /* Collect the next input line, handling backslash continuations */
+ resetStringInfo(&buf);
+
+ while (pg_get_line_append(file, &buf, NULL))
+ {
+ /* Strip trailing newline, including \r in case we're on Windows */
+ buf.len = pg_strip_crlf(buf.data);
+
+ /*
+ * Check for backslash continuation. The backslash must be after
+ * the last place we found a continuation, else two backslashes
+ * followed by two \n's would behave surprisingly.
+ */
+ if (buf.len > last_backslash_buflen &&
+ buf.data[buf.len - 1] == '\\')
+ {
+ /* Continuation, so strip it and keep reading */
+ buf.data[--buf.len] = '\0';
+ last_backslash_buflen = buf.len;
+ continuations++;
+ continue;
+ }
+
+ /* Nope, so we have the whole line */
+ break;
+ }
+
+ if (ferror(file))
+ {
+ /* I/O error! */
+ int save_errno = errno;
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m", filename)));
+ err_msg = psprintf("could not read file \"%s\": %s",
+ filename, strerror(save_errno));
+ break;
+ }
+
+ /* Parse fields */
+ lineptr = buf.data;
+ while (*lineptr && err_msg == NULL)
+ {
+ List *current_field;
+
+ current_field = next_field_expand(filename, &lineptr,
+ elevel, &err_msg);
+ /* add field to line, unless we are at EOL or comment start */
+ if (current_field != NIL)
+ current_line = lappend(current_line, current_field);
+ }
+
+ /*
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
+ */
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
+ {
+ HbaToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
+
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
+ char *inc_fullname;
+ FILE *inc_file;
+
+ inc_filename = second->string;
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename,
+ filename, elevel, &err_msg,
+ &inc_fullname);
+
+ /*
+ * The included file could be open, now recursively process it.
+ * Errors will be reported in the general TokenizedAuthLine
+ * processing.
+ */
+ if (inc_file != NULL)
+ {
+ tokenize_auth_file_with_context(linecxt, inc_fullname,
+ inc_file, tok_lines,
+ elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ /*
+ * The line is fulle processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ else
+ {
+ /* We should got an error */
+ Assert(err_msg != NULL);
+ }
+ }
+ }
+
+ /* General processing: Emit line to the TokenizedAuthLine */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
+ line_number += continuations + 1;
+
+ }
+
+ MemoryContextSwitchTo(oldcxt);
+}
/*
* Does user belong to role?
@@ -1796,6 +2008,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(tok_line->file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -2602,118 +2815,26 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel)
/*
* Tokenize the given file.
*
- * The output is a list of TokenizedAuthLine structs; see struct definition
- * above.
+ * Wrapper around tokenize_auth_file_with_context, creating a dedicated memory
+ * context.
*
- * filename: the absolute path to the target file
- * file: the already-opened target file
- * tok_lines: receives output list
- * elevel: message logging level
- *
- * Errors are reported by logging messages at ereport level elevel and by
- * adding TokenizedAuthLine structs containing non-null err_msg fields to the
- * output list.
- *
- * Return value is a memory context which contains all memory allocated by
+ * Return value is this memory context which contains all memory allocated by
* this function (it's a child of caller's context).
*/
MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel)
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
+ int elevel)
{
- int line_number = 1;
- StringInfoData buf;
MemoryContext linecxt;
- MemoryContext oldcxt;
linecxt = AllocSetContextCreate(CurrentMemoryContext,
"tokenize_auth_file",
ALLOCSET_SMALL_SIZES);
- oldcxt = MemoryContextSwitchTo(linecxt);
-
- initStringInfo(&buf);
*tok_lines = NIL;
- while (!feof(file) && !ferror(file))
- {
- char *lineptr;
- List *current_line = NIL;
- char *err_msg = NULL;
- int last_backslash_buflen = 0;
- int continuations = 0;
-
- /* Collect the next input line, handling backslash continuations */
- resetStringInfo(&buf);
-
- while (pg_get_line_append(file, &buf, NULL))
- {
- /* Strip trailing newline, including \r in case we're on Windows */
- buf.len = pg_strip_crlf(buf.data);
-
- /*
- * Check for backslash continuation. The backslash must be after
- * the last place we found a continuation, else two backslashes
- * followed by two \n's would behave surprisingly.
- */
- if (buf.len > last_backslash_buflen &&
- buf.data[buf.len - 1] == '\\')
- {
- /* Continuation, so strip it and keep reading */
- buf.data[--buf.len] = '\0';
- last_backslash_buflen = buf.len;
- continuations++;
- continue;
- }
-
- /* Nope, so we have the whole line */
- break;
- }
-
- if (ferror(file))
- {
- /* I/O error! */
- int save_errno = errno;
-
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m", filename)));
- err_msg = psprintf("could not read file \"%s\": %s",
- filename, strerror(save_errno));
- break;
- }
-
- /* Parse fields */
- lineptr = buf.data;
- while (*lineptr && err_msg == NULL)
- {
- List *current_field;
-
- current_field = next_field_expand(filename, &lineptr,
- elevel, &err_msg);
- /* add field to line, unless we are at EOL or comment start */
- if (current_field != NIL)
- current_line = lappend(current_line, current_field);
- }
-
- /*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
- * */
- if (current_line != NIL || err_msg != NULL)
- {
- TokenizedAuthLine *tok_line;
-
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
- }
-
- line_number += continuations + 1;
- }
-
- MemoryContextSwitchTo(oldcxt);
+ tokenize_auth_file_with_context(linecxt, filename, file, tok_lines,
+ elevel);
return linecxt;
}
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..0b6589a7b9 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,6 +9,7 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
+# include FILE
# local DATABASE USER METHOD [OPTIONS]
# host DATABASE USER ADDRESS METHOD [OPTIONS]
# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
@@ -18,7 +19,12 @@
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include", it's not a mapping record but a directive to
+# include records from another file, specified in the field. FILE is the file
+# to include. It can be specified with a relative or absolute path, and can be
+# double quoted if it contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..138359cf03 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,18 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
+# include FILE
# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include", it's not an authentication record but a
+# directive to include records from another file, specified in the field. FILE
+# is the file to include. It can be specified with a relative or absolute
+# path, and can be double quoted if it contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index 2189c298b8..4c9df29427 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,16 +26,18 @@
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg);
+ int tule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
+ int mappint_number, const char *filename,
int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static ArrayType *gethba_options(HbaLine *hba);
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 9
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line: build one row of pg_hba_file_rules view, add it to tuplestore
@@ -51,7 +53,8 @@ static ArrayType *gethba_options(HbaLine *hba);
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg)
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
+ const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
bool nulls[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -70,6 +73,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* rule_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -213,7 +223,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -235,6 +245,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *hba_lines = NIL;
ListCell *line;
+ int rule_number = 0;
MemoryContext linecxt;
MemoryContext hbacxt;
MemoryContext oldcxt;
@@ -265,12 +276,15 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
TokenizedAuthLine *tok_line = (TokenizedAuthLine *) lfirst(line);
HbaLine *hbaline = NULL;
- /* don't parse lines that already have errors */
+ /* Only parse lines that don't already have an error */
if (tok_line->err_msg == NULL)
+ {
hbaline = parse_hba_line(tok_line, DEBUG3);
+ rule_number++;
+ }
- fill_hba_line(tuple_store, tupdesc, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -281,7 +295,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
}
/* Number of columns in pg_hba_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -298,7 +312,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, IdentLine *ident, const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -311,6 +326,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* mapping_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -323,7 +345,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -345,6 +367,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *ident_lines = NIL;
ListCell *line;
+ int mapping_number = 0;
MemoryContext linecxt;
MemoryContext identcxt;
MemoryContext oldcxt;
@@ -375,12 +398,16 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
TokenizedAuthLine *tok_line = (TokenizedAuthLine *) lfirst(line);
IdentLine *identline = NULL;
- /* don't parse lines that already have errors */
+ /* Only parse lines that don't already have an error */
if (tok_line->err_msg == NULL)
+ {
identline = parse_ident_line(tok_line, DEBUG3);
+ mapping_number++;
+ }
- fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline,
- tok_line->err_msg);
+ fill_ident_line(tuple_store, tupdesc, mapping_number,
+ tok_line->file_name, tok_line->line_num, identline,
+ tok_line->err_msg);
}
/* Free tokenizer memory */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 04d4edb228..e3baafc3e3 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6116,16 +6116,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '9556', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,text,text,text}',
- proargmodes => '{o,o,o,o,o}',
- proargnames => '{line_number,map_name,sys_name,pg_usernamee,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_usernamee,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index fce7db248b..551e961585 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -153,6 +154,7 @@ typedef struct IdentLine
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of HbaTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62cf0d8674..e6f274cb59 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname,
WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
-pg_hba_file_rules| SELECT a.line_number,
+pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
+ a.line_number,
a.type,
a.database,
a.user_name,
@@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error);
-pg_ident_file_mappings| SELECT a.line_number,
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
+ a.line_number,
a.map_name,
a.sys_name,
a.pg_usernamee,
a.error
- FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_usernamee, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_usernamee, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.33.1
--uqjaptwdgaq4xou6
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v2-0004-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v1 2/3] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-02-21 07:45 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw)
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: FIXME
---
doc/src/sgml/catalogs.sgml | 48 ++++-
doc/src/sgml/client-auth.sgml | 34 +++-
src/backend/libpq/hba.c | 252 +++++++++++++++++++------
src/backend/libpq/pg_hba.conf.sample | 8 +-
src/backend/libpq/pg_ident.conf.sample | 8 +-
src/include/catalog/pg_proc.dat | 12 +-
src/include/libpq/hba.h | 1 +
src/test/regress/expected/rules.out | 12 +-
8 files changed, 304 insertions(+), 71 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 0de40a9626..07c6679a52 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -10430,12 +10430,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>rule_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this rule if the rule is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule in the given file_name
</para></entry>
</row>
@@ -10571,6 +10590,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>mapping_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this mapping if the mapping is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this mapping
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>line_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Line number of this mapping in the given file_name
+ </para></entry>
+ </row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index 142b0affcb..e1d0e103b3 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,17 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> directive and the file to be included. The file
+ can be a relative of absolute path, and can be double quoted if needed.
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,6 +112,7 @@
<para>
A record can have several formats:
<synopsis>
+include <replaceable>file</replaceable>
local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
@@ -118,6 +128,15 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -835,8 +854,9 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -847,6 +867,14 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ The lines can record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> directive and the file to be included. The file
+ can be a relative of absolute path, and can be double quoted if needed.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index 9a9ce96fee..8b72141342 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -89,11 +89,18 @@ typedef struct HbaToken
typedef struct TokenizedLine
{
List *fields; /* List of lists of HbaTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
} TokenizedLine;
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -140,17 +147,25 @@ static const char *const UserAuthName[] =
static MemoryContext tokenize_file(const char *filename, FILE *file,
List **tok_lines, int elevel);
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
const char *inc_filename, int elevel, char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
static ArrayType *gethba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
+ int mapping_number, const char *filename,
int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
/*
@@ -390,36 +405,11 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
- {
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
- }
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename,
+ elevel, err_msg, &inc_fullname);
- inc_file = AllocateFile(inc_fullname, "r");
if (inc_file == NULL)
- {
- int save_errno = errno;
-
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
return tokens;
- }
/* There is possible recursion here if the file contains @ */
linecxt = tokenize_file(inc_fullname, inc_file, &inc_lines, elevel);
@@ -458,11 +448,37 @@ tokenize_inc_file(List *tokens,
return tokens;
}
+/*
+ * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a decicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+static MemoryContext
+tokenize_file(const char *filename, FILE *file, List **tok_lines, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel);
+
+ return linecxt;
+}
+
/*
* Tokenize the given file.
*
* The output is a list of TokenizedLine structs; see struct definition above.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -471,27 +487,19 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-static MemoryContext
-tokenize_file(const char *filename, FILE *file, List **tok_lines, int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
char *lineptr;
@@ -553,28 +561,76 @@ tokenize_file(const char *filename, FILE *file, List **tok_lines, int elevel)
current_line = lappend(current_line, current_field);
}
- /* Reached EOL; emit line to TokenizedLine list unless it's boring */
- if (current_line != NIL || err_msg != NULL)
+ /*
+ * Reached EOL; no need to emit line to TokenizedLine list if it's
+ * boring
+ */
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
+ {
+ HbaToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
+
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
+ char *inc_fullname;
+ FILE *inc_file;
+
+ inc_filename = second->string;
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename,
+ filename, elevel, &err_msg,
+ &inc_fullname);
+
+ /*
+ * We could open the file, recursively process it. Errors will
+ * be reported in the general TokenizedLine processing.
+ */
+ if (inc_file != NULL)
+ {
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ goto next_line;
+ }
+ else
+ {
+ /* We should got an error */
+ Assert(err_msg != NULL);
+ }
+ }
+ }
+
+ /* Emit line to TokenizedLine */
{
TokenizedLine *tok_line;
tok_line = (TokenizedLine *) palloc(sizeof(TokenizedLine));
tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
tok_line->line_num = line_number;
tok_line->raw_line = pstrdup(buf.data);
tok_line->err_msg = err_msg;
*tok_lines = lappend(*tok_lines, tok_line);
}
+next_line:
line_number += continuations + 1;
+
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -981,6 +1037,7 @@ parse_hba_line(TokenizedLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(tok_line->file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -2453,7 +2510,7 @@ gethba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 9
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line: build one row of pg_hba_file_rules view, add it to tuplestore
@@ -2469,7 +2526,8 @@ gethba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg)
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
+ const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
bool nulls[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -2488,6 +2546,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* rule_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -2631,7 +2696,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -2653,6 +2718,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *hba_lines = NIL;
ListCell *line;
+ int rule_number = 0;
MemoryContext linecxt;
MemoryContext hbacxt;
MemoryContext oldcxt;
@@ -2687,8 +2753,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
hbaline = parse_hba_line(tok_line, DEBUG3);
- fill_hba_line(tuple_store, tupdesc, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ rule_number++;
+
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -2698,6 +2768,64 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
MemoryContextDelete(hbacxt);
}
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
+
/*
* SQL-accessible SRF to return all the entries in the pg_hba.conf file.
*/
@@ -3172,7 +3300,7 @@ hba_authname(UserAuth auth_method)
}
/* Number of columns in pg_hba_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -3189,7 +3317,8 @@ hba_authname(UserAuth auth_method)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, IdentLine *ident, const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -3202,6 +3331,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* mapping_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -3214,7 +3350,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -3236,6 +3372,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *ident_lines = NIL;
ListCell *line;
+ int mapping_number = 0;
MemoryContext linecxt;
MemoryContext hbacxt;
MemoryContext oldcxt;
@@ -3270,8 +3407,13 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
identline = parse_ident_line(tok_line, DEBUG3);
- fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline,
- tok_line->err_msg);
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ mapping_number++;
+
+ fill_ident_line(tuple_store, tupdesc, mapping_number,
+ tok_line->file_name, tok_line->line_num, identline,
+ tok_line->err_msg);
}
/* Free tokenizer memory */
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..0b6589a7b9 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,6 +9,7 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
+# include FILE
# local DATABASE USER METHOD [OPTIONS]
# host DATABASE USER ADDRESS METHOD [OPTIONS]
# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
@@ -18,7 +19,12 @@
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include", it's not a mapping record but a directive to
+# include records from another file, specified in the field. FILE is the file
+# to include. It can be specified with a relative or absolute path, and can be
+# double quoted if it contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..138359cf03 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,18 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
+# include FILE
# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include", it's not an authentication record but a
+# directive to include records from another file, specified in the field. FILE
+# is the file to include. It can be specified with a relative or absolute
+# path, and can be double quoted if it contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 2c8f5d9c13..2292115c85 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6117,16 +6117,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '9556', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,text,text,text}',
- proargmodes => '{o,o,o,o,o}',
- proargnames => '{line_number,map_name,sys_name,pg_usernamee,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_usernamee,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index 8d9f3821b1..45d6ce1f22 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62cf0d8674..e6f274cb59 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname,
WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
-pg_hba_file_rules| SELECT a.line_number,
+pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
+ a.line_number,
a.type,
a.database,
a.user_name,
@@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error);
-pg_ident_file_mappings| SELECT a.line_number,
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
+ a.line_number,
a.map_name,
a.sys_name,
a.pg_usernamee,
a.error
- FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_usernamee, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_usernamee, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.33.1
--j3tmyqztg67irqsk
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v1-0003-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v5 2/3] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-02-21 07:45 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw)
Catversion is bumped.
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/catalogs.sgml | 48 +++++-
doc/src/sgml/client-auth.sgml | 34 +++-
src/backend/libpq/hba.c | 229 +++++++++++++++++++------
src/backend/libpq/pg_hba.conf.sample | 8 +-
src/backend/libpq/pg_ident.conf.sample | 8 +-
src/backend/utils/adt/hbafuncs.c | 53 ++++--
src/include/catalog/pg_proc.dat | 12 +-
src/include/libpq/hba.h | 2 +
src/test/regress/expected/rules.out | 12 +-
9 files changed, 321 insertions(+), 85 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 75fedfa07e..bd1c9a8d21 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -10496,12 +10496,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>rule_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this rule if the rule is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule in the given file_name
</para></entry>
</row>
@@ -10637,6 +10656,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>mapping_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this mapping if the mapping is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this mapping
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>line_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Line number of this mapping in the given file_name
+ </para></entry>
+ </row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index 142b0affcb..e1d0e103b3 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,17 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> directive and the file to be included. The file
+ can be a relative of absolute path, and can be double quoted if needed.
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,6 +112,7 @@
<para>
A record can have several formats:
<synopsis>
+include <replaceable>file</replaceable>
local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
@@ -118,6 +128,15 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -835,8 +854,9 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -847,6 +867,14 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ The lines can record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> directive and the file to be included. The file
+ can be a relative of absolute path, and can be double quoted if needed.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index 556f473b41..1551b34c53 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -68,6 +68,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -112,10 +118,16 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
const char *inc_filename, int elevel, char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
/*
@@ -355,36 +367,11 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
- {
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
- }
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename,
+ elevel, err_msg, &inc_fullname);
- inc_file = AllocateFile(inc_fullname, "r");
if (inc_file == NULL)
- {
- int save_errno = errno;
-
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
return tokens;
- }
/* There is possible recursion here if the file contains @ */
linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
@@ -425,11 +412,36 @@ tokenize_inc_file(List *tokens,
/*
* tokenize_auth_file
- * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a decicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel);
+
+ return linecxt;
+}
+
+/*
+ * Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see the struct definition
* in libpq/hba.h.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -438,30 +450,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -522,29 +526,76 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ AuthToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
+ char *inc_fullname;
+ FILE *inc_file;
+
+ inc_filename = second->string;
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename,
+ filename, elevel, &err_msg,
+ &inc_fullname);
+
+ /*
+ * The included file could be open, now recursively process it.
+ * Errors will be reported in the general TokenizedAuthLine
+ * processing.
+ */
+ if (inc_file != NULL)
+ {
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ else
+ {
+ /* We should got an error */
+ Assert(err_msg != NULL);
+ }
+ }
}
+ /* General processing: emit line to the TokenizedAuthLine */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
+
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -859,7 +910,7 @@ do { \
errmsg("authentication option \"%s\" is only valid for authentication methods %s", \
optname, _(validmethods)), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \
optname, validmethods); \
return false; \
@@ -879,7 +930,7 @@ do { \
errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname); \
return NULL; \
@@ -901,7 +952,7 @@ do { \
ereport(elevel, \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("missing entry in file \"%s\" at end of line %d", \
- IdentFileName, line_num))); \
+ tok_line->file_name, line_num))); \
*err_msg = psprintf("missing entry at end of line"); \
return NULL; \
} \
@@ -914,7 +965,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("multiple values in ident field"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, tok_line->file_name))); \
*err_msg = psprintf("multiple values in ident field"); \
return NULL; \
} \
@@ -937,6 +988,7 @@ HbaLine *
parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
{
int line_num = tok_line->line_num;
+ char *file_name = tok_line->file_name;
char **err_msg = &tok_line->err_msg;
char *str;
struct addrinfo *gai_result;
@@ -951,6 +1003,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -1677,6 +1730,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg)
{
int line_num = hbaline->linenumber;
+ char *file_name = hbaline->sourcefile;
#ifdef USE_LDAP
hbaline->ldapscope = LDAP_SCOPE_SUBTREE;
@@ -2299,6 +2353,67 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..0b6589a7b9 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,6 +9,7 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
+# include FILE
# local DATABASE USER METHOD [OPTIONS]
# host DATABASE USER ADDRESS METHOD [OPTIONS]
# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
@@ -18,7 +19,12 @@
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include", it's not a mapping record but a directive to
+# include records from another file, specified in the field. FILE is the file
+# to include. It can be specified with a relative or absolute path, and can be
+# double quoted if it contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..138359cf03 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,18 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
+# include FILE
# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include", it's not an authentication record but a
+# directive to include records from another file, specified in the field. FILE
+# is the file to include. It can be specified with a relative or absolute
+# path, and can be double quoted if it contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index 1970b4c497..33cf5fb954 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,9 +26,11 @@
static ArrayType *get_hba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
+ int mapping_number, const char *filename,
int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
@@ -157,7 +159,7 @@ get_hba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 9
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line
@@ -174,7 +176,8 @@ get_hba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg)
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
+ const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
bool nulls[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -193,6 +196,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* rule_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -336,7 +346,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -359,6 +369,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *hba_lines = NIL;
ListCell *line;
+ int rule_number = 0;
MemoryContext linecxt;
MemoryContext hbacxt;
MemoryContext oldcxt;
@@ -393,8 +404,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
hbaline = parse_hba_line(tok_line, DEBUG3);
- fill_hba_line(tuple_store, tupdesc, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ rule_number++;
+
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -430,8 +445,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
PG_RETURN_NULL();
}
-/* Number of columns in pg_ident_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5
+/* Number of columns in pg_hba_file_mappings view */
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -448,7 +463,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, IdentLine *ident, const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -461,6 +477,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* mapping_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -473,7 +496,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -495,6 +518,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *ident_lines = NIL;
ListCell *line;
+ int mapping_number = 0;
MemoryContext linecxt;
MemoryContext identcxt;
MemoryContext oldcxt;
@@ -529,8 +553,13 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
identline = parse_ident_line(tok_line, DEBUG3);
- fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline,
- tok_line->err_msg);
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ mapping_number++;
+
+ fill_ident_line(tuple_store, tupdesc, mapping_number,
+ tok_line->file_name, tok_line->line_num, identline,
+ tok_line->err_msg);
}
/* Free tokenizer memory */
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index efd48741d8..04cab629f5 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6111,16 +6111,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '9556', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,text,text,text}',
- proargmodes => '{o,o,o,o,o}',
- proargnames => '{line_number,map_name,sys_name,pg_username,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index 90036f7bcd..59f6faf9f8 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -155,6 +156,7 @@ typedef struct AuthToken
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of AuthTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 5a20e5a7e1..e2c6aa8ffb 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname,
WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
-pg_hba_file_rules| SELECT a.line_number,
+pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
+ a.line_number,
a.type,
a.database,
a.user_name,
@@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error);
-pg_ident_file_mappings| SELECT a.line_number,
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
+ a.line_number,
a.map_name,
a.sys_name,
a.pg_username,
a.error
- FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_username, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.33.1
--w64sw4oo7jkfb67p
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0003-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v7 1/2] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-02-21 07:45 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw)
Catversion is bumped.
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/catalogs.sgml | 48 +++-
doc/src/sgml/client-auth.sgml | 48 +++-
src/backend/libpq/hba.c | 343 +++++++++++++++++++++----
src/backend/libpq/pg_hba.conf.sample | 10 +-
src/backend/libpq/pg_ident.conf.sample | 12 +-
src/backend/utils/adt/hbafuncs.c | 51 +++-
src/include/catalog/pg_proc.dat | 11 +-
src/include/libpq/hba.h | 2 +
src/test/regress/expected/rules.out | 12 +-
9 files changed, 453 insertions(+), 84 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index a533a2153e..b44e62d388 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -10554,12 +10554,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>rule_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this rule if the rule is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule in the given file_name
</para></entry>
</row>
@@ -10694,6 +10713,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>mapping_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this mapping if the mapping is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this mapping
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>line_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Line number of this mapping in the given file_name
+ </para></entry>
+ </row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index b2a459fb0d..f7d871e660 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,21 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> or <literal>include_dir</literal> directive and
+ the file or directory to be included. The file or directory can be a
+ relative of absolute path, and can be double quoted if needed. For the
+ <literal>include_dir</literal> form, all files not starting with a
+ <literal>.</literal> and ending with <literal>.conf</literal> will be
+ included.
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,6 +116,8 @@
<para>
A record can have several formats:
<synopsis>
+include <replaceable>file</replaceable>
+include_dir <replaceable>directory</replaceable>
local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
@@ -118,6 +133,26 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_dir</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of all the files found in
+ the directory, if they don't start with a <literal>.</literal> and end
+ with <literal>.conf</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -835,8 +870,10 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
+<replaceable>include_dir</replaceable> <replaceable>directory</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -847,6 +884,11 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ As for <filename>pg_hba.conf</filename>, the lines in this file can either
+ be inclusion directives or an authentication rule, and follow the same
+ rules.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index 327a4b42af..56b6cec9d5 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -21,6 +21,7 @@
#include <fcntl.h>
#include <sys/param.h>
#include <sys/socket.h>
+#include <sys/stat.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
@@ -68,6 +69,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -112,10 +119,19 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
const char *inc_filename, int elevel, char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
+static char *process_included_authfile(const char *inc_filename,
+ const char *outer_filename, int elevel,
+ MemoryContext linecxt, List **tok_lines);
/*
@@ -355,36 +371,11 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
- {
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
- }
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename,
+ elevel, err_msg, &inc_fullname);
- inc_file = AllocateFile(inc_fullname, "r");
if (inc_file == NULL)
- {
- int save_errno = errno;
-
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
return tokens;
- }
/* There is possible recursion here if the file contains @ */
linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
@@ -425,11 +416,36 @@ tokenize_inc_file(List *tokens,
/*
* tokenize_auth_file
- * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a decicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel);
+
+ return linecxt;
+}
+
+/*
+ * Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see the struct definition
* in libpq/hba.h.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -438,30 +454,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -522,29 +530,151 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ AuthToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
+
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, filename,
+ elevel, linecxt, tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
+ else if (strcmp(first->string, "include_dir") == 0)
+ {
+ char *dir_name;
+ DIR *d;
+ struct dirent *de;
+ StringInfoData err_buf;
+
+ dir_name = second->string;
+ d = AllocateDir(dir_name);
+ if (d == NULL)
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not open directory \"%s\": %m",
+ dir_name)));
+ err_msg = psprintf("coud not open directory \"%s\": %m",
+ dir_name);
+ goto process_line;
+ }
+
+ initStringInfo(&err_buf);
+ while ((de = ReadDir(d, dir_name)) != NULL)
+ {
+ struct stat st;
+ char inc_filename[MAXPGPATH];
+
+ /*
+ * Only parse files with names ending in ".conf".
+ * Explicitly reject files starting with ".". This
+ * excludes things like "." and "..", as well as typical
+ * hidden files, backup files, and editor debris.
+ */
+ if (strlen(de->d_name) < 6)
+ continue;
+ if (de->d_name[0] == '.')
+ continue;
+ if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
+ continue;
+
+ join_path_components(inc_filename, dir_name, de->d_name);
+ canonicalize_path(inc_filename);
+ if (stat(inc_filename, &st) == 0)
+ {
+ /* Ignore directories. */
+ if (S_ISDIR(st.st_mode))
+ continue;
+
+ /*
+ * err_msg is used here as a temp buffer, it will be
+ * overwritten at the end of the loop with the
+ * cumulated errors, if any.
+ */
+ err_msg = process_included_authfile(inc_filename,
+ filename, elevel, linecxt,
+ tok_lines);
+
+ /* Cumulate errors if any. */
+ if (err_msg)
+ {
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfoString(&err_buf, err_msg);
+ }
+ }
+ else
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not stat file \"%s\": %m",
+ filename)));
+
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfo(&err_buf,
+ "could not stat file \"%s\": %m",
+ inc_filename);
+ }
+ }
+ FreeDir(d);
+
+ /*
+ * If there were no errors, the line is fully processed, bypass
+ * the general TokenizedAuthLine processing.
+ */
+ if (err_buf.len == 0)
+ goto next_line;
+
+ /* Otherwise, process the cumulated errors, if any. */
+ err_msg = err_buf.data;
+ }
}
+process_line:
+ /*
+ * General processing: report the error if any and emit line to the
+ * TokenizedAuthLine
+ */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
+
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -859,7 +989,7 @@ do { \
errmsg("authentication option \"%s\" is only valid for authentication methods %s", \
optname, _(validmethods)), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \
optname, validmethods); \
return false; \
@@ -879,7 +1009,7 @@ do { \
errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname); \
return NULL; \
@@ -901,7 +1031,7 @@ do { \
ereport(elevel, \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("missing entry in file \"%s\" at end of line %d", \
- IdentFileName, line_num))); \
+ tok_line->file_name, line_num))); \
*err_msg = psprintf("missing entry at end of line"); \
return NULL; \
} \
@@ -914,7 +1044,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("multiple values in ident field"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, tok_line->file_name))); \
*err_msg = psprintf("multiple values in ident field"); \
return NULL; \
} \
@@ -937,6 +1067,7 @@ HbaLine *
parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
{
int line_num = tok_line->line_num;
+ char *file_name = tok_line->file_name;
char **err_msg = &tok_line->err_msg;
char *str;
struct addrinfo *gai_result;
@@ -951,6 +1082,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -1675,6 +1807,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg)
{
int line_num = hbaline->linenumber;
+ char *file_name = hbaline->sourcefile;
#ifdef USE_LDAP
hbaline->ldapscope = LDAP_SCOPE_SUBTREE;
@@ -2296,6 +2429,102 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
+
+/*
+ * Try to open an included file, and tokenize it using the given context.
+ */
+static char *
+process_included_authfile(const char *inc_filename, const char *outer_filename,
+ int elevel, MemoryContext linecxt, List **tok_lines)
+{
+ char *inc_fullname;
+ FILE *inc_file;
+ char *err_msg = NULL;
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename, outer_filename,
+ elevel, &err_msg, &inc_fullname);
+
+ if (inc_file == NULL)
+ {
+ /* open_inc_file should have reported an error. */
+ Assert(err_msg != NULL);
+ return err_msg;
+ }
+ else
+ {
+ /* No error message should have been reported. */
+ Assert(err_msg == NULL);
+ }
+
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ return NULL;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..0050e24186 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,6 +9,8 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
+# include FILE
+# include_dir DIRECTORY
# local DATABASE USER METHOD [OPTIONS]
# host DATABASE USER ADDRESS METHOD [OPTIONS]
# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
@@ -18,7 +20,13 @@
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include" or "include_dir", it's not a mapping record but a directive to
+# include records from other file(s), as specified in the field. FILE is the
+# file to include, and DIR is the directory containing the file(s) to include
+# It can be specified with a relative or absolute path, and can be double
+# quoted if it contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..9b700d9989 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,20 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
-# MAPNAME SYSTEM-USERNAME PG-USERNAME
+# include FILE
+# include_dir FILE
+# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include" or "include_dir", it's not an authentication
+# record but a directive to include records from other file(s), specified in
+# the field. FILE is the file to include and DIR is the directory containing
+# the file(s) to include. It can be specified with a relative or absolute
+# path, and can be double quoted if it contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index 9fe7b62c9a..4d902927e0 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,9 +26,11 @@
static ArrayType *get_hba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
+ int mapping_number, const char *filename,
int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
@@ -157,7 +159,7 @@ get_hba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 9
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line
@@ -174,7 +176,8 @@ get_hba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg)
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
+ const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
bool nulls[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -193,6 +196,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* rule_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -336,7 +346,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -359,6 +369,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *hba_lines = NIL;
ListCell *line;
+ int rule_number = 0;
MemoryContext linecxt;
MemoryContext hbacxt;
MemoryContext oldcxt;
@@ -393,8 +404,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
hbaline = parse_hba_line(tok_line, DEBUG3);
- fill_hba_line(tuple_store, tupdesc, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ rule_number++;
+
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -430,8 +445,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
PG_RETURN_NULL();
}
-/* Number of columns in pg_ident_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5
+/* Number of columns in pg_hba_file_mappings view */
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -448,7 +463,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, IdentLine *ident, const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -461,6 +477,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* mapping_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -473,7 +496,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -495,6 +518,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *ident_lines = NIL;
ListCell *line;
+ int mapping_number = 0;
MemoryContext linecxt;
MemoryContext identcxt;
MemoryContext oldcxt;
@@ -529,7 +553,12 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
identline = parse_ident_line(tok_line, DEBUG3);
- fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline,
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ mapping_number++;
+
+ fill_ident_line(tuple_store, tupdesc, mapping_number,
+ tok_line->file_name, tok_line->line_num, identline,
tok_line->err_msg);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index babe16f00a..65bfd32753 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6128,15 +6128,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '6250', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,text,text,text}', proargmodes => '{o,o,o,o,o}',
- proargnames => '{line_number,map_name,sys_name,pg_username,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index 90036f7bcd..59f6faf9f8 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -155,6 +156,7 @@ typedef struct AuthToken
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of AuthTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 21effe8315..11ffe1743b 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname,
WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
-pg_hba_file_rules| SELECT a.line_number,
+pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
+ a.line_number,
a.type,
a.database,
a.user_name,
@@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error);
-pg_ident_file_mappings| SELECT a.line_number,
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
+ a.line_number,
a.map_name,
a.sys_name,
a.pg_username,
a.error
- FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_username, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.33.1
--o45jk7yo2ulu7ifa
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0002-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v6 1/2] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-02-21 07:45 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw)
Catversion is bumped.
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/catalogs.sgml | 48 +++-
doc/src/sgml/client-auth.sgml | 48 +++-
src/backend/libpq/hba.c | 343 +++++++++++++++++++++----
src/backend/libpq/pg_hba.conf.sample | 10 +-
src/backend/libpq/pg_ident.conf.sample | 12 +-
src/backend/utils/adt/hbafuncs.c | 51 +++-
src/include/catalog/pg_proc.dat | 11 +-
src/include/libpq/hba.h | 2 +
src/test/regress/expected/rules.out | 12 +-
9 files changed, 453 insertions(+), 84 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 7f4f79d1b5..2b5b7ef5d6 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -10496,12 +10496,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>rule_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this rule if the rule is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule in the given file_name
</para></entry>
</row>
@@ -10636,6 +10655,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</thead>
<tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>mapping_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Rule number, in priority order, of this mapping if the mapping is valid,
+ otherwise null
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ File name of this mapping
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>line_number</structfield> <type>int4</type>
+ </para>
+ <para>
+ Line number of this mapping in the given file_name
+ </para></entry>
+ </row>
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index 142b0affcb..4e1438476e 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,21 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication rule.
+ Inclusion records specifies files that can be included, which contains
+ additional records. The records will be inserted in lieu of the inclusion
+ records. Those records only contains two fields: the
+ <literal>include</literal> or <literal>include_dir</literal> directive and
+ the file or directory to be included. The file or directory can be a
+ relative of absolute path, and can be double quoted if needed. For the
+ <literal>include_dir</literal> form, all files not starting with a
+ <literal>.</literal> and ending with <literal>.conf</literal> will be
+ included.
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,6 +116,8 @@
<para>
A record can have several formats:
<synopsis>
+include <replaceable>file</replaceable>
+include_dir <replaceable>directory</replaceable>
local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
@@ -118,6 +133,26 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_dir</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of all the files found in
+ the directory, if they don't start with a <literal>.</literal> and end
+ with <literal>.conf</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -835,8 +870,10 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
+<replaceable>include_dir</replaceable> <replaceable>directory</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -847,6 +884,11 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ As for <filename>pg_hba.conf</filename>, the lines in this file can either
+ be inclusion directives or an authentication rules, and follow the same
+ rules.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index f8393ca8ed..5d0caa587b 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -21,6 +21,7 @@
#include <fcntl.h>
#include <sys/param.h>
#include <sys/socket.h>
+#include <sys/stat.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
@@ -68,6 +69,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -112,10 +119,19 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
const char *inc_filename, int elevel, char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
+static char *process_included_authfile(const char *inc_filename,
+ const char *outer_filename, int elevel,
+ MemoryContext linecxt, List **tok_lines);
/*
@@ -355,36 +371,11 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
- {
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
- }
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename,
+ elevel, err_msg, &inc_fullname);
- inc_file = AllocateFile(inc_fullname, "r");
if (inc_file == NULL)
- {
- int save_errno = errno;
-
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
return tokens;
- }
/* There is possible recursion here if the file contains @ */
linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
@@ -425,11 +416,36 @@ tokenize_inc_file(List *tokens,
/*
* tokenize_auth_file
- * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a decicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel);
+
+ return linecxt;
+}
+
+/*
+ * Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see the struct definition
* in libpq/hba.h.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -438,30 +454,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -522,29 +530,151 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ AuthToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
+
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, filename,
+ elevel, linecxt, tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
+ else if (strcmp(first->string, "include_dir") == 0)
+ {
+ char *dir_name;
+ DIR *d;
+ struct dirent *de;
+ StringInfoData err_buf;
+
+ dir_name = second->string;
+ d = AllocateDir(dir_name);
+ if (d == NULL)
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not open directory \"%s\": %m",
+ dir_name)));
+ err_msg = psprintf("coud not open directory \"%s\": %m",
+ dir_name);
+ goto process_line;
+ }
+
+ initStringInfo(&err_buf);
+ while ((de = ReadDir(d, dir_name)) != NULL)
+ {
+ struct stat st;
+ char inc_filename[MAXPGPATH];
+
+ /*
+ * Only parse files with names ending in ".conf".
+ * Explicitly reject files starting with ".". This
+ * excludes things like "." and "..", as well as typical
+ * hidden files, backup files, and editor debris.
+ */
+ if (strlen(de->d_name) < 6)
+ continue;
+ if (de->d_name[0] == '.')
+ continue;
+ if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
+ continue;
+
+ join_path_components(inc_filename, dir_name, de->d_name);
+ canonicalize_path(inc_filename);
+ if (stat(inc_filename, &st) == 0)
+ {
+ /* Ignore directories. */
+ if (S_ISDIR(st.st_mode))
+ continue;
+
+ /*
+ * err_msg is used here as a temp buffer, it will be
+ * overwritten at the end of the loop with the
+ * cumulated errors, if any.
+ */
+ err_msg = process_included_authfile(inc_filename,
+ filename, elevel, linecxt,
+ tok_lines);
+
+ /* Cumulate errors if any. */
+ if (err_msg)
+ {
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfoString(&err_buf, err_msg);
+ }
+ }
+ else
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not stat file \"%s\": %m",
+ filename)));
+
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfo(&err_buf,
+ "could not stat file \"%s\": %m",
+ inc_filename);
+ }
+ }
+ FreeDir(d);
+
+ /*
+ * If there were no errors, the line is fully processed, bypass
+ * the general TokenizedAuthLine processing.
+ */
+ if (err_buf.len == 0)
+ goto next_line;
+
+ /* Otherwise, process the cumulated errors, if any. */
+ err_msg = err_buf.data;
+ }
}
+process_line:
+ /*
+ * General processing: report the error if any and emit line to the
+ * TokenizedAuthLine
+ */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
+
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -859,7 +989,7 @@ do { \
errmsg("authentication option \"%s\" is only valid for authentication methods %s", \
optname, _(validmethods)), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \
optname, validmethods); \
return false; \
@@ -879,7 +1009,7 @@ do { \
errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname); \
return NULL; \
@@ -901,7 +1031,7 @@ do { \
ereport(elevel, \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("missing entry in file \"%s\" at end of line %d", \
- IdentFileName, line_num))); \
+ tok_line->file_name, line_num))); \
*err_msg = psprintf("missing entry at end of line"); \
return NULL; \
} \
@@ -914,7 +1044,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("multiple values in ident field"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, tok_line->file_name))); \
*err_msg = psprintf("multiple values in ident field"); \
return NULL; \
} \
@@ -937,6 +1067,7 @@ HbaLine *
parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
{
int line_num = tok_line->line_num;
+ char *file_name = tok_line->file_name;
char **err_msg = &tok_line->err_msg;
char *str;
struct addrinfo *gai_result;
@@ -951,6 +1082,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -1677,6 +1809,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg)
{
int line_num = hbaline->linenumber;
+ char *file_name = hbaline->sourcefile;
#ifdef USE_LDAP
hbaline->ldapscope = LDAP_SCOPE_SUBTREE;
@@ -2299,6 +2432,102 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
+
+/*
+ * Try to open an included file, and tokenize it using the given context.
+ */
+static char *
+process_included_authfile(const char *inc_filename, const char *outer_filename,
+ int elevel, MemoryContext linecxt, List **tok_lines)
+{
+ char *inc_fullname;
+ FILE *inc_file;
+ char *err_msg = NULL;
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename, outer_filename,
+ elevel, &err_msg, &inc_fullname);
+
+ if (inc_file == NULL)
+ {
+ /* open_inc_file should have reported an error. */
+ Assert(err_msg != NULL);
+ return err_msg;
+ }
+ else
+ {
+ /* No error message should have been reported. */
+ Assert(err_msg == NULL);
+ }
+
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ return NULL;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..0050e24186 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,6 +9,8 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
+# include FILE
+# include_dir DIRECTORY
# local DATABASE USER METHOD [OPTIONS]
# host DATABASE USER ADDRESS METHOD [OPTIONS]
# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
@@ -18,7 +20,13 @@
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include" or "include_dir", it's not a mapping record but a directive to
+# include records from other file(s), as specified in the field. FILE is the
+# file to include, and DIR is the directory containing the file(s) to include
+# It can be specified with a relative or absolute path, and can be double
+# quoted if it contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..9b700d9989 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,20 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
-# MAPNAME SYSTEM-USERNAME PG-USERNAME
+# include FILE
+# include_dir FILE
+# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include" or "include_dir", it's not an authentication
+# record but a directive to include records from other file(s), specified in
+# the field. FILE is the file to include and DIR is the directory containing
+# the file(s) to include. It can be specified with a relative or absolute
+# path, and can be double quoted if it contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index 9fe7b62c9a..4d902927e0 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,9 +26,11 @@
static ArrayType *get_hba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
+ int mapping_number, const char *filename,
int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
@@ -157,7 +159,7 @@ get_hba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 9
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line
@@ -174,7 +176,8 @@ get_hba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, HbaLine *hba, const char *err_msg)
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
+ const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
bool nulls[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -193,6 +196,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* rule_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -336,7 +346,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -359,6 +369,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *hba_lines = NIL;
ListCell *line;
+ int rule_number = 0;
MemoryContext linecxt;
MemoryContext hbacxt;
MemoryContext oldcxt;
@@ -393,8 +404,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
hbaline = parse_hba_line(tok_line, DEBUG3);
- fill_hba_line(tuple_store, tupdesc, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ rule_number++;
+
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -430,8 +445,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
PG_RETURN_NULL();
}
-/* Number of columns in pg_ident_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5
+/* Number of columns in pg_hba_file_mappings view */
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -448,7 +463,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int lineno, IdentLine *ident, const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -461,6 +477,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
memset(nulls, 0, sizeof(nulls));
index = 0;
+ /* mapping_number */
+ if (err_msg)
+ nulls[index++] = true;
+ else
+ values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -473,7 +496,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -495,6 +518,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
FILE *file;
List *ident_lines = NIL;
ListCell *line;
+ int mapping_number = 0;
MemoryContext linecxt;
MemoryContext identcxt;
MemoryContext oldcxt;
@@ -529,7 +553,12 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
identline = parse_ident_line(tok_line, DEBUG3);
- fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline,
+ /* No error, set a rule number */
+ if (tok_line->err_msg == NULL)
+ mapping_number++;
+
+ fill_ident_line(tuple_store, tupdesc, mapping_number,
+ tok_line->file_name, tok_line->line_num, identline,
tok_line->err_msg);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 01e1dd4d6d..467f6f1293 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6111,15 +6111,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o}',
- proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '9556', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,text,text,text}', proargmodes => '{o,o,o,o,o}',
- proargnames => '{line_number,map_name,sys_name,pg_username,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index 90036f7bcd..59f6faf9f8 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -155,6 +156,7 @@ typedef struct AuthToken
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of AuthTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 423b9b99fb..e00185eb6d 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname,
WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
-pg_hba_file_rules| SELECT a.line_number,
+pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
+ a.line_number,
a.type,
a.database,
a.user_name,
@@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error);
-pg_ident_file_mappings| SELECT a.line_number,
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
+ a.line_number,
a.map_name,
a.sys_name,
a.pg_username,
a.error
- FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_username, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.33.1
--g2rofdrijxvb5udh
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0002-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
@ 2022-04-06 14:13 Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-05-25 20:42 ` Re: logical decoding and replication of sequences Zheng Li <[email protected]>
0 siblings, 2 replies; 33+ messages in thread
From: Tomas Vondra @ 2022-04-06 14:13 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 4/5/22 12:06, Amit Kapila wrote:
> On Mon, Apr 4, 2022 at 3:10 AM Tomas Vondra
> <[email protected]> wrote:
>>
>> I did some experiments over the weekend, exploring how to rework the
>> sequence decoding in various ways. Let me share some WIP patches,
>> hopefully that can be useful for trying more stuff and moving this
>> discussion forward.
>>
>> I tried two things - (1) accumulating sequence increments in global
>> array and then doing something with it, and (2) treating all sequence
>> increments as regular changes (in a TXN) and then doing something
>> special during the replay. Attached are two patchsets, one for each
>> approach.
>>
>> Note: It's important to remember decoding of sequences is not the only
>> code affected by this. The logical messages have the same issue,
>> certainly when it comes to transactional vs. non-transactional stuff and
>> handling of snapshots. Even if the sequence decoding ends up being
>> reverted, we still need to fix that, somehow. And my feeling is the
>> solutions ought to be pretty similar in both cases.
>>
>> Now, regarding the two approaches:
>>
>> (1) accumulating sequences in global hash table
>>
>> The main problem with regular sequence increments is that those need to
>> be non-transactional - a transaction may use a sequence without any
>> WAL-logging, if the WAL was written by an earlier transaction. The
>> problem is the earlier trasaction might have been rolled back, and thus
>> simply discarded by the logical decoding. But we still need to apply
>> that, in order not to lose the sequence increment.
>>
>> The current code just applies those non-transactional increments right
>> after decoding the increment, but that does not work because we may not
>> have a snapshot at that point. And we only have the snapshot when within
>> a transaction (AFAICS) so this queues all changes and then applies the
>> changes later.
>>
>> The changes need to be shared by all transactions, so queueing them in a
>> global works fairly well - otherwise we'd have to walk all transactions,
>> in order to see if there are relevant sequence increments.
>>
>> But some increments may be transactional, e.g. when the sequence is
>> created or altered in a transaction. To allow tracking this, this uses a
>> hash table, with relfilenode as a key.
>>
>> There's a couple issues with this, though. Firstly, stashing the changes
>> outside transactions, it's not included in memory accounting, it's not
>> spilled to disk or streamed, etc. I guess fixing this is possible, but
>> it's certainly not straightforward, because we mix increments from many
>> different transactions.
>>
>> A bigger issue is that I'm not sure this actually handles the snapshots
>> correctly either.
>>
>> The non-transactional increments affect all transactions, so when
>> ReorderBufferProcessSequences gets executed, it processes all of them,
>> no matter the source transaction. Can we be sure the snapshot in the
>> applying transaction is the same (or "compatible") as the snapshot in
>> the source transaction?
>>
>
> I don't think we can assume that. I think it is possible that some
> other transaction's WAL can be in-between start/end lsn of txn (which
> we decide to send) which may not finally reach a consistent state.
> Consider a case similar to shown in one of my previous emails:
> Session-2:
> Begin;
> SELECT pg_current_xact_id();
>
> Session-1:
> SELECT 'init' FROM pg_create_logical_replication_slot('test_slot',
> 'test_decoding', false, true);
>
> Session-3:
> Begin;
> SELECT pg_current_xact_id();
>
> Session-2:
> Commit;
> Begin;
> INSERT INTO t1_seq SELECT nextval('seq1') FROM generate_series(1,100);
>
> Session-3:
> Commit;
>
> Session-2:
> Commit;
>
> Here, we send changes (say insert from txn 700) from session-2 because
> session-3's commit happens before it. Now, consider another
> transaction parallel to txn 700 which generates some WAL related to
> sequences but it committed before session-3's commit. So though, its
> changes will be the in-between start/end LSN of txn 700 but those
> shouldn't be sent.
>
> I have not tried this and also this may be solvable in some way but I
> think processing changes from other TXNs sounds risky to me in terms
> of snapshot handling.
>
Yes, I know this can happen. I was only really thinking about what might
happen to the relfilenode of the sequence itself - and I don't think any
concurrent transaction could swoop in and change the relfilenode in any
meaningful way, due to locking.
But of course, if we expect/require to have a perfect snapshot for that
exact position in the transaction, this won't work. IMO the whole idea
that we can have non-transactional bits in naturally transactional
decoding seems a bit suspicious (at least in hindsight).
No matter what we do for sequences, though, this still affects logical
messages too. Not sure what to do there :-(
>>
>>
>> (2) treating sequence change as regular changes
>>
>> This adopts a different approach - instead of accumulating the sequence
>> increments in a global hash table, it treats them as regular changes.
>> Which solves the snapshot issue, and issues with spilling to disk,
>> streaming and so on.
>>
>> But it has various other issues with handling concurrent transactions,
>> unfortunately, which probably make this approach infeasible:
>>
>> * The non-transactional stuff has to be applied in the first transaction
>> that commits, not in the transaction that generated the WAL. That does
>> not work too well with this approach, because we have to walk changes in
>> all other transactions.
>>
>
> Why do you want to traverse other TXNs in this approach? Is it because
> the current TXN might be using some value of sequence which has been
> actually WAL logged in the other transaction but that other
> transaction has not been sent yet? I think if we don't send that then
> probably replica sequences columns (in some tables) have some values
> but actually the sequence itself won't have still that value which
> sounds problematic. Is that correct?
>
Well, how else would you get to sequence changes in the other TXNs?
Consider this:
T1: begin
T2: begin
T2: nextval('s') -> writes WAL for 32 values
T1: nextval('s') -> gets value without WAL
T1: commit
T2: commit
Now, if we commit T1 without "applying" the sequence change from T2, we
loose the sequence state. But we still write/replicate the value
generated from the sequence.
>> * Another serious issue seems to be streaming - if we already streamed
>> some of the changes, we can't iterate through them anymore.
>>
>> Also, having to walk the transactions over and over for each change, to
>> apply relevant sequence increments, that's mighty expensive. The other
>> approach needs to do that too, but walking the global hash table seems
>> much cheaper.
>>
>> The other issue this handling of aborted transactions - we need to apply
>> sequence increments even from those transactions, of course. The other
>> approach has this issue too, though.
>>
>>
>> (3) tracking sequences touched by transaction
>>
>> This is the approach proposed by Hannu Krosing. I haven't explored this
>> again yet, but I recall I wrote a PoC patch a couple months back.
>>
>> It seems to me most of the problems stems from trying to derive sequence
>> state from decoded WAL changes, which is problematic because of the
>> non-transactional nature of sequences (i.e. WAL for one transaction
>> affects other transactions in non-obvious ways). And this approach
>> simply works around that entirely - instead of trying to deduce the
>> sequence state from WAL, we'd make sure to write the current sequence
>> state (or maybe just ID of the sequence) at commit time. Which should
>> eliminate most of the complexity / problems, I think.
>>
>
> That sounds promising but I haven't thought in detail about that approach.
>
So, here's a patch doing that. It's a reworked/improved version of the
patch [1] shared in November.
It seems to be working pretty nicely. The behavior is a little bit
different, of course, because we only replicate "committed" changes, so
if you do nextval() in aborted transaction that is not replicated. Which
I think is fine, because we generally make no durability guarantees for
aborted transactions in general.
But there are a couple issues too:
1) locking
We have to read sequence change before the commit, but we must not allow
reordering (because then the state might go backwards again). I'm not
sure how serious impact could this have on performance.
2) dropped sequences
I'm not sure what to do about sequences dropped in the transaction. The
patch simply attempts to read the current sequence state before the
commit, but if the sequence was dropped (in that transaction), that
can't happen. I'm not sure if that's OK or not.
3) WAL record
To replicate the stuff the patch uses a LogicalMessage, but I guess a
separate WAL record would be better. But that's a technical detail.
regards
[1]
https://www.postgresql.org/message-id/[email protected]
>>
>> I'm not really sure what to do about this. All of those reworks seems
>> like an extensive redesign of the patch, and considering the last CF is
>> already over ... not great.
>>
>
> Yeah, I share the same feeling that even if we devise solutions to all
> the known problems it requires quite some time to ensure everything is
> correct.
>
True. Let's keep working on this for a bit more time and then we can
decide what to do.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
[text/x-patch] 0001-rework-sequence-decoding.patch (56.8K, ../../[email protected]/2-0001-rework-sequence-decoding.patch)
download | inline diff:
From 0ba154eef933a9b39bb7eaa0e339620a2f98a37b Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Tue, 5 Apr 2022 00:25:02 +0200
Subject: [PATCH] rework sequence decoding
---
contrib/test_decoding/test_decoding.c | 24 +-
src/backend/access/rmgrdesc/logicalmsgdesc.c | 13 +
src/backend/access/transam/xact.c | 25 ++
src/backend/commands/sequence.c | 263 +++++++-----
src/backend/replication/logical/decode.c | 112 ++---
src/backend/replication/logical/logical.c | 9 +-
src/backend/replication/logical/message.c | 3 +-
src/backend/replication/logical/proto.c | 4 +-
.../replication/logical/reorderbuffer.c | 387 ++----------------
src/backend/replication/logical/tablesync.c | 3 +-
src/backend/replication/logical/worker.c | 16 +-
src/backend/replication/pgoutput/pgoutput.c | 21 +-
src/include/access/rmgrlist.h | 2 +-
src/include/commands/sequence.h | 5 +-
src/include/replication/logicalproto.h | 2 -
src/include/replication/message.h | 15 +
src/include/replication/output_plugin.h | 2 -
src/include/replication/reorderbuffer.h | 16 +-
18 files changed, 330 insertions(+), 592 deletions(-)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index c7a87f5fe5b..851117b4155 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -79,7 +79,7 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
Size sz, const char *message);
static void pg_decode_sequence(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr sequence_lsn,
- Relation rel, bool transactional,
+ Relation rel,
int64 last_value, int64 log_cnt, bool is_called);
static bool pg_decode_filter_prepare(LogicalDecodingContext *ctx,
TransactionId xid,
@@ -123,7 +123,7 @@ static void pg_decode_stream_message(LogicalDecodingContext *ctx,
Size sz, const char *message);
static void pg_decode_stream_sequence(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr sequence_lsn,
- Relation rel, bool transactional,
+ Relation rel,
int64 last_value, int64 log_cnt, bool is_called);
static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
@@ -782,7 +782,6 @@ pg_decode_message(LogicalDecodingContext *ctx,
static void
pg_decode_sequence(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
XLogRecPtr sequence_lsn, Relation rel,
- bool transactional,
int64 last_value, int64 log_cnt, bool is_called)
{
TestDecodingData *data = ctx->output_plugin_private;
@@ -792,22 +791,19 @@ pg_decode_sequence(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
return;
/* output BEGIN if we haven't yet, but only for the transactional case */
- if (transactional)
+ if (data->skip_empty_xacts && !txndata->xact_wrote_changes)
{
- if (data->skip_empty_xacts && !txndata->xact_wrote_changes)
- {
- pg_output_begin(ctx, data, txn, false);
- }
- txndata->xact_wrote_changes = true;
+ pg_output_begin(ctx, data, txn, false);
}
+ txndata->xact_wrote_changes = true;
OutputPluginPrepareWrite(ctx, true);
appendStringInfoString(ctx->out, "sequence ");
appendStringInfoString(ctx->out,
quote_qualified_identifier(get_namespace_name(get_rel_namespace(RelationGetRelid(rel))),
RelationGetRelationName(rel)));
- appendStringInfo(ctx->out, ": transactional:%d last_value: " INT64_FORMAT " log_cnt: " INT64_FORMAT " is_called:%d",
- transactional, last_value, log_cnt, is_called);
+ appendStringInfo(ctx->out, ": last_value: %zu log_cnt: %zu is_called:%d",
+ last_value, log_cnt, is_called);
OutputPluginWrite(ctx, true);
}
@@ -1013,7 +1009,6 @@ pg_decode_stream_message(LogicalDecodingContext *ctx,
static void
pg_decode_stream_sequence(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
XLogRecPtr sequence_lsn, Relation rel,
- bool transactional,
int64 last_value, int64 log_cnt, bool is_called)
{
TestDecodingData *data = ctx->output_plugin_private;
@@ -1023,7 +1018,6 @@ pg_decode_stream_sequence(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
return;
/* output BEGIN if we haven't yet, but only for the transactional case */
- if (transactional)
{
if (data->skip_empty_xacts && !txndata->xact_wrote_changes)
{
@@ -1037,8 +1031,8 @@ pg_decode_stream_sequence(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
appendStringInfoString(ctx->out,
quote_qualified_identifier(get_namespace_name(get_rel_namespace(RelationGetRelid(rel))),
RelationGetRelationName(rel)));
- appendStringInfo(ctx->out, ": transactional:%d last_value: " INT64_FORMAT " log_cnt: " INT64_FORMAT " is_called:%d",
- transactional, last_value, log_cnt, is_called);
+ appendStringInfo(ctx->out, ": last_value: " INT64_FORMAT " log_cnt: " INT64_FORMAT " is_called:%d",
+ last_value, log_cnt, is_called);
OutputPluginWrite(ctx, true);
}
diff --git a/src/backend/access/rmgrdesc/logicalmsgdesc.c b/src/backend/access/rmgrdesc/logicalmsgdesc.c
index 099e11a84e7..de77dcb0dc2 100644
--- a/src/backend/access/rmgrdesc/logicalmsgdesc.c
+++ b/src/backend/access/rmgrdesc/logicalmsgdesc.c
@@ -40,6 +40,16 @@ logicalmsg_desc(StringInfo buf, XLogReaderState *record)
sep = " ";
}
}
+ else if (info == XLOG_LOGICAL_SEQUENCE)
+ {
+ xl_logical_sequence *xlrec = (xl_logical_sequence *) rec;
+
+ appendStringInfo(buf, "rel %u/%u/%u last: %lu log_cnt: %lu is_called: %d",
+ xlrec->node.spcNode,
+ xlrec->node.dbNode,
+ xlrec->node.relNode,
+ xlrec->last, xlrec->log_cnt, xlrec->is_called);
+ }
}
const char *
@@ -48,5 +58,8 @@ logicalmsg_identify(uint8 info)
if ((info & ~XLR_INFO_MASK) == XLOG_LOGICAL_MESSAGE)
return "MESSAGE";
+ if ((info & ~XLR_INFO_MASK) == XLOG_LOGICAL_SEQUENCE)
+ return "SEQUENCE";
+
return NULL;
}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 3596a7d7345..5b6a07ca892 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -37,6 +37,7 @@
#include "catalog/storage.h"
#include "commands/async.h"
#include "commands/tablecmds.h"
+#include "commands/sequence.h"
#include "commands/trigger.h"
#include "common/pg_prng.h"
#include "executor/spi.h"
@@ -2223,6 +2224,15 @@ CommitTransaction(void)
/* Prevent cancel/die interrupt while cleaning up */
HOLD_INTERRUPTS();
+ /*
+ * Write state of sequences to WAL. We need to do this early enough so
+ * that we can still write stuff to WAL, but probably before changing
+ * the transaction state.
+ *
+ * XXX Should this happen before/after holding the interrupts?
+ */
+ AtEOXact_Sequences(true);
+
/* Commit updates to the relation map --- do this as late as possible */
AtEOXact_RelationMap(true, is_parallel_worker);
@@ -2499,6 +2509,15 @@ PrepareTransaction(void)
/* Prevent cancel/die interrupt while cleaning up */
HOLD_INTERRUPTS();
+ /*
+ * Write state of sequences to WAL. We need to do this early enough so
+ * that we can still write stuff to WAL, but probably before changing
+ * the transaction state.
+ *
+ * XXX Should this happen before/after holding the interrupts?
+ */
+ AtEOXact_Sequences(true);
+
/*
* set the current transaction state information appropriately during
* prepare processing
@@ -2734,6 +2753,12 @@ AbortTransaction(void)
TransStateAsString(s->state));
Assert(s->parent == NULL);
+ /*
+ * For transaction abort we don't need to write anything to WAL, we just
+ * do cleanup of the hash table etc.
+ */
+ AtEOXact_Sequences(false);
+
/*
* set the current transaction state information appropriately during the
* abort processing
diff --git a/src/backend/commands/sequence.c b/src/backend/commands/sequence.c
index 717bb0b2aa9..50c03b71dac 100644
--- a/src/backend/commands/sequence.c
+++ b/src/backend/commands/sequence.c
@@ -37,6 +37,7 @@
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "parser/parse_type.h"
+#include "replication/message.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
#include "storage/smgr.h"
@@ -75,6 +76,7 @@ typedef struct SeqTableData
{
Oid relid; /* pg_class OID of this sequence (hash key) */
Oid filenode; /* last seen relfilenode of this sequence */
+ Oid tablespace; /* last seen tablespace of this sequence */
LocalTransactionId lxid; /* xact in which we last did a seq op */
bool last_valid; /* do we have a valid "last" value? */
int64 last; /* value last returned by nextval */
@@ -82,6 +84,7 @@ typedef struct SeqTableData
/* if last != cached, we have not used up all the cached values */
int64 increment; /* copy of sequence's increment field */
/* note that increment is zero until we first do nextval_internal() */
+ bool need_log; /* should be written to WAL at commit? */
} SeqTableData;
typedef SeqTableData *SeqTable;
@@ -336,83 +339,13 @@ ResetSequence(Oid seq_relid)
relation_close(seq_rel, NoLock);
}
-/*
- * Update the sequence state by modifying the existing sequence data row.
- *
- * This keeps the same relfilenode, so the behavior is non-transactional.
- */
-static void
-SetSequence_non_transactional(Oid seqrelid, int64 last_value, int64 log_cnt, bool is_called)
-{
- SeqTable elm;
- Relation seqrel;
- Buffer buf;
- HeapTupleData seqdatatuple;
- Form_pg_sequence_data seq;
-
- /* open and lock sequence */
- init_sequence(seqrelid, &elm, &seqrel);
-
- /* lock page' buffer and read tuple */
- seq = read_seq_tuple(seqrel, &buf, &seqdatatuple);
-
- /* check the comment above nextval_internal()'s equivalent call. */
- if (RelationNeedsWAL(seqrel))
- {
- GetTopTransactionId();
-
- if (XLogLogicalInfoActive())
- GetCurrentTransactionId();
- }
-
- /* ready to change the on-disk (or really, in-buffer) tuple */
- START_CRIT_SECTION();
-
- seq->last_value = last_value;
- seq->is_called = is_called;
- seq->log_cnt = log_cnt;
-
- MarkBufferDirty(buf);
-
- /* XLOG stuff */
- if (RelationNeedsWAL(seqrel))
- {
- xl_seq_rec xlrec;
- XLogRecPtr recptr;
- Page page = BufferGetPage(buf);
-
- XLogBeginInsert();
- XLogRegisterBuffer(0, buf, REGBUF_WILL_INIT);
-
- xlrec.node = seqrel->rd_node;
- xlrec.created = false;
-
- XLogRegisterData((char *) &xlrec, sizeof(xl_seq_rec));
- XLogRegisterData((char *) seqdatatuple.t_data, seqdatatuple.t_len);
-
- recptr = XLogInsert(RM_SEQ_ID, XLOG_SEQ_LOG);
-
- PageSetLSN(page, recptr);
- }
-
- END_CRIT_SECTION();
-
- UnlockReleaseBuffer(buf);
-
- /* Clear local cache so that we don't think we have cached numbers */
- /* Note that we do not change the currval() state */
- elm->cached = elm->last;
-
- relation_close(seqrel, NoLock);
-}
-
/*
* Update the sequence state by creating a new relfilenode.
*
* This creates a new relfilenode, to allow transactional behavior.
*/
-static void
-SetSequence_transactional(Oid seq_relid, int64 last_value, int64 log_cnt, bool is_called)
+void
+SetSequence(Oid seq_relid, int64 last_value, int64 log_cnt, bool is_called)
{
SeqTable elm;
Relation seqrel;
@@ -469,27 +402,6 @@ SetSequence_transactional(Oid seq_relid, int64 last_value, int64 log_cnt, bool i
relation_close(seqrel, NoLock);
}
-/*
- * Set a sequence to a specified internal state.
- *
- * The change is made transactionally, so that on failure of the current
- * transaction, the sequence will be restored to its previous state.
- * We do that by creating a whole new relfilenode for the sequence; so this
- * works much like the rewriting forms of ALTER TABLE.
- *
- * Caller is assumed to have acquired AccessExclusiveLock on the sequence,
- * which must not be released until end of transaction. Caller is also
- * responsible for permissions checking.
- */
-void
-SetSequence(Oid seq_relid, bool transactional, int64 last_value, int64 log_cnt, bool is_called)
-{
- if (transactional)
- SetSequence_transactional(seq_relid, last_value, log_cnt, is_called);
- else
- SetSequence_non_transactional(seq_relid, last_value, log_cnt, is_called);
-}
-
/*
* Initialize a sequence's relation with the specified tuple as content
*/
@@ -530,7 +442,13 @@ fill_seq_with_data(Relation rel, HeapTuple tuple)
tuple->t_data->t_infomask |= HEAP_XMAX_INVALID;
ItemPointerSet(&tuple->t_data->t_ctid, 0, FirstOffsetNumber);
- /* check the comment above nextval_internal()'s equivalent call. */
+ /*
+ * Remember we need to write this sequence to WAL at commit, and make sure
+ * we have XID if we may need to decode this sequence.
+ *
+ * XXX Maybe we should not be doing this except when actually reading a
+ * value from the sequence?
+ */
if (RelationNeedsWAL(rel))
{
GetTopTransactionId();
@@ -558,7 +476,6 @@ fill_seq_with_data(Relation rel, HeapTuple tuple)
XLogRegisterBuffer(0, buf, REGBUF_WILL_INIT);
xlrec.node = rel->rd_node;
- xlrec.created = true;
XLogRegisterData((char *) &xlrec, sizeof(xl_seq_rec));
XLogRegisterData((char *) tuple->t_data, tuple->t_len);
@@ -762,6 +679,23 @@ nextval_internal(Oid relid, bool check_permissions)
/* open and lock sequence */
init_sequence(relid, &elm, &seqrel);
+ /*
+ * Remember we need to write this sequence to WAL at commit, and make sure
+ * we have XID if we may need to decode this sequence.
+ *
+ * XXX Maybe we should not be doing this except when actually reading a
+ * value from the sequence?
+ */
+ if (RelationNeedsWAL(seqrel))
+ {
+ elm->need_log = true;
+
+ GetTopTransactionId();
+
+ if (XLogLogicalInfoActive())
+ GetCurrentTransactionId();
+ }
+
if (check_permissions &&
pg_class_aclcheck(elm->relid, GetUserId(),
ACL_USAGE | ACL_UPDATE) != ACLCHECK_OK)
@@ -973,7 +907,6 @@ nextval_internal(Oid relid, bool check_permissions)
seq->log_cnt = 0;
xlrec.node = seqrel->rd_node;
- xlrec.created = false;
XLogRegisterData((char *) &xlrec, sizeof(xl_seq_rec));
XLogRegisterData((char *) seqdatatuple.t_data, seqdatatuple.t_len);
@@ -1092,6 +1025,23 @@ do_setval(Oid relid, int64 next, bool iscalled)
/* open and lock sequence */
init_sequence(relid, &elm, &seqrel);
+ /*
+ * Remember we need to write this sequence to WAL at commit, and make sure
+ * we have XID if we may need to decode this sequence.
+ *
+ * XXX Maybe we should not be doing this except when actually reading a
+ * value from the sequence?
+ */
+ if (RelationNeedsWAL(seqrel))
+ {
+ elm->need_log = true;
+
+ GetTopTransactionId();
+
+ if (XLogLogicalInfoActive())
+ GetCurrentTransactionId();
+ }
+
if (pg_class_aclcheck(elm->relid, GetUserId(), ACL_UPDATE) != ACLCHECK_OK)
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
@@ -1166,8 +1116,6 @@ do_setval(Oid relid, int64 next, bool iscalled)
XLogRegisterBuffer(0, buf, REGBUF_WILL_INIT);
xlrec.node = seqrel->rd_node;
- xlrec.created = false;
-
XLogRegisterData((char *) &xlrec, sizeof(xl_seq_rec));
XLogRegisterData((char *) seqdatatuple.t_data, seqdatatuple.t_len);
@@ -1316,6 +1264,7 @@ init_sequence(Oid relid, SeqTable *p_elm, Relation *p_rel)
if (seqrel->rd_rel->relfilenode != elm->filenode)
{
elm->filenode = seqrel->rd_rel->relfilenode;
+ elm->tablespace = seqrel->rd_rel->reltablespace;
elm->cached = elm->last;
}
@@ -2051,3 +2000,121 @@ seq_mask(char *page, BlockNumber blkno)
mask_unused_space(page);
}
+
+
+static void
+read_sequence_info(Relation seqrel, int64 *last_value, int64 *log_cnt, bool *is_called)
+{
+ Buffer buf;
+ Form_pg_sequence_data seq;
+ HeapTupleData seqdatatuple;
+
+ seq = read_seq_tuple(seqrel, &buf, &seqdatatuple);
+
+ *last_value = seq->last_value;
+ *is_called = seq->is_called;
+ *log_cnt = seq->log_cnt;
+
+ UnlockReleaseBuffer(buf);
+}
+
+/* XXX Do this only for wal_level = logical, probably? */
+void
+AtEOXact_Sequences(bool isCommit)
+{
+ SeqTable entry;
+ HASH_SEQ_STATUS scan;
+
+ if (!seqhashtab)
+ return;
+
+ /* only do this with wal_level=logical */
+ if (!XLogLogicalInfoActive())
+ return;
+
+ /*
+ * XXX Maybe we could enforce having XID here? Or is it too late?
+ */
+ // Assert(GetTopTransactionId() != InvalidTransactionId);
+ // Assert(GetCurrentTransactionId() != InvalidTransactionId);
+
+ hash_seq_init(&scan, seqhashtab);
+
+ while ((entry = (SeqTable) hash_seq_search(&scan)))
+ {
+ Relation rel;
+ RelFileNode rnode;
+ xl_logical_sequence xlrec;
+
+ /* not commit, we don't guarantee any data to be durable */
+ if (!isCommit)
+ entry->need_log = false;
+
+ /*
+ * If not touched in the current transaction, don't log anything.
+ * We leave needs_log set, so that if future transactions touch
+ * the sequence we'll log it properly.
+ */
+ if (!entry->need_log)
+ continue;
+
+ /* if this is commit, we'll log the */
+ entry->need_log = false;
+
+ /*
+ * does the relation still exist?
+ *
+ * XXX We need to make sure another transaction can't jump ahead of
+ * this one, otherwise the ordering of sequence changes could change.
+ * Imagine T1 and T2, where T1 writes sequence state first, but then
+ * T2 does it too and commits first:
+ *
+ * T1: log sequence state
+ * T2: increment sequence
+ * T2: log sequence state
+ * T2: commit
+ * T1: commit
+ *
+ * If we apply the sequences in this order, it'd be broken as the
+ * value might go backwars. ShareUpdateExclusive protects against
+ * that, but it's also restricting commit throughtput, probably.
+ *
+ * XXX This might be an issue for deadlocks, too, if two xacts try
+ * to write sequences in different ordering. We may need to sort
+ * the OIDs first, to enforce the same lock ordering.
+ */
+ rel = try_relation_open(entry->relid, ShareUpdateExclusiveLock);
+
+ /* XXX relation might have been dropped, maybe we should have logged
+ * the last change? */
+ if (!rel)
+ continue;
+
+ /* tablespace */
+ if (OidIsValid(entry->tablespace))
+ rnode.spcNode = entry->tablespace;
+ else
+ rnode.spcNode = MyDatabaseTableSpace;
+
+ rnode.dbNode = MyDatabaseId; /* database */
+ rnode.relNode = entry->filenode; /* relation */
+
+ xlrec.node = rnode;
+ xlrec.reloid = entry->relid;
+
+ /* XXX is it good enough to log values we have in cache? seems
+ * wrong and we may need to re-read that. */
+ read_sequence_info(rel, &xlrec.last, &xlrec.log_cnt, &xlrec.is_called);
+
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, SizeOfLogicalSequence);
+
+ /* allow origin filtering */
+ XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
+
+ (void) XLogInsert(RM_LOGICALMSG_ID, XLOG_LOGICAL_SEQUENCE);
+
+ /* hold the sequence lock until the end of the transaction */
+ relation_close(rel, NoLock);
+ }
+}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 77bc7aea7a0..a8548f9ac64 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -44,6 +44,9 @@
#include "storage/standby.h"
#include "commands/sequence.h"
+static void DecodeLogicalMessage(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
+static void DecodeLogicalSequence(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
+
/* individual record(group)'s handlers */
static void DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
static void DecodeUpdate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
@@ -64,7 +67,6 @@ static void DecodePrepare(LogicalDecodingContext *ctx, XLogRecordBuffer *buf,
/* common function to decode tuples */
static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
-static void DecodeSeqTuple(char *data, Size len, ReorderBufferTupleBuf *tuple);
/* helper functions for decoding transactions */
static inline bool FilterPrepare(LogicalDecodingContext *ctx,
@@ -560,6 +562,27 @@ FilterByOrigin(LogicalDecodingContext *ctx, RepOriginId origin_id)
*/
void
logicalmsg_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
+{
+ XLogReaderState *r = buf->record;
+ uint8 info = XLogRecGetInfo(r) & ~XLR_INFO_MASK;
+
+ switch (info)
+ {
+ case XLOG_LOGICAL_MESSAGE:
+ DecodeLogicalMessage(ctx, buf);
+ break;
+
+ case XLOG_LOGICAL_SEQUENCE:
+ DecodeLogicalSequence(ctx, buf);
+ break;
+
+ default:
+ elog(ERROR, "unexpected RM_LOGICALMSG_ID record type: %u", info);
+ }
+}
+
+static void
+DecodeLogicalMessage(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
SnapBuild *builder = ctx->snapshot_builder;
XLogReaderState *r = buf->record;
@@ -1253,36 +1276,6 @@ DecodeTXNNeedSkip(LogicalDecodingContext *ctx, XLogRecordBuffer *buf,
ctx->fast_forward || FilterByOrigin(ctx, origin_id));
}
-/*
- * DecodeSeqTuple
- * decode tuple describing the sequence increment
- *
- * Sequences are represented as a table with a single row, which gets updated
- * by nextval(). The tuple is stored in WAL right after the xl_seq_rec, so we
- * simply copy it into the tuplebuf (similar to seq_redo).
- */
-static void
-DecodeSeqTuple(char *data, Size len, ReorderBufferTupleBuf *tuple)
-{
- int datalen = len - sizeof(xl_seq_rec) - SizeofHeapTupleHeader;
-
- Assert(datalen >= 0);
-
- tuple->tuple.t_len = datalen + SizeofHeapTupleHeader;
-
- ItemPointerSetInvalid(&tuple->tuple.t_self);
-
- tuple->tuple.t_tableOid = InvalidOid;
-
- memcpy(((char *) tuple->tuple.t_data),
- data + sizeof(xl_seq_rec),
- SizeofHeapTupleHeader);
-
- memcpy(((char *) tuple->tuple.t_data) + SizeofHeapTupleHeader,
- data + sizeof(xl_seq_rec) + SizeofHeapTupleHeader,
- datalen);
-}
-
/*
* Handle sequence decode
*
@@ -1301,24 +1294,18 @@ DecodeSeqTuple(char *data, Size len, ReorderBufferTupleBuf *tuple)
* plugin - it might get confused about which sequence it's related to etc.
*/
void
-sequence_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
+DecodeLogicalSequence(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
SnapBuild *builder = ctx->snapshot_builder;
- ReorderBufferTupleBuf *tuplebuf;
RelFileNode target_node;
XLogReaderState *r = buf->record;
- char *tupledata = NULL;
- Size tuplelen;
- Size datalen = 0;
TransactionId xid = XLogRecGetXid(r);
uint8 info = XLogRecGetInfo(buf->record) & ~XLR_INFO_MASK;
- xl_seq_rec *xlrec;
- Snapshot snapshot;
+ xl_logical_sequence *xlrec;
RepOriginId origin_id = XLogRecGetOrigin(r);
- bool transactional;
/* only decode changes flagged with XLOG_SEQ_LOG */
- if (info != XLOG_SEQ_LOG)
+ if (info != XLOG_LOGICAL_SEQUENCE)
elog(ERROR, "unexpected RM_SEQ_ID record type: %u", info);
ReorderBufferProcessXid(ctx->reorder, XLogRecGetXid(r), buf->origptr);
@@ -1331,8 +1318,11 @@ sequence_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
ctx->fast_forward)
return;
+ /* extract the WAL record, with "created" flag */
+ xlrec = (xl_logical_sequence *) XLogRecGetData(r);
+
/* only interested in our database */
- XLogRecGetBlockTag(r, 0, &target_node, NULL, NULL);
+ target_node = xlrec->node;
if (target_node.dbNode != ctx->slot->data.database)
return;
@@ -1340,44 +1330,12 @@ sequence_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
return;
- tupledata = XLogRecGetData(r);
- datalen = XLogRecGetDataLen(r);
- tuplelen = datalen - SizeOfHeapHeader - sizeof(xl_seq_rec);
-
- /* extract the WAL record, with "created" flag */
- xlrec = (xl_seq_rec *) XLogRecGetData(r);
-
- /* XXX how could we have sequence change without data? */
- if(!datalen || !tupledata)
- return;
-
- tuplebuf = ReorderBufferGetTupleBuf(ctx->reorder, tuplelen);
- DecodeSeqTuple(tupledata, datalen, tuplebuf);
-
- /*
- * Should we handle the sequence increment as transactional or not?
- *
- * If the sequence was created in a still-running transaction, treat
- * it as transactional and queue the increments. Otherwise it needs
- * to be treated as non-transactional, in which case we send it to
- * the plugin right away.
- */
- transactional = ReorderBufferSequenceIsTransactional(ctx->reorder,
- target_node,
- xlrec->created);
-
/* Skip the change if already processed (per the snapshot). */
- if (transactional &&
- !SnapBuildProcessChange(builder, xid, buf->origptr))
- return;
- else if (!transactional &&
- (SnapBuildCurrentState(builder) != SNAPBUILD_CONSISTENT ||
- SnapBuildXactNeedsSkip(builder, buf->origptr)))
+ if (!SnapBuildProcessChange(builder, xid, buf->origptr))
return;
/* Queue the increment (or send immediately if not transactional). */
- snapshot = SnapBuildGetOrBuildSnapshot(builder, xid);
- ReorderBufferQueueSequence(ctx->reorder, xid, snapshot, buf->endptr,
- origin_id, target_node, transactional,
- xlrec->created, tuplebuf);
+ ReorderBufferQueueSequence(ctx->reorder, xid, buf->endptr,
+ origin_id, xlrec->reloid, target_node,
+ xlrec->last, xlrec->log_cnt, xlrec->is_called);
}
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index e1f14aeecb5..d00cc08fb3a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -75,7 +75,6 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
const char *prefix, Size message_size, const char *message);
static void sequence_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr sequence_lsn, Relation rel,
- bool transactional,
int64 last_value, int64 log_cnt, bool is_called);
/* streaming callbacks */
@@ -96,7 +95,6 @@ static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *tx
const char *prefix, Size message_size, const char *message);
static void stream_sequence_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr sequence_lsn, Relation rel,
- bool transactional,
int64 last_value, int64 log_cnt, bool is_called);
static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
int nrelations, Relation relations[], ReorderBufferChange *change);
@@ -1218,7 +1216,7 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
static void
sequence_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
- XLogRecPtr sequence_lsn, Relation rel, bool transactional,
+ XLogRecPtr sequence_lsn, Relation rel,
int64 last_value, int64 log_cnt, bool is_called)
{
LogicalDecodingContext *ctx = cache->private_data;
@@ -1245,7 +1243,7 @@ sequence_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
ctx->write_location = sequence_lsn;
/* do the actual work: call callback */
- ctx->callbacks.sequence_cb(ctx, txn, sequence_lsn, rel, transactional,
+ ctx->callbacks.sequence_cb(ctx, txn, sequence_lsn, rel,
last_value, log_cnt, is_called);
/* Pop the error context stack */
@@ -1560,7 +1558,6 @@ stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
static void
stream_sequence_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr sequence_lsn, Relation rel,
- bool transactional,
int64 last_value, int64 log_cnt, bool is_called)
{
LogicalDecodingContext *ctx = cache->private_data;
@@ -1591,7 +1588,7 @@ stream_sequence_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
ctx->write_location = sequence_lsn;
/* do the actual work: call callback */
- ctx->callbacks.sequence_cb(ctx, txn, sequence_lsn, rel, transactional,
+ ctx->callbacks.sequence_cb(ctx, txn, sequence_lsn, rel,
last_value, log_cnt, is_called);
/* Pop the error context stack */
diff --git a/src/backend/replication/logical/message.c b/src/backend/replication/logical/message.c
index 1c34912610e..69178dc1cfb 100644
--- a/src/backend/replication/logical/message.c
+++ b/src/backend/replication/logical/message.c
@@ -82,7 +82,8 @@ logicalmsg_redo(XLogReaderState *record)
{
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
- if (info != XLOG_LOGICAL_MESSAGE)
+ if (info != XLOG_LOGICAL_MESSAGE &&
+ info != XLOG_LOGICAL_SEQUENCE)
elog(PANIC, "logicalmsg_redo: unknown op code %u", info);
/* This is only interesting for logical decoding, see decode.c. */
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 18d3cbb9248..ae078eed048 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -667,7 +667,7 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
*/
void
logicalrep_write_sequence(StringInfo out, Relation rel, TransactionId xid,
- XLogRecPtr lsn, bool transactional,
+ XLogRecPtr lsn,
int64 last_value, int64 log_cnt, bool is_called)
{
uint8 flags = 0;
@@ -686,7 +686,6 @@ logicalrep_write_sequence(StringInfo out, Relation rel, TransactionId xid,
relname = RelationGetRelationName(rel);
pq_sendstring(out, relname);
- pq_sendint8(out, transactional);
pq_sendint64(out, last_value);
pq_sendint64(out, log_cnt);
pq_sendint8(out, is_called);
@@ -706,7 +705,6 @@ logicalrep_read_sequence(StringInfo in, LogicalRepSequence *seqdata)
seqdata->nspname = pstrdup(logicalrep_read_namespace(in));
seqdata->seqname = pstrdup(pq_getmsgstring(in));
- seqdata->transactional = pq_getmsgint(in, 1);
seqdata->last_value = pq_getmsgint64(in);
seqdata->log_cnt = pq_getmsgint64(in);
seqdata->is_called = pq_getmsgint(in, 1);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 4702750a2e7..0cc7c8d3bc4 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -77,39 +77,7 @@
* a bit more memory to the oldest subtransactions, because it's likely
* they are the source for the next sequence of changes.
*
- * When decoding sequences, we differentiate between a sequences created
- * in a (running) transaction, and sequences created in other (already
- * committed) transactions. Changes for sequences created in the same
- * top-level transaction are treated as "transactional" i.e. just like
- * any other change from that transaction (and discarded in case of a
- * rollback). Changes for sequences created earlier are treated as not
- * transactional - are processed immediately, as if performed outside
- * any transaction (and thus not rolled back).
- *
- * This mixed behavior is necessary - sequences are non-transactional
- * (e.g. ROLLBACK does not undo the sequence increments). But for new
- * sequences, we need to handle them in a transactional way, because if
- * we ever get some DDL support, the sequence won't exist until the
- * transaction gets applied. So we need to ensure the increments don't
- * happen until the sequence gets created.
- *
- * To differentiate which sequences are "old" and which were created
- * in a still-running transaction, we track sequences created in running
- * transactions in a hash table. Sequences are identified by relfilenode,
- * and we track XID of the (sub)transaction that created it. This means
- * that if a transaction does something that changes the relfilenode
- * (like an alter / reset of a sequence), the new relfilenode will be
- * treated as if created in the transaction. The list of sequences gets
- * discarded when the transaction completes (commit/rollback).
- *
- * We don't use the XID to check if it's the same top-level transaction.
- * It's enough to know it was created in an in-progress transaction,
- * and we know it must be the current one because otherwise it wouldn't
- * see the sequence object.
- *
- * The XID may be valid even for non-transactional sequences - we simply
- * keep the XID logged to WAL, it's up to the reorderbuffer to decide if
- * the increment is transactional.
+ * FIXME
*
* -------------------------------------------------------------------------
*/
@@ -151,13 +119,6 @@ typedef struct ReorderBufferTXNByIdEnt
ReorderBufferTXN *txn;
} ReorderBufferTXNByIdEnt;
-/* entry for hash table we use to track sequences created in running xacts */
-typedef struct ReorderBufferSequenceEnt
-{
- RelFileNode rnode;
- TransactionId xid;
-} ReorderBufferSequenceEnt;
-
/* data structures for (relfilenode, ctid) => (cmin, cmax) mapping */
typedef struct ReorderBufferTupleCidKey
{
@@ -388,14 +349,6 @@ ReorderBufferAllocate(void)
buffer->by_txn = hash_create("ReorderBufferByXid", 1000, &hash_ctl,
HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
- /* hash table of sequences, mapping relfilenode to XID of transaction */
- hash_ctl.keysize = sizeof(RelFileNode);
- hash_ctl.entrysize = sizeof(ReorderBufferSequenceEnt);
- hash_ctl.hcxt = buffer->context;
-
- buffer->sequences = hash_create("ReorderBufferSequenceHash", 1000, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
-
buffer->by_txn_last_xid = InvalidTransactionId;
buffer->by_txn_last_txn = NULL;
@@ -582,17 +535,11 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change,
change->data.truncate.relids = NULL;
}
break;
- case REORDER_BUFFER_CHANGE_SEQUENCE:
- if (change->data.sequence.tuple)
- {
- ReorderBufferReturnTupleBuf(rb, change->data.sequence.tuple);
- change->data.sequence.tuple = NULL;
- }
- break;
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_SEQUENCE:
break;
}
@@ -923,57 +870,6 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
}
}
-/*
- * Treat the sequence increment as transactional?
- *
- * The hash table tracks all sequences created in in-progress transactions,
- * so we simply do a lookup (the sequence is identified by relfilende). If
- * we find a match, the increment should be handled as transactional.
- */
-bool
-ReorderBufferSequenceIsTransactional(ReorderBuffer *rb,
- RelFileNode rnode, bool created)
-{
- bool found = false;
-
- if (created)
- return true;
-
- hash_search(rb->sequences,
- (void *) &rnode,
- HASH_FIND,
- &found);
-
- return found;
-}
-
-/*
- * Cleanup sequences created in in-progress transactions.
- *
- * There's no way to search by XID, so we simply do a seqscan of all
- * the entries in the hash table. Hopefully there are only a couple
- * entries in most cases - people generally don't create many new
- * sequences over and over.
- */
-static void
-ReorderBufferSequenceCleanup(ReorderBuffer *rb, TransactionId xid)
-{
- HASH_SEQ_STATUS scan_status;
- ReorderBufferSequenceEnt *ent;
-
- hash_seq_init(&scan_status, rb->sequences);
- while ((ent = (ReorderBufferSequenceEnt *) hash_seq_search(&scan_status)) != NULL)
- {
- /* skip sequences not from this transaction */
- if (ent->xid != xid)
- continue;
-
- (void) hash_search(rb->sequences,
- (void *) &(ent->rnode),
- HASH_REMOVE, NULL);
- }
-}
-
/*
* A transactional sequence increment is queued to be processed upon commit
* and a non-transactional increment gets processed immediately.
@@ -985,166 +881,32 @@ ReorderBufferSequenceCleanup(ReorderBuffer *rb, TransactionId xid)
*/
void
ReorderBufferQueueSequence(ReorderBuffer *rb, TransactionId xid,
- Snapshot snapshot, XLogRecPtr lsn, RepOriginId origin_id,
- RelFileNode rnode, bool transactional, bool created,
- ReorderBufferTupleBuf *tuplebuf)
+ XLogRecPtr lsn, RepOriginId origin_id,
+ Oid reloid, RelFileNode rnode,
+ int64 last, int64 log_cnt, bool is_called)
{
- /*
- * Change needs to be handled as transactional, because the sequence was
- * created in a transaction that is still running. In that case all the
- * changes need to be queued in that transaction, we must not send them
- * to the downstream until the transaction commits.
- *
- * There's a bit of a trouble with subtransactions - we can't queue it
- * into the subxact, because it might be rolled back and we'd lose the
- * increment. We need to queue it into the same (sub)xact that created
- * the sequence, which is why we track the XID in the hash table.
- */
- if (transactional)
- {
- MemoryContext oldcontext;
- ReorderBufferChange *change;
-
- /* lookup sequence by relfilenode */
- ReorderBufferSequenceEnt *ent;
- bool found;
-
- /* transactional changes require a transaction */
- Assert(xid != InvalidTransactionId);
-
- /* search the lookup table (we ignore the return value, found is enough) */
- ent = hash_search(rb->sequences,
- (void *) &rnode,
- created ? HASH_ENTER : HASH_FIND,
- &found);
-
- /*
- * If this is the "create" increment, we must not have found any
- * pre-existing entry in the hash table (i.e. there must not be
- * any conflicting sequence).
- */
- Assert(!(created && found));
-
- /* But we must have either created or found an existing entry. */
- Assert(created || found);
-
- /*
- * When creating the sequence, remember the XID of the transaction
- * that created id.
- */
- if (created)
- ent->xid = xid;
-
- /* XXX Maybe check that we're still in the same top-level xact? */
-
- /* OK, allocate and queue the change */
- oldcontext = MemoryContextSwitchTo(rb->context);
-
- change = ReorderBufferGetChange(rb);
-
- change->action = REORDER_BUFFER_CHANGE_SEQUENCE;
- change->origin_id = origin_id;
-
- memcpy(&change->data.sequence.relnode, &rnode, sizeof(RelFileNode));
-
- change->data.sequence.tuple = tuplebuf;
-
- /* add it to the same subxact that created the sequence */
- ReorderBufferQueueChange(rb, ent->xid, lsn, change, false);
-
- MemoryContextSwitchTo(oldcontext);
- }
- else
- {
- /*
- * This increment is for a sequence that was not created in any
- * running transaction, so we treat it as non-transactional and
- * just send it to the output plugin directly.
- */
- ReorderBufferTXN *txn = NULL;
- volatile Snapshot snapshot_now = snapshot;
- bool using_subtxn;
-
-#ifdef USE_ASSERT_CHECKING
- /* All "creates" have to be handled as transactional. */
- Assert(!created);
-
- /* Make sure the sequence is not in the hash table. */
- {
- bool found;
- hash_search(rb->sequences,
- (void *) &rnode,
- HASH_FIND, &found);
- Assert(!found);
- }
-#endif
-
- if (xid != InvalidTransactionId)
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
-
- /* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
-
- /*
- * Decoding needs access to syscaches et al., which in turn use
- * heavyweight locks and such. Thus we need to have enough state around to
- * keep track of those. The easiest way is to simply use a transaction
- * internally. That also allows us to easily enforce that nothing writes
- * to the database by checking for xid assignments.
- *
- * When we're called via the SQL SRF there's already a transaction
- * started, so start an explicit subtransaction there.
- */
- using_subtxn = IsTransactionOrTransactionBlock();
-
- PG_TRY();
- {
- Relation relation;
- HeapTuple tuple;
- Form_pg_sequence_data seq;
- Oid reloid;
-
- if (using_subtxn)
- BeginInternalSubTransaction("sequence");
- else
- StartTransactionCommand();
-
- reloid = RelidByRelfilenode(rnode.spcNode, rnode.relNode);
-
- if (reloid == InvalidOid)
- elog(ERROR, "could not map filenode \"%s\" to relation OID",
- relpathperm(rnode,
- MAIN_FORKNUM));
-
- relation = RelationIdGetRelation(reloid);
- tuple = &tuplebuf->tuple;
- seq = (Form_pg_sequence_data) GETSTRUCT(tuple);
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
- rb->sequence(rb, txn, lsn, relation, transactional,
- seq->last_value, seq->log_cnt, seq->is_called);
+ /* OK, allocate and queue the change */
+ oldcontext = MemoryContextSwitchTo(rb->context);
- RelationClose(relation);
+ change = ReorderBufferGetChange(rb);
- TeardownHistoricSnapshot(false);
+ change->action = REORDER_BUFFER_CHANGE_SEQUENCE;
+ change->origin_id = origin_id;
- AbortCurrentTransaction();
+ memcpy(&change->data.sequence.relnode, &rnode, sizeof(RelFileNode));
- if (using_subtxn)
- RollbackAndReleaseCurrentSubTransaction();
- }
- PG_CATCH();
- {
- TeardownHistoricSnapshot(true);
+ change->data.sequence.reloid = reloid;
+ change->data.sequence.last = last;
+ change->data.sequence.log_cnt = log_cnt;
+ change->data.sequence.is_called = is_called;
- AbortCurrentTransaction();
-
- if (using_subtxn)
- RollbackAndReleaseCurrentSubTransaction();
+ /* add it to the same subxact that created the sequence */
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
- PG_RE_THROW();
- }
- PG_END_TRY();
- }
+ MemoryContextSwitchTo(oldcontext);
}
/*
@@ -1823,9 +1585,6 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
&found);
Assert(found);
- /* Remove sequences created in this transaction (if any). */
- ReorderBufferSequenceCleanup(rb, txn->xid);
-
/* remove entries spilled to disk */
if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
@@ -2249,19 +2008,23 @@ ReorderBufferApplySequence(ReorderBuffer *rb, ReorderBufferTXN *txn,
Relation relation, ReorderBufferChange *change,
bool streaming)
{
- HeapTuple tuple;
- Form_pg_sequence_data seq;
-
- tuple = &change->data.sequence.tuple->tuple;
- seq = (Form_pg_sequence_data) GETSTRUCT(tuple);
+ int64 last_value, log_cnt;
+ bool is_called;
+/*
+ * FIXME is it possible that we write sequence state for T1 before T2, but
+ * then end up committing T2 first? If the sequence could be incremented
+ * in between, that might cause data corruption result.
+ */
+ last_value = change->data.sequence.last;
+ log_cnt = change->data.sequence.log_cnt;
+ is_called = change->data.sequence.is_called;
- /* Only ever called from ReorderBufferApplySequence, so transational. */
if (streaming)
- rb->stream_sequence(rb, txn, change->lsn, relation, true,
- seq->last_value, seq->log_cnt, seq->is_called);
+ rb->stream_sequence(rb, txn, change->lsn, relation,
+ last_value, log_cnt, is_called);
else
- rb->sequence(rb, txn, change->lsn, relation, true,
- seq->last_value, seq->log_cnt, seq->is_called);
+ rb->sequence(rb, txn, change->lsn, relation,
+ last_value, log_cnt, is_called);
}
/*
@@ -2710,15 +2473,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_SEQUENCE:
Assert(snapshot_now);
+ /*
reloid = RelidByRelfilenode(change->data.sequence.relnode.spcNode,
change->data.sequence.relnode.relNode);
if (reloid == InvalidOid)
- elog(ERROR, "could not map filenode \"%s\" to relation OID",
+ elog(ERROR, "zz could not map filenode \"%s\" to relation OID",
relpathperm(change->data.sequence.relnode,
MAIN_FORKNUM));
- relation = RelationIdGetRelation(reloid);
+ */
+ relation = RelationIdGetRelation(change->data.sequence.reloid);
if (!RelationIsValid(relation))
elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
@@ -4115,45 +3880,13 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
memcpy(data, change->data.truncate.relids, size);
data += size;
- break;
- }
- case REORDER_BUFFER_CHANGE_SEQUENCE:
- {
- char *data;
- ReorderBufferTupleBuf *tup;
- Size len = 0;
-
- tup = change->data.sequence.tuple;
-
- if (tup)
- {
- sz += sizeof(HeapTupleData);
- len = tup->tuple.t_len;
- sz += len;
- }
-
- /* make sure we have enough space */
- ReorderBufferSerializeReserve(rb, sz);
-
- data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange);
- /* might have been reallocated above */
- ondisk = (ReorderBufferDiskChange *) rb->outbuf;
-
- if (len)
- {
- memcpy(data, &tup->tuple, sizeof(HeapTupleData));
- data += sizeof(HeapTupleData);
-
- memcpy(data, tup->tuple.t_data, len);
- data += len;
- }
-
break;
}
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_SEQUENCE:
/* ReorderBufferChange contains everything important */
break;
}
@@ -4412,28 +4145,13 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
{
sz += sizeof(Oid) * change->data.truncate.nrelids;
- break;
- }
- case REORDER_BUFFER_CHANGE_SEQUENCE:
- {
- ReorderBufferTupleBuf *tup;
- Size len = 0;
-
- tup = change->data.sequence.tuple;
-
- if (tup)
- {
- sz += sizeof(HeapTupleData);
- len = tup->tuple.t_len;
- sz += len;
- }
-
break;
}
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_SEQUENCE:
/* ReorderBufferChange contains everything important */
break;
}
@@ -4729,34 +4447,11 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
break;
}
-
- case REORDER_BUFFER_CHANGE_SEQUENCE:
- if (change->data.sequence.tuple)
- {
- uint32 tuplelen = ((HeapTuple) data)->t_len;
-
- change->data.sequence.tuple =
- ReorderBufferGetTupleBuf(rb, tuplelen - SizeofHeapTupleHeader);
-
- /* restore ->tuple */
- memcpy(&change->data.sequence.tuple->tuple, data,
- sizeof(HeapTupleData));
- data += sizeof(HeapTupleData);
-
- /* reset t_data pointer into the new tuplebuf */
- change->data.sequence.tuple->tuple.t_data =
- ReorderBufferTupleBufData(change->data.sequence.tuple);
-
- /* restore tuple data itself */
- memcpy(change->data.sequence.tuple->tuple.t_data, data, tuplelen);
- data += tuplelen;
- }
- break;
-
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_SEQUENCE:
break;
}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 697fb23634c..d6adbd6e46d 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -1220,8 +1220,7 @@ copy_sequence(Relation rel)
fetch_sequence_data(lrel.nspname, lrel.relname, &last_value, &log_cnt, &is_called);
- /* tablesync sets the sequences in non-transactional way */
- SetSequence(RelationGetRelid(rel), false, last_value, log_cnt, is_called);
+ SetSequence(RelationGetRelid(rel), last_value, log_cnt, is_called);
logicalrep_rel_close(relmapentry, NoLock);
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f3868b3e1f8..fbc65f277ea 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1159,12 +1159,9 @@ apply_handle_sequence(StringInfo s)
logicalrep_read_sequence(s, &seq);
/*
- * Non-transactional sequence updates should not be part of a remote
- * transaction. There should not be any running transaction.
+ * We should be in remote transaction.
*/
- Assert((!seq.transactional) || in_remote_transaction);
- Assert(!(!seq.transactional && in_remote_transaction));
- Assert(!(!seq.transactional && IsTransactionState()));
+ Assert(in_remote_transaction);
/*
* Make sure we're in a transaction (needed by SetSequence). For
@@ -1185,14 +1182,7 @@ apply_handle_sequence(StringInfo s)
LockRelationOid(relid, AccessExclusiveLock);
/* apply the sequence change */
- SetSequence(relid, seq.transactional, seq.last_value, seq.log_cnt, seq.is_called);
-
- /*
- * Commit the per-stream transaction (we only do this when not in
- * remote transaction, i.e. for non-transactional sequence updates.
- */
- if (!in_remote_transaction)
- CommitTransactionCommand();
+ SetSequence(relid, seq.last_value, seq.log_cnt, seq.is_called);
}
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 9d33630464c..31be8064ebe 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -57,7 +57,7 @@ static void pgoutput_message(LogicalDecodingContext *ctx,
Size sz, const char *message);
static void pgoutput_sequence(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr sequence_lsn,
- Relation relation, bool transactional,
+ Relation relation,
int64 last_value, int64 log_cnt, bool is_called);
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
@@ -1712,12 +1712,13 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
static void
pgoutput_sequence(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr sequence_lsn,
- Relation relation, bool transactional,
+ Relation relation,
int64 last_value, int64 log_cnt, bool is_called)
{
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
TransactionId xid = InvalidTransactionId;
RelationSyncEntry *relentry;
+ PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
if (!data->sequences)
return;
@@ -1742,25 +1743,15 @@ pgoutput_sequence(LogicalDecodingContext *ctx,
if (!relentry->pubactions.pubsequence)
return;
- /*
- * Output BEGIN if we haven't yet. Avoid for non-transactional
- * sequence changes.
- */
- if (transactional)
- {
- PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
-
- /* Send BEGIN if we haven't yet */
- if (txndata && !txndata->sent_begin_txn)
- pgoutput_send_begin(ctx, txn);
- }
+ /* Send BEGIN if we haven't yet */
+ if (txndata && !txndata->sent_begin_txn)
+ pgoutput_send_begin(ctx, txn);
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_sequence(ctx->out,
relation,
xid,
sequence_lsn,
- transactional,
last_value,
log_cnt,
is_called);
diff --git a/src/include/access/rmgrlist.h b/src/include/access/rmgrlist.h
index cf8b6d48193..9a74721c97c 100644
--- a/src/include/access/rmgrlist.h
+++ b/src/include/access/rmgrlist.h
@@ -40,7 +40,7 @@ PG_RMGR(RM_BTREE_ID, "Btree", btree_redo, btree_desc, btree_identify, btree_xlog
PG_RMGR(RM_HASH_ID, "Hash", hash_redo, hash_desc, hash_identify, NULL, NULL, hash_mask, NULL)
PG_RMGR(RM_GIN_ID, "Gin", gin_redo, gin_desc, gin_identify, gin_xlog_startup, gin_xlog_cleanup, gin_mask, NULL)
PG_RMGR(RM_GIST_ID, "Gist", gist_redo, gist_desc, gist_identify, gist_xlog_startup, gist_xlog_cleanup, gist_mask, NULL)
-PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask, sequence_decode)
+PG_RMGR(RM_SEQ_ID, "Sequence", seq_redo, seq_desc, seq_identify, NULL, NULL, seq_mask, NULL)
PG_RMGR(RM_SPGIST_ID, "SPGist", spg_redo, spg_desc, spg_identify, spg_xlog_startup, spg_xlog_cleanup, spg_mask, NULL)
PG_RMGR(RM_BRIN_ID, "BRIN", brin_redo, brin_desc, brin_identify, NULL, NULL, brin_mask, NULL)
PG_RMGR(RM_COMMIT_TS_ID, "CommitTs", commit_ts_redo, commit_ts_desc, commit_ts_identify, NULL, NULL, NULL, NULL)
diff --git a/src/include/commands/sequence.h b/src/include/commands/sequence.h
index 5bab90db8e0..0bc90c922ba 100644
--- a/src/include/commands/sequence.h
+++ b/src/include/commands/sequence.h
@@ -48,7 +48,6 @@ typedef FormData_pg_sequence_data *Form_pg_sequence_data;
typedef struct xl_seq_rec
{
RelFileNode node;
- bool created; /* creates a new relfilenode (CREATE/ALTER) */
/* SEQUENCE TUPLE DATA FOLLOWS AT THE END */
} xl_seq_rec;
@@ -60,9 +59,11 @@ extern ObjectAddress DefineSequence(ParseState *pstate, CreateSeqStmt *stmt);
extern ObjectAddress AlterSequence(ParseState *pstate, AlterSeqStmt *stmt);
extern void DeleteSequenceTuple(Oid relid);
extern void ResetSequence(Oid seq_relid);
-extern void SetSequence(Oid seq_relid, bool transactional, int64 last_value, int64 log_cnt, bool is_called);
+extern void SetSequence(Oid seq_relid, int64 last_value, int64 log_cnt, bool is_called);
extern void ResetSequenceCaches(void);
+extern void AtEOXact_Sequences(bool isCommit);
+
extern void seq_redo(XLogReaderState *rptr);
extern void seq_desc(StringInfo buf, XLogReaderState *rptr);
extern const char *seq_identify(uint8 info);
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 13ee10fdd4e..a466db394b8 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -125,7 +125,6 @@ typedef struct LogicalRepSequence
Oid remoteid; /* unique id of the remote sequence */
char *nspname; /* schema name of remote sequence */
char *seqname; /* name of the remote sequence */
- bool transactional;
int64 last_value;
int64 log_cnt;
bool is_called;
@@ -245,7 +244,6 @@ extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecP
bool transactional, const char *prefix, Size sz, const char *message);
extern void logicalrep_write_sequence(StringInfo out, Relation rel,
TransactionId xid, XLogRecPtr lsn,
- bool transactional,
int64 last_value, int64 log_cnt,
bool is_called);
extern void logicalrep_read_sequence(StringInfo in, LogicalRepSequence *seqdata);
diff --git a/src/include/replication/message.h b/src/include/replication/message.h
index 7d7785292f1..a084acc06ab 100644
--- a/src/include/replication/message.h
+++ b/src/include/replication/message.h
@@ -27,13 +27,28 @@ typedef struct xl_logical_message
char message[FLEXIBLE_ARRAY_MEMBER];
} xl_logical_message;
+/*
+ * Generic logical decoding sequence wal record.
+ */
+typedef struct xl_logical_sequence
+{
+ RelFileNode node;
+ Oid reloid;
+ int64 last; /* last value emitted for sequence */
+ int64 log_cnt; /* last value cached for sequence */
+ bool is_called;
+} xl_logical_sequence;
+
#define SizeOfLogicalMessage (offsetof(xl_logical_message, message))
+#define SizeOfLogicalSequence (sizeof(xl_logical_sequence))
extern XLogRecPtr LogLogicalMessage(const char *prefix, const char *message,
size_t size, bool transactional);
/* RMGR API*/
#define XLOG_LOGICAL_MESSAGE 0x00
+#define XLOG_LOGICAL_SEQUENCE 0x10
+
void logicalmsg_redo(XLogReaderState *record);
void logicalmsg_desc(StringInfo buf, XLogReaderState *record);
const char *logicalmsg_identify(uint8 info);
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index fe85d49a030..abc05644516 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -95,7 +95,6 @@ typedef void (*LogicalDecodeSequenceCB) (struct LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
XLogRecPtr sequence_lsn,
Relation rel,
- bool transactional,
int64 last_value,
int64 log_cnt,
bool is_called);
@@ -219,7 +218,6 @@ typedef void (*LogicalDecodeStreamSequenceCB) (struct LogicalDecodingContext *ct
ReorderBufferTXN *txn,
XLogRecPtr sequence_lsn,
Relation rel,
- bool transactional,
int64 last_value,
int64 log_cnt,
bool is_called);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0bcc150b331..260630c2ba5 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -164,7 +164,10 @@ typedef struct ReorderBufferChange
struct
{
RelFileNode relnode;
- ReorderBufferTupleBuf *tuple;
+ Oid reloid;
+ int64 last;
+ int64 log_cnt;
+ bool is_called;
} sequence;
} data;
@@ -443,7 +446,6 @@ typedef void (*ReorderBufferSequenceCB) (ReorderBuffer *rb,
ReorderBufferTXN *txn,
XLogRecPtr sequence_lsn,
Relation rel,
- bool transactional,
int64 last_value, int64 log_cnt,
bool is_called);
@@ -518,7 +520,6 @@ typedef void (*ReorderBufferStreamSequenceCB) (ReorderBuffer *rb,
ReorderBufferTXN *txn,
XLogRecPtr sequence_lsn,
Relation rel,
- bool transactional,
int64 last_value, int64 log_cnt,
bool is_called);
@@ -670,9 +671,9 @@ void ReorderBufferQueueMessage(ReorderBuffer *, TransactionId, Snapshot snapsho
bool transactional, const char *prefix,
Size message_size, const char *message);
void ReorderBufferQueueSequence(ReorderBuffer *rb, TransactionId xid,
- Snapshot snapshot, XLogRecPtr lsn, RepOriginId origin_id,
- RelFileNode rnode, bool transactional, bool created,
- ReorderBufferTupleBuf *tuplebuf);
+ XLogRecPtr lsn, RepOriginId origin_id,
+ Oid reloid, RelFileNode rnode,
+ int64 last, int64 log_cnt, bool is_called);
void ReorderBufferCommit(ReorderBuffer *, TransactionId,
XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
TimestampTz commit_time, RepOriginId origin_id, XLogRecPtr origin_lsn);
@@ -720,7 +721,4 @@ void ReorderBufferSetRestartPoint(ReorderBuffer *, XLogRecPtr ptr);
void StartupReorderBuffer(void);
-bool ReorderBufferSequenceIsTransactional(ReorderBuffer *rb,
- RelFileNode rnode, bool created);
-
#endif
--
2.34.1
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
@ 2022-04-07 18:34 ` Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
1 sibling, 1 reply; 33+ messages in thread
From: Tomas Vondra @ 2022-04-07 18:34 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 4/6/22 16:13, Tomas Vondra wrote:
>
>
> On 4/5/22 12:06, Amit Kapila wrote:
>> On Mon, Apr 4, 2022 at 3:10 AM Tomas Vondra
>> <[email protected]> wrote:
>>>
>>> I did some experiments over the weekend, exploring how to rework the
>>> sequence decoding in various ways. Let me share some WIP patches,
>>> hopefully that can be useful for trying more stuff and moving this
>>> discussion forward.
>>>
>>> I tried two things - (1) accumulating sequence increments in global
>>> array and then doing something with it, and (2) treating all sequence
>>> increments as regular changes (in a TXN) and then doing something
>>> special during the replay. Attached are two patchsets, one for each
>>> approach.
>>>
>>> Note: It's important to remember decoding of sequences is not the only
>>> code affected by this. The logical messages have the same issue,
>>> certainly when it comes to transactional vs. non-transactional stuff and
>>> handling of snapshots. Even if the sequence decoding ends up being
>>> reverted, we still need to fix that, somehow. And my feeling is the
>>> solutions ought to be pretty similar in both cases.
>>>
>>> Now, regarding the two approaches:
>>>
>>> (1) accumulating sequences in global hash table
>>>
>>> The main problem with regular sequence increments is that those need to
>>> be non-transactional - a transaction may use a sequence without any
>>> WAL-logging, if the WAL was written by an earlier transaction. The
>>> problem is the earlier trasaction might have been rolled back, and thus
>>> simply discarded by the logical decoding. But we still need to apply
>>> that, in order not to lose the sequence increment.
>>>
>>> The current code just applies those non-transactional increments right
>>> after decoding the increment, but that does not work because we may not
>>> have a snapshot at that point. And we only have the snapshot when within
>>> a transaction (AFAICS) so this queues all changes and then applies the
>>> changes later.
>>>
>>> The changes need to be shared by all transactions, so queueing them in a
>>> global works fairly well - otherwise we'd have to walk all transactions,
>>> in order to see if there are relevant sequence increments.
>>>
>>> But some increments may be transactional, e.g. when the sequence is
>>> created or altered in a transaction. To allow tracking this, this uses a
>>> hash table, with relfilenode as a key.
>>>
>>> There's a couple issues with this, though. Firstly, stashing the changes
>>> outside transactions, it's not included in memory accounting, it's not
>>> spilled to disk or streamed, etc. I guess fixing this is possible, but
>>> it's certainly not straightforward, because we mix increments from many
>>> different transactions.
>>>
>>> A bigger issue is that I'm not sure this actually handles the snapshots
>>> correctly either.
>>>
>>> The non-transactional increments affect all transactions, so when
>>> ReorderBufferProcessSequences gets executed, it processes all of them,
>>> no matter the source transaction. Can we be sure the snapshot in the
>>> applying transaction is the same (or "compatible") as the snapshot in
>>> the source transaction?
>>>
>>
>> I don't think we can assume that. I think it is possible that some
>> other transaction's WAL can be in-between start/end lsn of txn (which
>> we decide to send) which may not finally reach a consistent state.
>> Consider a case similar to shown in one of my previous emails:
>> Session-2:
>> Begin;
>> SELECT pg_current_xact_id();
>>
>> Session-1:
>> SELECT 'init' FROM pg_create_logical_replication_slot('test_slot',
>> 'test_decoding', false, true);
>>
>> Session-3:
>> Begin;
>> SELECT pg_current_xact_id();
>>
>> Session-2:
>> Commit;
>> Begin;
>> INSERT INTO t1_seq SELECT nextval('seq1') FROM generate_series(1,100);
>>
>> Session-3:
>> Commit;
>>
>> Session-2:
>> Commit;
>>
>> Here, we send changes (say insert from txn 700) from session-2 because
>> session-3's commit happens before it. Now, consider another
>> transaction parallel to txn 700 which generates some WAL related to
>> sequences but it committed before session-3's commit. So though, its
>> changes will be the in-between start/end LSN of txn 700 but those
>> shouldn't be sent.
>>
>> I have not tried this and also this may be solvable in some way but I
>> think processing changes from other TXNs sounds risky to me in terms
>> of snapshot handling.
>>
>
> Yes, I know this can happen. I was only really thinking about what might
> happen to the relfilenode of the sequence itself - and I don't think any
> concurrent transaction could swoop in and change the relfilenode in any
> meaningful way, due to locking.
>
> But of course, if we expect/require to have a perfect snapshot for that
> exact position in the transaction, this won't work. IMO the whole idea
> that we can have non-transactional bits in naturally transactional
> decoding seems a bit suspicious (at least in hindsight).
>
> No matter what we do for sequences, though, this still affects logical
> messages too. Not sure what to do there :-(
>
>>>
>>>
>>> (2) treating sequence change as regular changes
>>>
>>> This adopts a different approach - instead of accumulating the sequence
>>> increments in a global hash table, it treats them as regular changes.
>>> Which solves the snapshot issue, and issues with spilling to disk,
>>> streaming and so on.
>>>
>>> But it has various other issues with handling concurrent transactions,
>>> unfortunately, which probably make this approach infeasible:
>>>
>>> * The non-transactional stuff has to be applied in the first transaction
>>> that commits, not in the transaction that generated the WAL. That does
>>> not work too well with this approach, because we have to walk changes in
>>> all other transactions.
>>>
>>
>> Why do you want to traverse other TXNs in this approach? Is it because
>> the current TXN might be using some value of sequence which has been
>> actually WAL logged in the other transaction but that other
>> transaction has not been sent yet? I think if we don't send that then
>> probably replica sequences columns (in some tables) have some values
>> but actually the sequence itself won't have still that value which
>> sounds problematic. Is that correct?
>>
>
> Well, how else would you get to sequence changes in the other TXNs?
>
> Consider this:
>
> T1: begin
> T2: begin
>
> T2: nextval('s') -> writes WAL for 32 values
> T1: nextval('s') -> gets value without WAL
>
> T1: commit
> T2: commit
>
> Now, if we commit T1 without "applying" the sequence change from T2, we
> loose the sequence state. But we still write/replicate the value
> generated from the sequence.
>
>>> * Another serious issue seems to be streaming - if we already streamed
>>> some of the changes, we can't iterate through them anymore.
>>>
>>> Also, having to walk the transactions over and over for each change, to
>>> apply relevant sequence increments, that's mighty expensive. The other
>>> approach needs to do that too, but walking the global hash table seems
>>> much cheaper.
>>>
>>> The other issue this handling of aborted transactions - we need to apply
>>> sequence increments even from those transactions, of course. The other
>>> approach has this issue too, though.
>>>
>>>
>>> (3) tracking sequences touched by transaction
>>>
>>> This is the approach proposed by Hannu Krosing. I haven't explored this
>>> again yet, but I recall I wrote a PoC patch a couple months back.
>>>
>>> It seems to me most of the problems stems from trying to derive sequence
>>> state from decoded WAL changes, which is problematic because of the
>>> non-transactional nature of sequences (i.e. WAL for one transaction
>>> affects other transactions in non-obvious ways). And this approach
>>> simply works around that entirely - instead of trying to deduce the
>>> sequence state from WAL, we'd make sure to write the current sequence
>>> state (or maybe just ID of the sequence) at commit time. Which should
>>> eliminate most of the complexity / problems, I think.
>>>
>>
>> That sounds promising but I haven't thought in detail about that approach.
>>
>
> So, here's a patch doing that. It's a reworked/improved version of the
> patch [1] shared in November.
>
> It seems to be working pretty nicely. The behavior is a little bit
> different, of course, because we only replicate "committed" changes, so
> if you do nextval() in aborted transaction that is not replicated. Which
> I think is fine, because we generally make no durability guarantees for
> aborted transactions in general.
>
> But there are a couple issues too:
>
> 1) locking
>
> We have to read sequence change before the commit, but we must not allow
> reordering (because then the state might go backwards again). I'm not
> sure how serious impact could this have on performance.
>
> 2) dropped sequences
>
> I'm not sure what to do about sequences dropped in the transaction. The
> patch simply attempts to read the current sequence state before the
> commit, but if the sequence was dropped (in that transaction), that
> can't happen. I'm not sure if that's OK or not.
>
> 3) WAL record
>
> To replicate the stuff the patch uses a LogicalMessage, but I guess a
> separate WAL record would be better. But that's a technical detail.
>
>
> regards
>
> [1]
> https://www.postgresql.org/message-id/[email protected]
>
>>>
>>> I'm not really sure what to do about this. All of those reworks seems
>>> like an extensive redesign of the patch, and considering the last CF is
>>> already over ... not great.
>>>
>>
>> Yeah, I share the same feeling that even if we devise solutions to all
>> the known problems it requires quite some time to ensure everything is
>> correct.
>>
>
> True. Let's keep working on this for a bit more time and then we can
> decide what to do.
>
I've pushed a revert af all the commits related to this - decoding of
sequences and test_decoding / built-in replication changes. The approach
combining transactional and non-transactional behavior implemented by
the patch clearly has issues, and it seems foolish to hope we'll find a
simple fix. So the changes would have to be much more extensive, and
doing that after the last CF seems like an obviously bad idea.
Attached is a rebased patch, implementing the approach based on
WAL-logging sequences at commit time.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
[text/x-patch] decoding-sequences-tracking-20220407.patch (576.1K, ../../[email protected]/2-decoding-sequences-tracking-20220407.patch)
download | inline diff:
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index b2209064790..36929dd97d3 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -5,7 +5,8 @@ PGFILEDESC = "test_decoding - example of a logical decoding output plugin"
REGRESS = ddl xact rewrite toast permissions decoding_in_xact \
decoding_into_rel binary prepared replorigin time messages \
- spill slot truncate stream stats twophase twophase_stream
+ spill slot truncate stream stats twophase twophase_stream \
+ sequence
ISOLATION = mxact delayed_startup ondisk_startup concurrent_ddl_dml \
oldest_xmin snapshot_transfer subxact_without_top concurrent_stream \
twophase_snapshot slot_creation_error
diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out
index 9a28b5ddc5a..1e37c8c8979 100644
--- a/contrib/test_decoding/expected/ddl.out
+++ b/contrib/test_decoding/expected/ddl.out
@@ -40,7 +40,7 @@ SELECT 'init' FROM pg_create_physical_replication_slot('repl');
init
(1 row)
-SELECT data FROM pg_logical_slot_get_changes('repl', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('repl', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
ERROR: cannot use physical replication slot for logical decoding
SELECT pg_drop_replication_slot('repl');
pg_drop_replication_slot
@@ -89,7 +89,7 @@ COMMIT;
ALTER TABLE replication_example RENAME COLUMN text TO somenum;
INSERT INTO replication_example(somedata, somenum) VALUES (4, 1);
-- collect all changes
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
---------------------------------------------------------------------------------------------------------------------------
BEGIN
@@ -118,7 +118,7 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
ALTER TABLE replication_example ALTER COLUMN somenum TYPE int4 USING (somenum::int4);
-- check that this doesn't produce any changes from the heap rewrite
-SELECT count(data) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT count(data) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
count
-------
0
@@ -134,7 +134,7 @@ INSERT INTO replication_example(somedata, somenum, zaphod2) VALUES (6, 3, 1);
INSERT INTO replication_example(somedata, somenum, zaphod1) VALUES (6, 4, 2);
COMMIT;
-- show changes
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
@@ -154,7 +154,7 @@ INSERT INTO replication_example(id, somedata, somenum) SELECT i, i, i FROM gener
ON CONFLICT (id) DO UPDATE SET somenum = excluded.somenum + 1;
COMMIT;
/* display results */
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
--------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
@@ -242,7 +242,7 @@ CREATE TABLE tr_unique(id2 serial unique NOT NULL, data int);
INSERT INTO tr_unique(data) VALUES(10);
ALTER TABLE tr_unique RENAME TO tr_pkey;
ALTER TABLE tr_pkey ADD COLUMN id serial primary key;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-rewrites', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-rewrites', '1', 'include-sequences', '0');
data
-----------------------------------------------------------------------------
BEGIN
@@ -257,7 +257,7 @@ INSERT INTO tr_pkey(data) VALUES(1);
--show deletion with primary key
DELETE FROM tr_pkey;
/* display results */
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
----------------------------------------------------------------------------
BEGIN
@@ -288,7 +288,7 @@ INSERT INTO tr_oddlength VALUES('ab', 'foo');
COMMIT;
/* display results, but hide most of the output */
SELECT count(*), min(data), max(data)
-FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')
+FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0')
GROUP BY substring(data, 1, 24)
ORDER BY 1,2;
count | min | max
@@ -310,7 +310,7 @@ DELETE FROM spoolme;
DROP TABLE spoolme;
COMMIT;
SELECT data
-FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')
+FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0')
WHERE data ~ 'UPDATE';
data
-------------------------------------------------------------------------------------------------------------
@@ -324,7 +324,7 @@ INSERT INTO tr_etoomuch (id, data)
SELECT g.i, -g.i FROM generate_series(8000, 12000) g(i)
ON CONFLICT(id) DO UPDATE SET data = EXCLUDED.data;
SELECT substring(data, 1, 29), count(*)
-FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1') WITH ORDINALITY
+FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0') WITH ORDINALITY
GROUP BY 1
ORDER BY min(ordinality);
substring | count
@@ -355,7 +355,7 @@ RELEASE SAVEPOINT c;
INSERT INTO tr_sub(path) VALUES ('1-top-2-#1');
RELEASE SAVEPOINT b;
COMMIT;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
----------------------------------------------------------------------
BEGIN
@@ -394,7 +394,7 @@ INSERT INTO tr_sub(path) VALUES ('2-top-1...--#3');
RELEASE SAVEPOINT subtop;
INSERT INTO tr_sub(path) VALUES ('2-top-#1');
COMMIT;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
------------------------------------------------------------------------
BEGIN
@@ -415,7 +415,7 @@ INSERT INTO tr_sub(path) VALUES ('3-top-2-2-#1');
ROLLBACK TO SAVEPOINT b;
INSERT INTO tr_sub(path) VALUES ('3-top-2-#2');
COMMIT;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
-----------------------------------------------------------------------
BEGIN
@@ -444,7 +444,7 @@ BEGIN;
SAVEPOINT a;
INSERT INTO tr_sub(path) VALUES ('5-top-1-#1');
COMMIT;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
---------------------------------------------------------------------
BEGIN
@@ -465,7 +465,7 @@ ROLLBACK TO SAVEPOINT a;
ALTER TABLE tr_sub_ddl ALTER COLUMN data TYPE bigint;
INSERT INTO tr_sub_ddl VALUES(43);
COMMIT;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
--------------------------------------------------
BEGIN
@@ -542,7 +542,7 @@ Options: user_catalog_table=false
INSERT INTO replication_metadata(relation, options)
VALUES ('zaphod', NULL);
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
------------------------------------------------------------------------------------------------------------------------------------
BEGIN
@@ -659,7 +659,7 @@ INSERT INTO toasttable(toasted_col2) SELECT repeat(string_agg(to_char(g.i, 'FM00
UPDATE toasttable
SET toasted_col1 = (SELECT string_agg(g.i::text, '') FROM generate_series(1, 2000) g(i))
WHERE id = 1;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
@@ -814,7 +814,7 @@ UPDATE toasttable
WHERE id = 1;
-- make sure we decode correctly even if the toast table is gone
DROP TABLE toasttable;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
@@ -826,7 +826,7 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
(6 rows)
-- done, free logical replication slot
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
------
(0 rows)
diff --git a/contrib/test_decoding/expected/decoding_in_xact.out b/contrib/test_decoding/expected/decoding_in_xact.out
index b65253f4630..0816c780fe1 100644
--- a/contrib/test_decoding/expected/decoding_in_xact.out
+++ b/contrib/test_decoding/expected/decoding_in_xact.out
@@ -58,7 +58,7 @@ SELECT pg_current_xact_id() = '0';
-- don't show yet, haven't committed
INSERT INTO nobarf(data) VALUES('2');
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
-----------------------------------------------------------
BEGIN
@@ -68,7 +68,7 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
COMMIT;
INSERT INTO nobarf(data) VALUES('3');
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
-----------------------------------------------------------
BEGIN
diff --git a/contrib/test_decoding/expected/decoding_into_rel.out b/contrib/test_decoding/expected/decoding_into_rel.out
index 8fd3390066d..03966b8b1ca 100644
--- a/contrib/test_decoding/expected/decoding_into_rel.out
+++ b/contrib/test_decoding/expected/decoding_into_rel.out
@@ -19,7 +19,7 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
CREATE TABLE somechange(id serial primary key);
INSERT INTO somechange DEFAULT VALUES;
CREATE TABLE changeresult AS
- SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
SELECT * FROM changeresult;
data
------------------------------------------------
@@ -29,9 +29,9 @@ SELECT * FROM changeresult;
(3 rows)
INSERT INTO changeresult
- SELECT data FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ SELECT data FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
INSERT INTO changeresult
- SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
SELECT * FROM changeresult;
data
--------------------------------------------------------------------------------------------------------------------------------------------------
@@ -63,7 +63,7 @@ DROP TABLE somechange;
CREATE FUNCTION slot_changes_wrapper(slot_name name) RETURNS SETOF TEXT AS $$
BEGIN
RETURN QUERY
- SELECT data FROM pg_logical_slot_peek_changes(slot_name, NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ SELECT data FROM pg_logical_slot_peek_changes(slot_name, NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
END$$ LANGUAGE plpgsql;
SELECT * FROM slot_changes_wrapper('regression_slot');
slot_changes_wrapper
@@ -84,7 +84,7 @@ SELECT * FROM slot_changes_wrapper('regression_slot');
COMMIT
(14 rows)
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
diff --git a/contrib/test_decoding/expected/mxact.out b/contrib/test_decoding/expected/mxact.out
index 03ad3df0999..c5bc26c8044 100644
--- a/contrib/test_decoding/expected/mxact.out
+++ b/contrib/test_decoding/expected/mxact.out
@@ -7,7 +7,7 @@ step s0init: SELECT 'init' FROM pg_create_logical_replication_slot('isolation_sl
init
(1 row)
-step s0start: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false');
+step s0start: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false', 'include-sequences', 'false');
data
----
(0 rows)
@@ -27,7 +27,7 @@ t
(1 row)
step s0w: INSERT INTO do_write DEFAULT VALUES;
-step s0start: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false');
+step s0start: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false', 'include-sequences', 'false');
data
--------------------------------------------
BEGIN
@@ -50,7 +50,7 @@ step s0init: SELECT 'init' FROM pg_create_logical_replication_slot('isolation_sl
init
(1 row)
-step s0start: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false');
+step s0start: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false', 'include-sequences', 'false');
data
----
(0 rows)
@@ -71,7 +71,7 @@ t
step s0alter: ALTER TABLE do_write ADD column ts timestamptz;
step s0w: INSERT INTO do_write DEFAULT VALUES;
-step s0start: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false');
+step s0start: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false', 'include-sequences', 'false');
data
------------------------------------------------------------------------------
BEGIN
diff --git a/contrib/test_decoding/expected/ondisk_startup.out b/contrib/test_decoding/expected/ondisk_startup.out
index bc7ff071648..3d2fa4a92e0 100644
--- a/contrib/test_decoding/expected/ondisk_startup.out
+++ b/contrib/test_decoding/expected/ondisk_startup.out
@@ -35,7 +35,7 @@ init
step s2c: COMMIT;
step s1insert: INSERT INTO do_write DEFAULT VALUES;
step s1checkpoint: CHECKPOINT;
-step s1start: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false');
+step s1start: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false', 'include-sequences', 'false');
data
--------------------------------------------------------------------
BEGIN
@@ -46,7 +46,7 @@ COMMIT
step s1insert: INSERT INTO do_write DEFAULT VALUES;
step s1alter: ALTER TABLE do_write ADD COLUMN addedbys1 int;
step s1insert: INSERT INTO do_write DEFAULT VALUES;
-step s1start: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false');
+step s1start: SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false', 'include-sequences', 'false');
data
--------------------------------------------------------------------------------------------
BEGIN
diff --git a/contrib/test_decoding/expected/replorigin.out b/contrib/test_decoding/expected/replorigin.out
index 2e9ef7c823b..7468c24f2b4 100644
--- a/contrib/test_decoding/expected/replorigin.out
+++ b/contrib/test_decoding/expected/replorigin.out
@@ -72,9 +72,9 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_d
-- origin tx
INSERT INTO origin_tbl(data) VALUES ('will be replicated and decoded and decoded again');
INSERT INTO target_tbl(data)
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
-- as is normal, the insert into target_tbl shows up
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
@@ -110,7 +110,7 @@ SELECT pg_replication_origin_xact_setup('0/aabbccdd', '2013-01-01 00:00');
(1 row)
INSERT INTO target_tbl(data)
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local', '1', 'include-sequences', '0');
COMMIT;
-- check replication progress for the session is correct
SELECT pg_replication_origin_session_progress(false);
@@ -154,14 +154,14 @@ SELECT pg_replication_origin_progress('regress_test_decoding: regression_slot',
SELECT pg_replication_origin_session_reset();
ERROR: no replication origin is configured
-- and magically the replayed xact will be filtered!
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local', '1', 'include-sequences', '0');
data
------
(0 rows)
--but new original changes still show up
INSERT INTO origin_tbl(data) VALUES ('will be replicated');
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local', '1', 'include-sequences', '0');
data
--------------------------------------------------------------------------------
BEGIN
@@ -227,7 +227,7 @@ SELECT local_id, external_id,
1 | regress_test_decoding: regression_slot_no_lsn | f | t
(1 row)
-SELECT data FROM pg_logical_slot_get_changes('regression_slot_no_lsn', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot_no_lsn', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0', 'include-sequences', '0');
data
-------------------------------------------------------------------------------------
BEGIN
diff --git a/contrib/test_decoding/expected/rewrite.out b/contrib/test_decoding/expected/rewrite.out
index b30999c436b..5d15b192edc 100644
--- a/contrib/test_decoding/expected/rewrite.out
+++ b/contrib/test_decoding/expected/rewrite.out
@@ -64,7 +64,7 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_d
CREATE TABLE replication_example(id SERIAL PRIMARY KEY, somedata int, text varchar(120));
INSERT INTO replication_example(somedata) VALUES (1);
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
----------------------------------------------------------------------------------------------------------
BEGIN
@@ -115,7 +115,7 @@ INSERT INTO replication_example(somedata, testcolumn1, testcolumn3) VALUES (7, 5
COMMIT;
-- make old files go away
CHECKPOINT;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
@@ -141,7 +141,7 @@ VACUUM FULL pg_proc; VACUUM FULL pg_description; VACUUM FULL pg_shdescription; V
INSERT INTO replication_example(somedata, testcolumn1, testcolumn3) VALUES (8, 6, 1);
VACUUM FULL pg_proc; VACUUM FULL pg_description; VACUUM FULL pg_shdescription; VACUUM FULL iamalargetable;
INSERT INTO replication_example(somedata, testcolumn1, testcolumn3) VALUES (9, 7, 1);
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
diff --git a/contrib/test_decoding/expected/sequence.out b/contrib/test_decoding/expected/sequence.out
new file mode 100644
index 00000000000..dc1bf670122
--- /dev/null
+++ b/contrib/test_decoding/expected/sequence.out
@@ -0,0 +1,325 @@
+-- predictability
+SET synchronous_commit = on;
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+ ?column?
+----------
+ init
+(1 row)
+
+CREATE SEQUENCE test_sequence;
+-- test the sequence changes by several nextval() calls
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 1
+(1 row)
+
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 2
+(1 row)
+
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 3
+(1 row)
+
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 4
+(1 row)
+
+-- test the sequence changes by several ALTER commands
+ALTER SEQUENCE test_sequence INCREMENT BY 10;
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 14
+(1 row)
+
+ALTER SEQUENCE test_sequence START WITH 3000;
+ALTER SEQUENCE test_sequence MAXVALUE 10000;
+ALTER SEQUENCE test_sequence RESTART WITH 4000;
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 4000
+(1 row)
+
+-- test the sequence changes by several setval() calls
+SELECT setval('test_sequence', 3500);
+ setval
+--------
+ 3500
+(1 row)
+
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 3510
+(1 row)
+
+SELECT setval('test_sequence', 3500, true);
+ setval
+--------
+ 3500
+(1 row)
+
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 3510
+(1 row)
+
+SELECT setval('test_sequence', 3500, false);
+ setval
+--------
+ 3500
+(1 row)
+
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 3500
+(1 row)
+
+-- show results and drop sequence
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+----------------------------------------------------------------------------------------
+ BEGIN
+ sequence public.test_sequence: transactional:1 last_value: 1 log_cnt: 0 is_called:0
+ COMMIT
+ sequence public.test_sequence: transactional:0 last_value: 33 log_cnt: 0 is_called:1
+ BEGIN
+ sequence public.test_sequence: transactional:1 last_value: 4 log_cnt: 0 is_called:1
+ COMMIT
+ sequence public.test_sequence: transactional:0 last_value: 334 log_cnt: 0 is_called:1
+ BEGIN
+ sequence public.test_sequence: transactional:1 last_value: 14 log_cnt: 32 is_called:1
+ COMMIT
+ BEGIN
+ sequence public.test_sequence: transactional:1 last_value: 14 log_cnt: 0 is_called:1
+ COMMIT
+ BEGIN
+ sequence public.test_sequence: transactional:1 last_value: 4000 log_cnt: 0 is_called:0
+ COMMIT
+ sequence public.test_sequence: transactional:0 last_value: 4320 log_cnt: 0 is_called:1
+ sequence public.test_sequence: transactional:0 last_value: 3500 log_cnt: 0 is_called:1
+ sequence public.test_sequence: transactional:0 last_value: 3830 log_cnt: 0 is_called:1
+ sequence public.test_sequence: transactional:0 last_value: 3500 log_cnt: 0 is_called:1
+ sequence public.test_sequence: transactional:0 last_value: 3830 log_cnt: 0 is_called:1
+ sequence public.test_sequence: transactional:0 last_value: 3500 log_cnt: 0 is_called:0
+ sequence public.test_sequence: transactional:0 last_value: 3820 log_cnt: 0 is_called:1
+(24 rows)
+
+DROP SEQUENCE test_sequence;
+-- rollback on sequence creation and update
+BEGIN;
+CREATE SEQUENCE test_sequence;
+CREATE TABLE test_table (a INT);
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 1
+(1 row)
+
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 2
+(1 row)
+
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 3
+(1 row)
+
+SELECT setval('test_sequence', 3000);
+ setval
+--------
+ 3000
+(1 row)
+
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 3001
+(1 row)
+
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 3002
+(1 row)
+
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 3003
+(1 row)
+
+ALTER SEQUENCE test_sequence RESTART WITH 6000;
+INSERT INTO test_table VALUES( (SELECT nextval('test_sequence')) );
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 6001
+(1 row)
+
+ROLLBACK;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+------
+(0 rows)
+
+-- rollback on table creation with serial column
+BEGIN;
+CREATE TABLE test_table (a SERIAL, b INT);
+INSERT INTO test_table (b) VALUES (100);
+INSERT INTO test_table (b) VALUES (200);
+INSERT INTO test_table (b) VALUES (300);
+SELECT setval('test_table_a_seq', 3000);
+ setval
+--------
+ 3000
+(1 row)
+
+INSERT INTO test_table (b) VALUES (400);
+INSERT INTO test_table (b) VALUES (500);
+INSERT INTO test_table (b) VALUES (600);
+ALTER SEQUENCE test_table_a_seq RESTART WITH 6000;
+INSERT INTO test_table (b) VALUES (700);
+INSERT INTO test_table (b) VALUES (800);
+INSERT INTO test_table (b) VALUES (900);
+ROLLBACK;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+------
+(0 rows)
+
+-- rollback on table with serial column
+CREATE TABLE test_table (a SERIAL, b INT);
+BEGIN;
+INSERT INTO test_table (b) VALUES (100);
+INSERT INTO test_table (b) VALUES (200);
+INSERT INTO test_table (b) VALUES (300);
+SELECT setval('test_table_a_seq', 3000);
+ setval
+--------
+ 3000
+(1 row)
+
+INSERT INTO test_table (b) VALUES (400);
+INSERT INTO test_table (b) VALUES (500);
+INSERT INTO test_table (b) VALUES (600);
+ALTER SEQUENCE test_table_a_seq RESTART WITH 6000;
+INSERT INTO test_table (b) VALUES (700);
+INSERT INTO test_table (b) VALUES (800);
+INSERT INTO test_table (b) VALUES (900);
+ROLLBACK;
+-- check table and sequence values after rollback
+SELECT * from test_table_a_seq;
+ last_value | log_cnt | is_called
+------------+---------+-----------
+ 3003 | 30 | t
+(1 row)
+
+SELECT nextval('test_table_a_seq');
+ nextval
+---------
+ 3004
+(1 row)
+
+DROP TABLE test_table;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-------------------------------------------------------------------------------------------
+ BEGIN
+ sequence public.test_table_a_seq: transactional:1 last_value: 1 log_cnt: 0 is_called:0
+ COMMIT
+ sequence public.test_table_a_seq: transactional:0 last_value: 33 log_cnt: 0 is_called:1
+ sequence public.test_table_a_seq: transactional:0 last_value: 3000 log_cnt: 0 is_called:1
+ sequence public.test_table_a_seq: transactional:0 last_value: 3033 log_cnt: 0 is_called:1
+(6 rows)
+
+-- savepoint test on table with serial column
+BEGIN;
+CREATE TABLE test_table (a SERIAL, b INT);
+INSERT INTO test_table (b) VALUES (100);
+INSERT INTO test_table (b) VALUES (200);
+SAVEPOINT a;
+INSERT INTO test_table (b) VALUES (300);
+ROLLBACK TO SAVEPOINT a;
+DROP TABLE test_table;
+COMMIT;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+-----------------------------------------------------------------------------------------
+ BEGIN
+ sequence public.test_table_a_seq: transactional:1 last_value: 1 log_cnt: 0 is_called:0
+ sequence public.test_table_a_seq: transactional:1 last_value: 33 log_cnt: 0 is_called:1
+ table public.test_table: INSERT: a[integer]:1 b[integer]:100
+ table public.test_table: INSERT: a[integer]:2 b[integer]:200
+ COMMIT
+(6 rows)
+
+-- savepoint test on table with serial column
+BEGIN;
+CREATE SEQUENCE test_sequence;
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 1
+(1 row)
+
+SELECT setval('test_sequence', 3000);
+ setval
+--------
+ 3000
+(1 row)
+
+SELECT nextval('test_sequence');
+ nextval
+---------
+ 3001
+(1 row)
+
+SAVEPOINT a;
+ALTER SEQUENCE test_sequence START WITH 7000;
+SELECT setval('test_sequence', 5000);
+ setval
+--------
+ 5000
+(1 row)
+
+ROLLBACK TO SAVEPOINT a;
+SELECT * FROM test_sequence;
+ last_value | log_cnt | is_called
+------------+---------+-----------
+ 3001 | 32 | t
+(1 row)
+
+DROP SEQUENCE test_sequence;
+COMMIT;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ data
+----------------------------------------------------------------------------------------
+ BEGIN
+ sequence public.test_sequence: transactional:1 last_value: 1 log_cnt: 0 is_called:0
+ sequence public.test_sequence: transactional:1 last_value: 33 log_cnt: 0 is_called:1
+ sequence public.test_sequence: transactional:1 last_value: 3000 log_cnt: 0 is_called:1
+ sequence public.test_sequence: transactional:1 last_value: 3033 log_cnt: 0 is_called:1
+ COMMIT
+(6 rows)
+
+SELECT pg_drop_replication_slot('regression_slot');
+ pg_drop_replication_slot
+--------------------------
+
+(1 row)
+
diff --git a/contrib/test_decoding/expected/slot.out b/contrib/test_decoding/expected/slot.out
index 63a9940f73a..de59d544ae8 100644
--- a/contrib/test_decoding/expected/slot.out
+++ b/contrib/test_decoding/expected/slot.out
@@ -95,7 +95,7 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot2', 'test_
(1 row)
INSERT INTO replication_example(somedata, text) VALUES (1, 3);
-SELECT data FROM pg_logical_slot_get_changes('regression_slot1', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot1', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
---------------------------------------------------------------------------------------------------------
BEGIN
@@ -107,7 +107,7 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot1', NULL, NULL, 'in
COMMIT
(7 rows)
-SELECT data FROM pg_logical_slot_get_changes('regression_slot2', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot2', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
---------------------------------------------------------------------------------------------------------
BEGIN
@@ -132,7 +132,7 @@ SELECT :'wal_lsn' = :'end_lsn';
t
(1 row)
-SELECT data FROM pg_logical_slot_get_changes('regression_slot1', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot1', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
---------------------------------------------------------------------------------------------------------
BEGIN
@@ -140,7 +140,7 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot1', NULL, NULL, 'in
COMMIT
(3 rows)
-SELECT data FROM pg_logical_slot_get_changes('regression_slot2', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot2', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
------
(0 rows)
diff --git a/contrib/test_decoding/expected/toast.out b/contrib/test_decoding/expected/toast.out
index a757e7dc8d5..d0ab0e06394 100644
--- a/contrib/test_decoding/expected/toast.out
+++ b/contrib/test_decoding/expected/toast.out
@@ -52,7 +52,7 @@ CREATE TABLE toasted_copy (
);
ALTER TABLE toasted_copy ALTER COLUMN data SET STORAGE EXTERNAL;
\copy toasted_copy FROM STDIN
-SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
substr
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
@@ -316,7 +316,7 @@ SELECT pg_column_size(toasted_key) > 2^16 FROM toasted_several;
t
(1 row)
-SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
regexp_replace
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
@@ -327,7 +327,7 @@ SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_sl
-- test update of a toasted key without changing it
UPDATE toasted_several SET toasted_col1 = toasted_key;
UPDATE toasted_several SET toasted_col2 = toasted_col1;
-SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
regexp_replace
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
@@ -350,7 +350,7 @@ UPDATE toasted_several SET toasted_col1 = toasted_col2 WHERE id = 1;
DELETE FROM toasted_several WHERE id = 1;
COMMIT;
DROP TABLE toasted_several;
-SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')
+SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0')
WHERE data NOT LIKE '%INSERT: %';
regexp_replace
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
@@ -373,7 +373,7 @@ INSERT INTO tbl1 VALUES(1, repeat('a', 4000)) ;
ALTER TABLE tbl1 ADD COLUMN id serial primary key;
INSERT INTO tbl2 VALUES(1);
commit;
-SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
substr
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
BEGIN
diff --git a/contrib/test_decoding/expected/truncate.out b/contrib/test_decoding/expected/truncate.out
index e64d377214a..961b0ebdc89 100644
--- a/contrib/test_decoding/expected/truncate.out
+++ b/contrib/test_decoding/expected/truncate.out
@@ -11,7 +11,7 @@ CREATE TABLE tab2 (a int primary key, b int);
TRUNCATE tab1;
TRUNCATE tab1, tab1 RESTART IDENTITY CASCADE;
TRUNCATE tab1, tab2;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
data
------------------------------------------------------
BEGIN
diff --git a/contrib/test_decoding/specs/mxact.spec b/contrib/test_decoding/specs/mxact.spec
index ea5b1aa2d67..19f3af14b30 100644
--- a/contrib/test_decoding/specs/mxact.spec
+++ b/contrib/test_decoding/specs/mxact.spec
@@ -13,7 +13,7 @@ teardown
session "s0"
setup { SET synchronous_commit=on; }
step "s0init" {SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot', 'test_decoding');}
-step "s0start" {SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false');}
+step "s0start" {SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false', 'include-sequences', 'false');}
step "s0alter" {ALTER TABLE do_write ADD column ts timestamptz; }
step "s0w" { INSERT INTO do_write DEFAULT VALUES; }
diff --git a/contrib/test_decoding/specs/ondisk_startup.spec b/contrib/test_decoding/specs/ondisk_startup.spec
index 96ce87f1a45..fb43cbc5e15 100644
--- a/contrib/test_decoding/specs/ondisk_startup.spec
+++ b/contrib/test_decoding/specs/ondisk_startup.spec
@@ -16,7 +16,7 @@ session "s1"
setup { SET synchronous_commit=on; }
step "s1init" {SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot', 'test_decoding');}
-step "s1start" {SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false');}
+step "s1start" {SELECT data FROM pg_logical_slot_get_changes('isolation_slot', NULL, NULL, 'include-xids', 'false', 'include-sequences', 'false');}
step "s1insert" { INSERT INTO do_write DEFAULT VALUES; }
step "s1checkpoint" { CHECKPOINT; }
step "s1alter" { ALTER TABLE do_write ADD COLUMN addedbys1 int; }
diff --git a/contrib/test_decoding/sql/ddl.sql b/contrib/test_decoding/sql/ddl.sql
index 4f76bed72c1..807bc56a451 100644
--- a/contrib/test_decoding/sql/ddl.sql
+++ b/contrib/test_decoding/sql/ddl.sql
@@ -19,7 +19,7 @@ SELECT pg_drop_replication_slot('regression_slot');
-- check that we're detecting a streaming rep slot used for logical decoding
SELECT 'init' FROM pg_create_physical_replication_slot('repl');
-SELECT data FROM pg_logical_slot_get_changes('repl', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('repl', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
SELECT pg_drop_replication_slot('repl');
@@ -64,11 +64,11 @@ ALTER TABLE replication_example RENAME COLUMN text TO somenum;
INSERT INTO replication_example(somedata, somenum) VALUES (4, 1);
-- collect all changes
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
ALTER TABLE replication_example ALTER COLUMN somenum TYPE int4 USING (somenum::int4);
-- check that this doesn't produce any changes from the heap rewrite
-SELECT count(data) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT count(data) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
INSERT INTO replication_example(somedata, somenum) VALUES (5, 1);
@@ -82,7 +82,7 @@ INSERT INTO replication_example(somedata, somenum, zaphod1) VALUES (6, 4, 2);
COMMIT;
-- show changes
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
-- ON CONFLICT DO UPDATE support
BEGIN;
@@ -91,7 +91,7 @@ INSERT INTO replication_example(id, somedata, somenum) SELECT i, i, i FROM gener
COMMIT;
/* display results */
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
-- MERGE support
BEGIN;
@@ -113,14 +113,14 @@ CREATE TABLE tr_unique(id2 serial unique NOT NULL, data int);
INSERT INTO tr_unique(data) VALUES(10);
ALTER TABLE tr_unique RENAME TO tr_pkey;
ALTER TABLE tr_pkey ADD COLUMN id serial primary key;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-rewrites', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-rewrites', '1', 'include-sequences', '0');
INSERT INTO tr_pkey(data) VALUES(1);
--show deletion with primary key
DELETE FROM tr_pkey;
/* display results */
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
/*
* check that disk spooling works (also for logical messages)
@@ -137,7 +137,7 @@ COMMIT;
/* display results, but hide most of the output */
SELECT count(*), min(data), max(data)
-FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')
+FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0')
GROUP BY substring(data, 1, 24)
ORDER BY 1,2;
@@ -152,7 +152,7 @@ DROP TABLE spoolme;
COMMIT;
SELECT data
-FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')
+FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0')
WHERE data ~ 'UPDATE';
-- check that a large, spooled, upsert works
@@ -161,7 +161,7 @@ SELECT g.i, -g.i FROM generate_series(8000, 12000) g(i)
ON CONFLICT(id) DO UPDATE SET data = EXCLUDED.data;
SELECT substring(data, 1, 29), count(*)
-FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1') WITH ORDINALITY
+FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0') WITH ORDINALITY
GROUP BY 1
ORDER BY min(ordinality);
@@ -189,7 +189,7 @@ INSERT INTO tr_sub(path) VALUES ('1-top-2-#1');
RELEASE SAVEPOINT b;
COMMIT;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
-- check that we handle xlog assignments correctly
BEGIN;
@@ -218,7 +218,7 @@ RELEASE SAVEPOINT subtop;
INSERT INTO tr_sub(path) VALUES ('2-top-#1');
COMMIT;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
-- make sure rollbacked subtransactions aren't decoded
BEGIN;
@@ -231,7 +231,7 @@ ROLLBACK TO SAVEPOINT b;
INSERT INTO tr_sub(path) VALUES ('3-top-2-#2');
COMMIT;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
-- test whether a known, but not yet logged toplevel xact, followed by a
-- subxact commit is handled correctly
@@ -250,7 +250,7 @@ INSERT INTO tr_sub(path) VALUES ('5-top-1-#1');
COMMIT;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
-- check that DDL in aborted subtransactions handled correctly
CREATE TABLE tr_sub_ddl(data int);
@@ -263,7 +263,7 @@ ALTER TABLE tr_sub_ddl ALTER COLUMN data TYPE bigint;
INSERT INTO tr_sub_ddl VALUES(43);
COMMIT;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
/*
@@ -303,7 +303,7 @@ ALTER TABLE replication_metadata SET (user_catalog_table = false);
INSERT INTO replication_metadata(relation, options)
VALUES ('zaphod', NULL);
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
/*
* check whether we handle updates/deletes correct with & without a pkey
@@ -414,7 +414,7 @@ UPDATE toasttable
SET toasted_col1 = (SELECT string_agg(g.i::text, '') FROM generate_series(1, 2000) g(i))
WHERE id = 1;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
INSERT INTO toasttable(toasted_col1) SELECT string_agg(g.i::text, '') FROM generate_series(1, 2000) g(i);
@@ -426,10 +426,10 @@ WHERE id = 1;
-- make sure we decode correctly even if the toast table is gone
DROP TABLE toasttable;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
-- done, free logical replication slot
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
SELECT pg_drop_replication_slot('regression_slot');
diff --git a/contrib/test_decoding/sql/decoding_in_xact.sql b/contrib/test_decoding/sql/decoding_in_xact.sql
index 108782dc2e9..b343b745661 100644
--- a/contrib/test_decoding/sql/decoding_in_xact.sql
+++ b/contrib/test_decoding/sql/decoding_in_xact.sql
@@ -32,10 +32,10 @@ BEGIN;
SELECT pg_current_xact_id() = '0';
-- don't show yet, haven't committed
INSERT INTO nobarf(data) VALUES('2');
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
COMMIT;
INSERT INTO nobarf(data) VALUES('3');
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
diff --git a/contrib/test_decoding/sql/decoding_into_rel.sql b/contrib/test_decoding/sql/decoding_into_rel.sql
index 1068cec5888..27d5d08879e 100644
--- a/contrib/test_decoding/sql/decoding_into_rel.sql
+++ b/contrib/test_decoding/sql/decoding_into_rel.sql
@@ -15,14 +15,14 @@ CREATE TABLE somechange(id serial primary key);
INSERT INTO somechange DEFAULT VALUES;
CREATE TABLE changeresult AS
- SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
SELECT * FROM changeresult;
INSERT INTO changeresult
- SELECT data FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ SELECT data FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
INSERT INTO changeresult
- SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
SELECT * FROM changeresult;
DROP TABLE changeresult;
@@ -32,11 +32,11 @@ DROP TABLE somechange;
CREATE FUNCTION slot_changes_wrapper(slot_name name) RETURNS SETOF TEXT AS $$
BEGIN
RETURN QUERY
- SELECT data FROM pg_logical_slot_peek_changes(slot_name, NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+ SELECT data FROM pg_logical_slot_peek_changes(slot_name, NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
END$$ LANGUAGE plpgsql;
SELECT * FROM slot_changes_wrapper('regression_slot');
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
SELECT 'stop' FROM pg_drop_replication_slot('regression_slot');
diff --git a/contrib/test_decoding/sql/replorigin.sql b/contrib/test_decoding/sql/replorigin.sql
index 2e28a487773..cd0a370208e 100644
--- a/contrib/test_decoding/sql/replorigin.sql
+++ b/contrib/test_decoding/sql/replorigin.sql
@@ -41,10 +41,10 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_d
-- origin tx
INSERT INTO origin_tbl(data) VALUES ('will be replicated and decoded and decoded again');
INSERT INTO target_tbl(data)
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
-- as is normal, the insert into target_tbl shows up
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
INSERT INTO origin_tbl(data) VALUES ('will be replicated, but not decoded again');
@@ -60,7 +60,7 @@ BEGIN;
-- setup transaction origin
SELECT pg_replication_origin_xact_setup('0/aabbccdd', '2013-01-01 00:00');
INSERT INTO target_tbl(data)
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local', '1', 'include-sequences', '0');
COMMIT;
-- check replication progress for the session is correct
@@ -79,11 +79,11 @@ SELECT pg_replication_origin_progress('regress_test_decoding: regression_slot',
SELECT pg_replication_origin_session_reset();
-- and magically the replayed xact will be filtered!
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local', '1', 'include-sequences', '0');
--but new original changes still show up
INSERT INTO origin_tbl(data) VALUES ('will be replicated');
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'only-local', '1', 'include-sequences', '0');
SELECT pg_drop_replication_slot('regression_slot');
SELECT pg_replication_origin_drop('regress_test_decoding: regression_slot');
@@ -114,7 +114,7 @@ SELECT local_id, external_id,
remote_lsn <> '0/0' AS valid_remote_lsn,
local_lsn <> '0/0' AS valid_local_lsn
FROM pg_replication_origin_status;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot_no_lsn', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot_no_lsn', NULL, NULL, 'skip-empty-xacts', '1', 'include-xids', '0', 'include-sequences', '0');
-- Clean up
SELECT pg_replication_origin_session_reset();
SELECT pg_drop_replication_slot('regression_slot_no_lsn');
diff --git a/contrib/test_decoding/sql/rewrite.sql b/contrib/test_decoding/sql/rewrite.sql
index 62dead3a9b1..1715bd289dd 100644
--- a/contrib/test_decoding/sql/rewrite.sql
+++ b/contrib/test_decoding/sql/rewrite.sql
@@ -35,7 +35,7 @@ SELECT pg_relation_size((SELECT reltoastrelid FROM pg_class WHERE oid = 'pg_shde
SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
CREATE TABLE replication_example(id SERIAL PRIMARY KEY, somedata int, text varchar(120));
INSERT INTO replication_example(somedata) VALUES (1);
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
BEGIN;
INSERT INTO replication_example(somedata) VALUES (2);
@@ -90,7 +90,7 @@ COMMIT;
-- make old files go away
CHECKPOINT;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
-- trigger repeated rewrites of a system catalog with a toast table,
-- that previously was buggy: [email protected]
@@ -98,7 +98,7 @@ VACUUM FULL pg_proc; VACUUM FULL pg_description; VACUUM FULL pg_shdescription; V
INSERT INTO replication_example(somedata, testcolumn1, testcolumn3) VALUES (8, 6, 1);
VACUUM FULL pg_proc; VACUUM FULL pg_description; VACUUM FULL pg_shdescription; VACUUM FULL iamalargetable;
INSERT INTO replication_example(somedata, testcolumn1, testcolumn3) VALUES (9, 7, 1);
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
SELECT pg_drop_replication_slot('regression_slot');
DROP TABLE IF EXISTS replication_example;
diff --git a/contrib/test_decoding/sql/sequence.sql b/contrib/test_decoding/sql/sequence.sql
new file mode 100644
index 00000000000..978ee8088e1
--- /dev/null
+++ b/contrib/test_decoding/sql/sequence.sql
@@ -0,0 +1,119 @@
+-- predictability
+SET synchronous_commit = on;
+SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot', 'test_decoding');
+
+CREATE SEQUENCE test_sequence;
+
+-- test the sequence changes by several nextval() calls
+SELECT nextval('test_sequence');
+SELECT nextval('test_sequence');
+SELECT nextval('test_sequence');
+SELECT nextval('test_sequence');
+
+-- test the sequence changes by several ALTER commands
+ALTER SEQUENCE test_sequence INCREMENT BY 10;
+SELECT nextval('test_sequence');
+
+ALTER SEQUENCE test_sequence START WITH 3000;
+ALTER SEQUENCE test_sequence MAXVALUE 10000;
+ALTER SEQUENCE test_sequence RESTART WITH 4000;
+SELECT nextval('test_sequence');
+
+-- test the sequence changes by several setval() calls
+SELECT setval('test_sequence', 3500);
+SELECT nextval('test_sequence');
+SELECT setval('test_sequence', 3500, true);
+SELECT nextval('test_sequence');
+SELECT setval('test_sequence', 3500, false);
+SELECT nextval('test_sequence');
+
+-- show results and drop sequence
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+DROP SEQUENCE test_sequence;
+
+-- rollback on sequence creation and update
+BEGIN;
+CREATE SEQUENCE test_sequence;
+CREATE TABLE test_table (a INT);
+SELECT nextval('test_sequence');
+SELECT nextval('test_sequence');
+SELECT nextval('test_sequence');
+SELECT setval('test_sequence', 3000);
+SELECT nextval('test_sequence');
+SELECT nextval('test_sequence');
+SELECT nextval('test_sequence');
+ALTER SEQUENCE test_sequence RESTART WITH 6000;
+INSERT INTO test_table VALUES( (SELECT nextval('test_sequence')) );
+SELECT nextval('test_sequence');
+ROLLBACK;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- rollback on table creation with serial column
+BEGIN;
+CREATE TABLE test_table (a SERIAL, b INT);
+INSERT INTO test_table (b) VALUES (100);
+INSERT INTO test_table (b) VALUES (200);
+INSERT INTO test_table (b) VALUES (300);
+SELECT setval('test_table_a_seq', 3000);
+INSERT INTO test_table (b) VALUES (400);
+INSERT INTO test_table (b) VALUES (500);
+INSERT INTO test_table (b) VALUES (600);
+ALTER SEQUENCE test_table_a_seq RESTART WITH 6000;
+INSERT INTO test_table (b) VALUES (700);
+INSERT INTO test_table (b) VALUES (800);
+INSERT INTO test_table (b) VALUES (900);
+ROLLBACK;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- rollback on table with serial column
+CREATE TABLE test_table (a SERIAL, b INT);
+
+BEGIN;
+INSERT INTO test_table (b) VALUES (100);
+INSERT INTO test_table (b) VALUES (200);
+INSERT INTO test_table (b) VALUES (300);
+SELECT setval('test_table_a_seq', 3000);
+INSERT INTO test_table (b) VALUES (400);
+INSERT INTO test_table (b) VALUES (500);
+INSERT INTO test_table (b) VALUES (600);
+ALTER SEQUENCE test_table_a_seq RESTART WITH 6000;
+INSERT INTO test_table (b) VALUES (700);
+INSERT INTO test_table (b) VALUES (800);
+INSERT INTO test_table (b) VALUES (900);
+ROLLBACK;
+
+-- check table and sequence values after rollback
+SELECT * from test_table_a_seq;
+SELECT nextval('test_table_a_seq');
+
+DROP TABLE test_table;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- savepoint test on table with serial column
+BEGIN;
+CREATE TABLE test_table (a SERIAL, b INT);
+INSERT INTO test_table (b) VALUES (100);
+INSERT INTO test_table (b) VALUES (200);
+SAVEPOINT a;
+INSERT INTO test_table (b) VALUES (300);
+ROLLBACK TO SAVEPOINT a;
+DROP TABLE test_table;
+COMMIT;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+-- savepoint test on table with serial column
+BEGIN;
+CREATE SEQUENCE test_sequence;
+SELECT nextval('test_sequence');
+SELECT setval('test_sequence', 3000);
+SELECT nextval('test_sequence');
+SAVEPOINT a;
+ALTER SEQUENCE test_sequence START WITH 7000;
+SELECT setval('test_sequence', 5000);
+ROLLBACK TO SAVEPOINT a;
+SELECT * FROM test_sequence;
+DROP SEQUENCE test_sequence;
+COMMIT;
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+
+SELECT pg_drop_replication_slot('regression_slot');
diff --git a/contrib/test_decoding/sql/slot.sql b/contrib/test_decoding/sql/slot.sql
index 1aa27c56674..52a740c43de 100644
--- a/contrib/test_decoding/sql/slot.sql
+++ b/contrib/test_decoding/sql/slot.sql
@@ -50,8 +50,8 @@ SELECT 'init' FROM pg_create_logical_replication_slot('regression_slot2', 'test_
INSERT INTO replication_example(somedata, text) VALUES (1, 3);
-SELECT data FROM pg_logical_slot_get_changes('regression_slot1', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
-SELECT data FROM pg_logical_slot_get_changes('regression_slot2', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot1', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot2', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
INSERT INTO replication_example(somedata, text) VALUES (1, 4);
INSERT INTO replication_example(somedata, text) VALUES (1, 5);
@@ -65,8 +65,8 @@ SELECT slot_name FROM pg_replication_slot_advance('regression_slot2', pg_current
SELECT :'wal_lsn' = :'end_lsn';
-SELECT data FROM pg_logical_slot_get_changes('regression_slot1', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
-SELECT data FROM pg_logical_slot_get_changes('regression_slot2', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot1', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot2', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
DROP TABLE replication_example;
diff --git a/contrib/test_decoding/sql/toast.sql b/contrib/test_decoding/sql/toast.sql
index d1c560a174d..f5d4fc082ed 100644
--- a/contrib/test_decoding/sql/toast.sql
+++ b/contrib/test_decoding/sql/toast.sql
@@ -264,7 +264,7 @@ ALTER TABLE toasted_copy ALTER COLUMN data SET STORAGE EXTERNAL;
202 untoasted199
203 untoasted200
\.
-SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
-- test we can decode "old" tuples bigger than the max heap tuple size correctly
DROP TABLE IF EXISTS toasted_several;
@@ -287,13 +287,13 @@ UPDATE pg_attribute SET attstorage = 'x' WHERE attrelid = 'toasted_several_pkey'
INSERT INTO toasted_several(toasted_key) VALUES(repeat('9876543210', 10000));
SELECT pg_column_size(toasted_key) > 2^16 FROM toasted_several;
-SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_peek_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
-- test update of a toasted key without changing it
UPDATE toasted_several SET toasted_col1 = toasted_key;
UPDATE toasted_several SET toasted_col2 = toasted_col1;
-SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
/*
* update with large tuplebuf, in a transaction large enough to force to spool to disk
@@ -306,7 +306,7 @@ COMMIT;
DROP TABLE toasted_several;
-SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')
+SELECT regexp_replace(data, '^(.{100}).*(.{100})$', '\1..\2') FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0')
WHERE data NOT LIKE '%INSERT: %';
/*
@@ -322,6 +322,6 @@ INSERT INTO tbl1 VALUES(1, repeat('a', 4000)) ;
ALTER TABLE tbl1 ADD COLUMN id serial primary key;
INSERT INTO tbl2 VALUES(1);
commit;
-SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT substr(data, 1, 200) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
SELECT pg_drop_replication_slot('regression_slot');
diff --git a/contrib/test_decoding/sql/truncate.sql b/contrib/test_decoding/sql/truncate.sql
index 5633854e0df..b0f36fd681b 100644
--- a/contrib/test_decoding/sql/truncate.sql
+++ b/contrib/test_decoding/sql/truncate.sql
@@ -10,5 +10,5 @@ TRUNCATE tab1;
TRUNCATE tab1, tab1 RESTART IDENTITY CASCADE;
TRUNCATE tab1, tab2;
-SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
+SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'include-sequences', '0');
SELECT pg_drop_replication_slot('regression_slot');
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 08d366a594e..851117b4155 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -35,6 +35,7 @@ typedef struct
bool include_timestamp;
bool skip_empty_xacts;
bool only_local;
+ bool include_sequences;
} TestDecodingData;
/*
@@ -76,6 +77,10 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_sequence(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr sequence_lsn,
+ Relation rel,
+ int64 last_value, int64 log_cnt, bool is_called);
static bool pg_decode_filter_prepare(LogicalDecodingContext *ctx,
TransactionId xid,
const char *gid);
@@ -116,6 +121,10 @@ static void pg_decode_stream_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_sequence(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr sequence_lsn,
+ Relation rel,
+ int64 last_value, int64 log_cnt, bool is_called);
static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
int nrelations, Relation relations[],
@@ -141,6 +150,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->sequence_cb = pg_decode_sequence;
cb->filter_prepare_cb = pg_decode_filter_prepare;
cb->begin_prepare_cb = pg_decode_begin_prepare_txn;
cb->prepare_cb = pg_decode_prepare_txn;
@@ -153,6 +163,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->stream_commit_cb = pg_decode_stream_commit;
cb->stream_change_cb = pg_decode_stream_change;
cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_sequence_cb = pg_decode_stream_sequence;
cb->stream_truncate_cb = pg_decode_stream_truncate;
}
@@ -173,6 +184,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
data->include_xids = true;
data->include_timestamp = false;
data->skip_empty_xacts = false;
+ data->include_sequences = true;
data->only_local = false;
ctx->output_plugin_private = data;
@@ -265,6 +277,17 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
errmsg("could not parse value \"%s\" for parameter \"%s\"",
strVal(elem->arg), elem->defname)));
}
+ else if (strcmp(elem->defname, "include-sequences") == 0)
+ {
+
+ if (elem->arg == NULL)
+ data->include_sequences = false;
+ else if (!parse_bool(strVal(elem->arg), &data->include_sequences))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("could not parse value \"%s\" for parameter \"%s\"",
+ strVal(elem->arg), elem->defname)));
+ }
else
{
ereport(ERROR,
@@ -756,6 +779,34 @@ pg_decode_message(LogicalDecodingContext *ctx,
OutputPluginWrite(ctx, true);
}
+static void
+pg_decode_sequence(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ XLogRecPtr sequence_lsn, Relation rel,
+ int64 last_value, int64 log_cnt, bool is_called)
+{
+ TestDecodingData *data = ctx->output_plugin_private;
+ TestDecodingTxnData *txndata = txn->output_plugin_private;
+
+ if (!data->include_sequences)
+ return;
+
+ /* output BEGIN if we haven't yet, but only for the transactional case */
+ if (data->skip_empty_xacts && !txndata->xact_wrote_changes)
+ {
+ pg_output_begin(ctx, data, txn, false);
+ }
+ txndata->xact_wrote_changes = true;
+
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfoString(ctx->out, "sequence ");
+ appendStringInfoString(ctx->out,
+ quote_qualified_identifier(get_namespace_name(get_rel_namespace(RelationGetRelid(rel))),
+ RelationGetRelationName(rel)));
+ appendStringInfo(ctx->out, ": last_value: %zu log_cnt: %zu is_called:%d",
+ last_value, log_cnt, is_called);
+ OutputPluginWrite(ctx, true);
+}
+
static void
pg_decode_stream_start(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn)
@@ -955,6 +1006,36 @@ pg_decode_stream_message(LogicalDecodingContext *ctx,
OutputPluginWrite(ctx, true);
}
+static void
+pg_decode_stream_sequence(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ XLogRecPtr sequence_lsn, Relation rel,
+ int64 last_value, int64 log_cnt, bool is_called)
+{
+ TestDecodingData *data = ctx->output_plugin_private;
+ TestDecodingTxnData *txndata = txn->output_plugin_private;
+
+ if (!data->include_sequences)
+ return;
+
+ /* output BEGIN if we haven't yet, but only for the transactional case */
+ {
+ if (data->skip_empty_xacts && !txndata->xact_wrote_changes)
+ {
+ pg_output_begin(ctx, data, txn, false);
+ }
+ txndata->xact_wrote_changes = true;
+ }
+
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfoString(ctx->out, "streaming sequence ");
+ appendStringInfoString(ctx->out,
+ quote_qualified_identifier(get_namespace_name(get_rel_namespace(RelationGetRelid(rel))),
+ RelationGetRelationName(rel)));
+ appendStringInfo(ctx->out, ": last_value: " INT64_FORMAT " log_cnt: " INT64_FORMAT " is_called:%d",
+ last_value, log_cnt, is_called);
+ OutputPluginWrite(ctx, true);
+}
+
/*
* In streaming mode, we don't display the detailed information of Truncate.
* See pg_decode_stream_change.
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index e8f850a9c07..646ab74d049 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6354,6 +6354,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
Reference to schema
</para></entry>
</row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pntype</structfield> <type>char</type>
+ Determines which object type is included from this schema.
+ </para>
+ <para>
+ Reference to schema
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -9689,6 +9699,11 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>prepared transactions</entry>
</row>
+ <row>
+ <entry><link linkend="view-pg-publication-sequences"><structname>pg_publication_sequences</structname></link></entry>
+ <entry>publications and their associated sequences</entry>
+ </row>
+
<row>
<entry><link linkend="view-pg-publication-tables"><structname>pg_publication_tables</structname></link></entry>
<entry>publications and their associated tables</entry>
@@ -11626,6 +11641,72 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</sect1>
+ <sect1 id="view-pg-publication-sequences">
+ <title><structname>pg_publication_sequences</structname></title>
+
+ <indexterm zone="view-pg-publication-sequences">
+ <primary>pg_publication_sequences</primary>
+ </indexterm>
+
+ <para>
+ The view <structname>pg_publication_sequences</structname> provides
+ information about the mapping between publications and the sequences they
+ contain. Unlike the underlying catalog
+ <link linkend="catalog-pg-publication-rel"><structname>pg_publication_rel</structname></link>,
+ this view expands
+ publications defined as <literal>FOR ALL SEQUENCES</literal>, so for such
+ publications there will be a row for each eligible sequence.
+ </para>
+
+ <table>
+ <title><structname>pg_publication_sequences</structname> Columns</title>
+ <tgroup cols="1">
+ <thead>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ Column Type
+ </para>
+ <para>
+ Description
+ </para></entry>
+ </row>
+ </thead>
+
+ <tbody>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>pubname</structfield> <type>name</type>
+ (references <link linkend="catalog-pg-publication"><structname>pg_publication</structname></link>.<structfield>pubname</structfield>)
+ </para>
+ <para>
+ Name of publication
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>schemaname</structfield> <type>name</type>
+ (references <link linkend="catalog-pg-namespace"><structname>pg_namespace</structname></link>.<structfield>nspname</structfield>)
+ </para>
+ <para>
+ Name of schema containing sequence
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>sequencename</structfield> <type>name</type>
+ (references <link linkend="catalog-pg-class"><structname>pg_class</structname></link>.<structfield>relname</structfield>)
+ </para>
+ <para>
+ Name of sequence
+ </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ </sect1>
+
<sect1 id="view-pg-publication-tables">
<title><structname>pg_publication_tables</structname></title>
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8b2c31a87fa..a6ea6ff3fcf 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -458,6 +458,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeTruncateCB truncate_cb;
LogicalDecodeCommitCB commit_cb;
LogicalDecodeMessageCB message_cb;
+ LogicalDecodeSequenceCB sequence_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
LogicalDecodeFilterPrepareCB filter_prepare_cb;
@@ -472,6 +473,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeStreamCommitCB stream_commit_cb;
LogicalDecodeStreamChangeCB stream_change_cb;
LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamSequenceCB stream_sequence_cb;
LogicalDecodeStreamTruncateCB stream_truncate_cb;
} OutputPluginCallbacks;
@@ -481,9 +483,11 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
and <function>commit_cb</function> callbacks are required,
while <function>startup_cb</function>,
<function>filter_by_origin_cb</function>, <function>truncate_cb</function>,
- and <function>shutdown_cb</function> are optional.
- If <function>truncate_cb</function> is not set but a
+ <function>sequence_cb</function>, and <function>shutdown_cb</function> are
+ optional. If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
+ Similarly, if <function>sequence_cb</function> is not set and a sequence
+ change is to be decoded, the action will be ignored.
</para>
<para>
@@ -492,7 +496,8 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
<function>stream_stop_cb</function>, <function>stream_abort_cb</function>,
<function>stream_commit_cb</function>, <function>stream_change_cb</function>,
and <function>stream_prepare_cb</function>
- are required, while <function>stream_message_cb</function> and
+ are required, while <function>stream_message_cb</function>,
+ <function>stream_sequence_cb</function>, and
<function>stream_truncate_cb</function> are optional.
</para>
@@ -808,6 +813,35 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-sequence">
+ <title>Sequence Callback</title>
+
+ <para>
+ The optional <function>sequence_cb</function> callback is called for
+ actions that update a sequence value.
+<programlisting>
+typedef void (*LogicalDecodeSequenceCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr sequence_lsn,
+ Relation rel,
+ bool transactional,
+ int64 last_value,
+ int64 log_cnt,
+ bool is_called);
+</programlisting>
+ The <parameter>txn</parameter> parameter contains meta information about
+ the transaction the sequence change is part of. Note however that for
+ non-transactional increments, the transaction may be either NULL or not
+ NULL, depending on if the transaction already has an XID assigned.
+ The <parameter>sequence_lsn</parameter> has the WAL location of the
+ sequence update. <parameter>transactional</parameter> says if the
+ sequence has to be replayed as part of the transaction or directly.
+
+ The <parameter>last_value</parameter>, <parameter>log_cnt</parameter> and
+ <parameter>is_called</parameter> parameters describe the sequence change.
+ </para>
+ </sect3>
+
<sect3 id="logicaldecoding-output-plugin-filter-prepare">
<title>Prepare Filter Callback</title>
@@ -1017,6 +1051,26 @@ typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-sequence">
+ <title>Stream Sequence Callback</title>
+ <para>
+ The optional <function>stream_sequence_cb</function> callback is called
+ for actions that change a sequence in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamSequenceCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr sequence_lsn,
+ Relation rel,
+ bool transactional,
+ int64 last_value,
+ int64 log_cnt,
+ bool is_called);
+</programlisting>
+ </para>
+ </sect3>
+
<sect3 id="logicaldecoding-output-plugin-stream-truncate">
<title>Stream Truncate Callback</title>
<para>
@@ -1197,8 +1251,9 @@ OutputPluginWrite(ctx, true);
in-progress transactions. There are multiple required streaming callbacks
(<function>stream_start_cb</function>, <function>stream_stop_cb</function>,
<function>stream_abort_cb</function>, <function>stream_commit_cb</function>
- and <function>stream_change_cb</function>) and two optional callbacks
- (<function>stream_message_cb</function> and <function>stream_truncate_cb</function>).
+ and <function>stream_change_cb</function>) and multiple optional callbacks
+ (<function>stream_message_cb</function>, <function>stream_sequence_cb</function>,
+ and <function>stream_truncate_cb</function>).
Also, if streaming of two-phase commands is to be supported, then additional
callbacks must be provided. (See <xref linkend="logicaldecoding-two-phase-commits"/>
for details).
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 75a7a76a558..98f0bc3cc34 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -7072,6 +7072,125 @@ Relation
</listitem>
</varlistentry>
+<varlistentry id="protocol-logicalrep-message-formats-Sequence">
+<term>
+Sequence
+</term>
+<listitem>
+<para>
+
+<variablelist>
+<varlistentry>
+<term>
+ Byte1('X')
+</term>
+<listitem>
+<para>
+ Identifies the message as a sequence message.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int32 (TransactionId)
+</term>
+<listitem>
+<para>
+ Xid of the transaction (only present for streamed transactions).
+ This field is available since protocol version 2.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int8(0)
+</term>
+<listitem>
+<para>
+ Flags; currently unused.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ Int64 (XLogRecPtr)
+</term>
+<listitem>
+<para>
+ The LSN of the sequence increment.
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ String
+</term>
+<listitem>
+<para>
+ Namespace (empty string for <literal>pg_catalog</literal>).
+</para>
+</listitem>
+</varlistentry>
+<varlistentry>
+<term>
+ String
+</term>
+<listitem>
+<para>
+ Relation name.
+</para>
+</listitem>
+</varlistentry>
+
+<varlistentry>
+<term>
+ Int8
+</term>
+<listitem>
+<para>
+ 1 if the sequence update is transactional, 0 otherwise.
+</para>
+</listitem>
+</varlistentry>
+
+<varlistentry>
+<term>
+ Int64
+</term>
+<listitem>
+<para>
+ <structfield>last_value</structfield> value of the sequence.
+</para>
+</listitem>
+</varlistentry>
+
+<varlistentry>
+<term>
+ Int64
+</term>
+<listitem>
+<para>
+ <structfield>log_cnt</structfield> value of the sequence.
+</para>
+</listitem>
+</varlistentry>
+
+<varlistentry>
+<term>
+ Int8
+</term>
+<listitem>
+<para>
+ <structfield>is_called</structfield> value of the sequence.
+</para>
+</listitem>
+</varlistentry>
+
+</variablelist>
+</para>
+</listitem>
+</varlistentry>
+
<varlistentry id="protocol-logicalrep-message-formats-Type">
<term>
Type
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index e2cce49471b..db14d7a772c 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -31,7 +31,9 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ SEQUENCE <replaceable class="parameter">sequence_name</replaceable> [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
+ ALL SEQUENCES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -44,13 +46,13 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
</para>
<para>
- The first three variants change which tables/schemas are part of the
- publication. The <literal>SET</literal> clause will replace the list of
- tables/schemas in the publication with the specified list; the existing
- tables/schemas that were present in the publication will be removed. The
- <literal>ADD</literal> and <literal>DROP</literal> clauses will add and
- remove one or more tables/schemas from the publication. Note that adding
- tables/schemas to a publication that is already subscribed to will require an
+ The first three variants change which objects (tables, sequences or schemas)
+ are part of the publication. The <literal>SET</literal> clause will replace
+ the list of objects in the publication with the specified list; the existing
+ objects that were present in the publication will be removed.
+ The <literal>ADD</literal> and <literal>DROP</literal> clauses will add and
+ remove one or more objects from the publication. Note that adding objects
+ to a publication that is already subscribed to will require an
<literal>ALTER SUBSCRIPTION ... REFRESH PUBLICATION</literal> action on the
subscribing side in order to become effective. Note also that the combination
of <literal>DROP</literal> with a <literal>WHERE</literal> clause is not
@@ -130,6 +132,15 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><replaceable class="parameter">sequence_name</replaceable></term>
+ <listitem>
+ <para>
+ Name of an existing sequence.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><replaceable class="parameter">schema_name</replaceable></term>
<listitem>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 7c5203b6d3b..c1994e3a94a 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -150,8 +150,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<listitem>
<para>
Fetch missing table information from publisher. This will start
- replication of tables that were added to the subscribed-to publications
- since <command>CREATE SUBSCRIPTION</command> or
+ replication of tables and sequences that were added to the subscribed-to
+ publications since <command>CREATE SUBSCRIPTION</command> or
the last invocation of <command>REFRESH PUBLICATION</command>.
</para>
@@ -169,8 +169,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
The default is <literal>true</literal>.
</para>
<para>
- Previously subscribed tables are not copied, even if a table's row
- filter <literal>WHERE</literal> clause has since been modified.
+ Previously subscribed tables and sequences are not copied, even if a
+ table's row filter <literal>WHERE</literal> clause has since been modified.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index fb2d013393b..d2739968d9c 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -22,14 +22,21 @@ PostgreSQL documentation
<refsynopsisdiv>
<synopsis>
CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
- [ FOR ALL TABLES
+ [ FOR ALL <replaceable class="parameter">object_type</replaceable> [, ...]
| FOR <replaceable class="parameter">publication_object</replaceable> [, ... ] ]
[ WITH ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
+<phrase>where <replaceable class="parameter">object type</replaceable> is one of:</phrase>
+
+ TABLES
+ SEQUENCES
+
<phrase>where <replaceable class="parameter">publication_object</replaceable> is one of:</phrase>
TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ ( <replaceable class="parameter">column_name</replaceable> [, ... ] ) ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ... ]
+ SEQUENCE <replaceable class="parameter">sequence_name</replaceable> [ * ] [, ... ]
ALL TABLES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
+ ALL SEQUENCES IN SCHEMA { <replaceable class="parameter">schema_name</replaceable> | CURRENT_SCHEMA } [, ... ]
</synopsis>
</refsynopsisdiv>
@@ -114,27 +121,43 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>FOR SEQUENCE</literal></term>
+ <listitem>
+ <para>
+ Specifies a list of sequences to add to the publication.
+ </para>
+
+ <para>
+ Specifying a sequence that is part of a schema specified by <literal>FOR
+ ALL SEQUENCES IN SCHEMA</literal> is not supported.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>FOR ALL TABLES</literal></term>
+ <term><literal>FOR ALL SEQUENCES</literal></term>
<listitem>
<para>
- Marks the publication as one that replicates changes for all tables in
- the database, including tables created in the future.
+ Marks the publication as one that replicates changes for all tables/sequences in
+ the database, including tables/sequences created in the future.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>FOR ALL TABLES IN SCHEMA</literal></term>
+ <term><literal>FOR ALL SEQUENCES IN SCHEMA</literal></term>
<listitem>
<para>
- Marks the publication as one that replicates changes for all tables in
- the specified list of schemas, including tables created in the future.
+ Marks the publication as one that replicates changes for all sequences/tables in
+ the specified list of schemas, including sequences/tables created in the future.
</para>
<para>
- Specifying a schema along with a table which belongs to the specified
- schema using <literal>FOR TABLE</literal> is not supported.
+ Specifying a schema along with a sequence/table which belongs to the specified
+ schema using <literal>FOR SEQUENCE</literal>/<literal>FOR TABLE</literal> is not supported.
</para>
<para>
@@ -209,10 +232,9 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
<title>Notes</title>
<para>
- If <literal>FOR TABLE</literal>, <literal>FOR ALL TABLES</literal> or
- <literal>FOR ALL TABLES IN SCHEMA</literal> are not specified, then the
- publication starts out with an empty set of tables. That is useful if
- tables or schemas are to be added later.
+ If <literal>FOR TABLE</literal>, <literal>FOR SEQUENCE</literal>, etc. is
+ not specified, then the publication starts out with an empty set of tables
+ and sequences. That is useful if objects are to be added later.
</para>
<para>
@@ -227,10 +249,9 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
</para>
<para>
- To add a table to a publication, the invoking user must have ownership
- rights on the table. The <command>FOR ALL TABLES</command> and
- <command>FOR ALL TABLES IN SCHEMA</command> clauses require the invoking
- user to be a superuser.
+ To add a table or a sequence to a publication, the invoking user must
+ have ownership rights on the object. The <command>FOR ALL ...</command>
+ clauses require the invoking user to be a superuser.
</para>
<para>
diff --git a/src/backend/access/rmgrdesc/logicalmsgdesc.c b/src/backend/access/rmgrdesc/logicalmsgdesc.c
index 099e11a84e7..de77dcb0dc2 100644
--- a/src/backend/access/rmgrdesc/logicalmsgdesc.c
+++ b/src/backend/access/rmgrdesc/logicalmsgdesc.c
@@ -40,6 +40,16 @@ logicalmsg_desc(StringInfo buf, XLogReaderState *record)
sep = " ";
}
}
+ else if (info == XLOG_LOGICAL_SEQUENCE)
+ {
+ xl_logical_sequence *xlrec = (xl_logical_sequence *) rec;
+
+ appendStringInfo(buf, "rel %u/%u/%u last: %lu log_cnt: %lu is_called: %d",
+ xlrec->node.spcNode,
+ xlrec->node.dbNode,
+ xlrec->node.relNode,
+ xlrec->last, xlrec->log_cnt, xlrec->is_called);
+ }
}
const char *
@@ -48,5 +58,8 @@ logicalmsg_identify(uint8 info)
if ((info & ~XLR_INFO_MASK) == XLOG_LOGICAL_MESSAGE)
return "MESSAGE";
+ if ((info & ~XLR_INFO_MASK) == XLOG_LOGICAL_SEQUENCE)
+ return "SEQUENCE";
+
return NULL;
}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index bf2fc08d940..eb1727a5d4c 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -37,6 +37,7 @@
#include "catalog/storage.h"
#include "commands/async.h"
#include "commands/tablecmds.h"
+#include "commands/sequence.h"
#include "commands/trigger.h"
#include "common/pg_prng.h"
#include "executor/spi.h"
@@ -2237,6 +2238,15 @@ CommitTransaction(void)
/* Prevent cancel/die interrupt while cleaning up */
HOLD_INTERRUPTS();
+ /*
+ * Write state of sequences to WAL. We need to do this early enough so
+ * that we can still write stuff to WAL, but probably before changing
+ * the transaction state.
+ *
+ * XXX Should this happen before/after holding the interrupts?
+ */
+ AtEOXact_Sequences(true);
+
/* Commit updates to the relation map --- do this as late as possible */
AtEOXact_RelationMap(true, is_parallel_worker);
@@ -2513,6 +2523,15 @@ PrepareTransaction(void)
/* Prevent cancel/die interrupt while cleaning up */
HOLD_INTERRUPTS();
+ /*
+ * Write state of sequences to WAL. We need to do this early enough so
+ * that we can still write stuff to WAL, but probably before changing
+ * the transaction state.
+ *
+ * XXX Should this happen before/after holding the interrupts?
+ */
+ AtEOXact_Sequences(true);
+
/*
* set the current transaction state information appropriately during
* prepare processing
@@ -2748,6 +2767,12 @@ AbortTransaction(void)
TransStateAsString(s->state));
Assert(s->parent == NULL);
+ /*
+ * For transaction abort we don't need to write anything to WAL, we just
+ * do cleanup of the hash table etc.
+ */
+ AtEOXact_Sequences(false);
+
/*
* set the current transaction state information appropriately during the
* abort processing
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index ac6043514f1..31c80f7209f 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -1941,12 +1941,14 @@ get_object_address_publication_schema(List *object, bool missing_ok)
char *pubname;
char *schemaname;
Oid schemaid;
+ char *objtype;
ObjectAddressSet(address, PublicationNamespaceRelationId, InvalidOid);
/* Fetch schema name and publication name from input list */
schemaname = strVal(linitial(object));
pubname = strVal(lsecond(object));
+ objtype = strVal(lthird(object));
schemaid = get_namespace_oid(schemaname, missing_ok);
if (!OidIsValid(schemaid))
@@ -1959,10 +1961,12 @@ get_object_address_publication_schema(List *object, bool missing_ok)
/* Find the publication schema mapping in syscache */
address.objectId =
- GetSysCacheOid2(PUBLICATIONNAMESPACEMAP,
+ GetSysCacheOid3(PUBLICATIONNAMESPACEMAP,
Anum_pg_publication_namespace_oid,
ObjectIdGetDatum(schemaid),
- ObjectIdGetDatum(pub->oid));
+ ObjectIdGetDatum(pub->oid),
+ CharGetDatum(objtype[0]));
+
if (!OidIsValid(address.objectId) && !missing_ok)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_OBJECT),
@@ -2243,7 +2247,6 @@ pg_get_object_address(PG_FUNCTION_ARGS)
case OBJECT_DOMCONSTRAINT:
case OBJECT_CAST:
case OBJECT_USER_MAPPING:
- case OBJECT_PUBLICATION_NAMESPACE:
case OBJECT_PUBLICATION_REL:
case OBJECT_DEFACL:
case OBJECT_TRANSFORM:
@@ -2268,6 +2271,7 @@ pg_get_object_address(PG_FUNCTION_ARGS)
/* fall through to check args length */
/* FALLTHROUGH */
case OBJECT_OPERATOR:
+ case OBJECT_PUBLICATION_NAMESPACE:
if (list_length(args) != 2)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
@@ -2339,6 +2343,8 @@ pg_get_object_address(PG_FUNCTION_ARGS)
objnode = (Node *) list_make2(name, linitial(args));
break;
case OBJECT_PUBLICATION_NAMESPACE:
+ objnode = (Node *) list_make3(linitial(name), linitial(args), lsecond(args));
+ break;
case OBJECT_USER_MAPPING:
objnode = (Node *) list_make2(linitial(name), linitial(args));
break;
@@ -2894,11 +2900,12 @@ get_catalog_object_by_oid(Relation catalog, AttrNumber oidcol, Oid objectId)
*
* Get publication name and schema name from the object address into pubname and
* nspname. Both pubname and nspname are palloc'd strings which will be freed by
- * the caller.
+ * the caller. The last parameter specifies which object type is included from
+ * the schema.
*/
static bool
getPublicationSchemaInfo(const ObjectAddress *object, bool missing_ok,
- char **pubname, char **nspname)
+ char **pubname, char **nspname, char **objtype)
{
HeapTuple tup;
Form_pg_publication_namespace pnform;
@@ -2934,6 +2941,13 @@ getPublicationSchemaInfo(const ObjectAddress *object, bool missing_ok,
return false;
}
+ /*
+ * The type is always a single character, but we need to pass it as a string,
+ * so allocate two charaters and set the first one. The second one is \0.
+ */
+ *objtype = palloc0(2);
+ *objtype[0] = pnform->pntype;
+
ReleaseSysCache(tup);
return true;
}
@@ -3965,15 +3979,17 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok)
{
char *pubname;
char *nspname;
+ char *objtype;
if (!getPublicationSchemaInfo(object, missing_ok,
- &pubname, &nspname))
+ &pubname, &nspname, &objtype))
break;
- appendStringInfo(&buffer, _("publication of schema %s in publication %s"),
- nspname, pubname);
+ appendStringInfo(&buffer, _("publication of schema %s in publication %s type %s"),
+ nspname, pubname, objtype);
pfree(pubname);
pfree(nspname);
+ pfree(objtype);
break;
}
@@ -5800,18 +5816,24 @@ getObjectIdentityParts(const ObjectAddress *object,
{
char *pubname;
char *nspname;
+ char *objtype;
if (!getPublicationSchemaInfo(object, missing_ok, &pubname,
- &nspname))
+ &nspname, &objtype))
break;
- appendStringInfo(&buffer, "%s in publication %s",
- nspname, pubname);
+ appendStringInfo(&buffer, "%s in publication %s type %s",
+ nspname, pubname, objtype);
if (objargs)
*objargs = list_make1(pubname);
else
pfree(pubname);
+ if (objargs)
+ *objargs = lappend(*objargs, objtype);
+ else
+ pfree(objtype);
+
if (objname)
*objname = list_make1(nspname);
else
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 2631558ff11..9fe3b189269 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -55,9 +55,10 @@ static void publication_translate_columns(Relation targetrel, List *columns,
static void
check_publication_add_relation(Relation targetrel)
{
- /* Must be a regular or partitioned table */
+ /* Must be a regular or partitioned table, or a sequence */
if (RelationGetForm(targetrel)->relkind != RELKIND_RELATION &&
- RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE)
+ RelationGetForm(targetrel)->relkind != RELKIND_PARTITIONED_TABLE &&
+ RelationGetForm(targetrel)->relkind != RELKIND_SEQUENCE)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("cannot add relation \"%s\" to publication",
@@ -134,7 +135,8 @@ static bool
is_publishable_class(Oid relid, Form_pg_class reltuple)
{
return (reltuple->relkind == RELKIND_RELATION ||
- reltuple->relkind == RELKIND_PARTITIONED_TABLE) &&
+ reltuple->relkind == RELKIND_PARTITIONED_TABLE ||
+ reltuple->relkind == RELKIND_SEQUENCE) &&
!IsCatalogRelationOid(relid) &&
reltuple->relpersistence == RELPERSISTENCE_PERMANENT &&
relid >= FirstNormalObjectId;
@@ -179,6 +181,52 @@ filter_partitions(List *relids)
return result;
}
+/*
+ * Check the character is a valid object type for schema publication.
+ *
+ * This recognizes either 't' for tables or 's' for sequences. Places that
+ * need to handle 'u' for unsupported relkinds need to do that explicitlyl
+ */
+static void
+AssertObjectTypeValid(char objectType)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(objectType == PUB_OBJTYPE_SEQUENCE || objectType == PUB_OBJTYPE_TABLE);
+#endif
+}
+
+/*
+ * Determine object type matching a given a relkind value.
+ */
+char
+pub_get_object_type_for_relkind(char relkind)
+{
+ /* sequence maps directly to sequence relkind */
+ if (relkind == RELKIND_SEQUENCE)
+ return PUB_OBJTYPE_SEQUENCE;
+
+ /* for table, we match either regular or partitioned table */
+ if (relkind == RELKIND_RELATION ||
+ relkind == RELKIND_PARTITIONED_TABLE)
+ return PUB_OBJTYPE_TABLE;
+
+ return PUB_OBJTYPE_UNSUPPORTED;
+}
+
+/*
+ * Determine if publication object type matches the relkind.
+ *
+ * Returns true if the relation matches object type replicated by this schema,
+ * false otherwise.
+ */
+static bool
+pub_object_type_matches_relkind(char objectType, char relkind)
+{
+ AssertObjectTypeValid(objectType);
+
+ return (pub_get_object_type_for_relkind(relkind) == objectType);
+}
+
/*
* Another variant of this, taking a Relation.
*/
@@ -208,7 +256,7 @@ is_schema_publication(Oid pubid)
ObjectIdGetDatum(pubid));
scan = systable_beginscan(pubschsrel,
- PublicationNamespacePnnspidPnpubidIndexId,
+ PublicationNamespacePnnspidPnpubidPntypeIndexId,
true, NULL, 1, &scankey);
tup = systable_getnext(scan);
result = HeapTupleIsValid(tup);
@@ -316,7 +364,9 @@ GetTopMostAncestorInPublication(Oid puboid, List *ancestors, int *ancestor_level
}
else
{
- aschemaPubids = GetSchemaPublications(get_rel_namespace(ancestor));
+ /* we only search for ancestors of tables, so PUB_OBJTYPE_TABLE */
+ aschemaPubids = GetSchemaPublications(get_rel_namespace(ancestor),
+ PUB_OBJTYPE_TABLE);
if (list_member_oid(aschemaPubids, puboid))
{
topmost_relid = ancestor;
@@ -585,7 +635,7 @@ pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols, MemoryContext mcxt)
* Insert new publication / schema mapping.
*/
ObjectAddress
-publication_add_schema(Oid pubid, Oid schemaid, bool if_not_exists)
+publication_add_schema(Oid pubid, Oid schemaid, char objectType, bool if_not_exists)
{
Relation rel;
HeapTuple tup;
@@ -597,6 +647,8 @@ publication_add_schema(Oid pubid, Oid schemaid, bool if_not_exists)
ObjectAddress myself,
referenced;
+ AssertObjectTypeValid(objectType);
+
rel = table_open(PublicationNamespaceRelationId, RowExclusiveLock);
/*
@@ -604,9 +656,10 @@ publication_add_schema(Oid pubid, Oid schemaid, bool if_not_exists)
* duplicates, it's here just to provide nicer error message in common
* case. The real protection is the unique key on the catalog.
*/
- if (SearchSysCacheExists2(PUBLICATIONNAMESPACEMAP,
+ if (SearchSysCacheExists3(PUBLICATIONNAMESPACEMAP,
ObjectIdGetDatum(schemaid),
- ObjectIdGetDatum(pubid)))
+ ObjectIdGetDatum(pubid),
+ CharGetDatum(objectType)))
{
table_close(rel, RowExclusiveLock);
@@ -632,6 +685,8 @@ publication_add_schema(Oid pubid, Oid schemaid, bool if_not_exists)
ObjectIdGetDatum(pubid);
values[Anum_pg_publication_namespace_pnnspid - 1] =
ObjectIdGetDatum(schemaid);
+ values[Anum_pg_publication_namespace_pntype - 1] =
+ CharGetDatum(objectType);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -657,7 +712,7 @@ publication_add_schema(Oid pubid, Oid schemaid, bool if_not_exists)
* publication_add_relation for why we need to consider all the
* partitions.
*/
- schemaRels = GetSchemaPublicationRelations(schemaid,
+ schemaRels = GetSchemaPublicationRelations(schemaid, objectType,
PUBLICATION_PART_ALL);
InvalidatePublicationRels(schemaRels);
@@ -691,11 +746,14 @@ GetRelationPublications(Oid relid)
/*
* Gets list of relation oids for a publication.
*
- * This should only be used FOR TABLE publications, the FOR ALL TABLES
- * should use GetAllTablesPublicationRelations().
+ * This should only be used FOR TABLE / FOR SEQUENCE publications, the FOR
+ * ALL TABLES / SEQUENCES should use GetAllTablesPublicationRelations()
+ * and GetAllSequencesPublicationRelations().
+ *
+ * XXX pub_partopt only matters for tables, not sequences.
*/
List *
-GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt)
+GetPublicationRelations(Oid pubid, char objectType, PublicationPartOpt pub_partopt)
{
List *result;
Relation pubrelsrel;
@@ -703,6 +761,8 @@ GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt)
SysScanDesc scan;
HeapTuple tup;
+ AssertObjectTypeValid(objectType);
+
/* Find all publications associated with the relation. */
pubrelsrel = table_open(PublicationRelRelationId, AccessShareLock);
@@ -717,11 +777,29 @@ GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt)
result = NIL;
while (HeapTupleIsValid(tup = systable_getnext(scan)))
{
+ char relkind;
Form_pg_publication_rel pubrel;
pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
- result = GetPubPartitionOptionRelations(result, pub_partopt,
- pubrel->prrelid);
+ relkind = get_rel_relkind(pubrel->prrelid);
+
+ /*
+ * If the relkind does not match the requested object type, ignore the
+ * relation. For example we might be interested only in sequences, so
+ * we ignore tables.
+ */
+ if (!pub_object_type_matches_relkind(objectType, relkind))
+ continue;
+
+ /*
+ * We don't have partitioned sequences, so just add them to the list.
+ * Otherwise consider adding all child relations, if requested.
+ */
+ if (relkind == RELKIND_SEQUENCE)
+ result = lappend_oid(result, pubrel->prrelid);
+ else
+ result = GetPubPartitionOptionRelations(result, pub_partopt,
+ pubrel->prrelid);
}
systable_endscan(scan);
@@ -771,6 +849,43 @@ GetAllTablesPublications(void)
return result;
}
+/*
+ * Gets list of publication oids for publications marked as FOR ALL SEQUENCES.
+ */
+List *
+GetAllSequencesPublications(void)
+{
+ List *result;
+ Relation rel;
+ ScanKeyData scankey;
+ SysScanDesc scan;
+ HeapTuple tup;
+
+ /* Find all publications that are marked as for all sequences. */
+ rel = table_open(PublicationRelationId, AccessShareLock);
+
+ ScanKeyInit(&scankey,
+ Anum_pg_publication_puballsequences,
+ BTEqualStrategyNumber, F_BOOLEQ,
+ BoolGetDatum(true));
+
+ scan = systable_beginscan(rel, InvalidOid, false,
+ NULL, 1, &scankey);
+
+ result = NIL;
+ while (HeapTupleIsValid(tup = systable_getnext(scan)))
+ {
+ Oid oid = ((Form_pg_publication) GETSTRUCT(tup))->oid;
+
+ result = lappend_oid(result, oid);
+ }
+
+ systable_endscan(scan);
+ table_close(rel, AccessShareLock);
+
+ return result;
+}
+
/*
* Gets list of all relation published by FOR ALL TABLES publication(s).
*
@@ -837,28 +952,38 @@ GetAllTablesPublicationRelations(bool pubviaroot)
/*
* Gets the list of schema oids for a publication.
*
- * This should only be used FOR ALL TABLES IN SCHEMA publications.
+ * This should only be used FOR ALL TABLES IN SCHEMA and FOR ALL SEQUENCES
+ * publications.
+ *
+ * 'objectType' determines whether to get FOR TABLE or FOR SEQUENCES schemas
*/
List *
-GetPublicationSchemas(Oid pubid)
+GetPublicationSchemas(Oid pubid, char objectType)
{
List *result = NIL;
Relation pubschsrel;
- ScanKeyData scankey;
+ ScanKeyData scankey[2];
SysScanDesc scan;
HeapTuple tup;
+ AssertObjectTypeValid(objectType);
+
/* Find all schemas associated with the publication */
pubschsrel = table_open(PublicationNamespaceRelationId, AccessShareLock);
- ScanKeyInit(&scankey,
+ ScanKeyInit(&scankey[0],
Anum_pg_publication_namespace_pnpubid,
BTEqualStrategyNumber, F_OIDEQ,
ObjectIdGetDatum(pubid));
+ ScanKeyInit(&scankey[1],
+ Anum_pg_publication_namespace_pntype,
+ BTEqualStrategyNumber, F_CHAREQ,
+ CharGetDatum(objectType));
+
scan = systable_beginscan(pubschsrel,
- PublicationNamespacePnnspidPnpubidIndexId,
- true, NULL, 1, &scankey);
+ PublicationNamespacePnnspidPnpubidPntypeIndexId,
+ true, NULL, 2, scankey);
while (HeapTupleIsValid(tup = systable_getnext(scan)))
{
Form_pg_publication_namespace pubsch;
@@ -876,14 +1001,26 @@ GetPublicationSchemas(Oid pubid)
/*
* Gets the list of publication oids associated with a specified schema.
+ *
+ * objectType specifies whether we're looking for schemas including tables or
+ * sequences.
+ *
+ * Note: relcache calls this for all object types, not just tables and sequences.
+ * Which is why we handle the PUB_OBJTYPE_UNSUPPORTED object type too.
*/
List *
-GetSchemaPublications(Oid schemaid)
+GetSchemaPublications(Oid schemaid, char objectType)
{
List *result = NIL;
CatCList *pubschlist;
int i;
+ /* unsupported object type */
+ if (objectType == PUB_OBJTYPE_UNSUPPORTED)
+ return result;
+
+ AssertObjectTypeValid(objectType);
+
/* Find all publications associated with the schema */
pubschlist = SearchSysCacheList1(PUBLICATIONNAMESPACEMAP,
ObjectIdGetDatum(schemaid));
@@ -891,6 +1028,11 @@ GetSchemaPublications(Oid schemaid)
{
HeapTuple tup = &pubschlist->members[i]->tuple;
Oid pubid = ((Form_pg_publication_namespace) GETSTRUCT(tup))->pnpubid;
+ char pntype = ((Form_pg_publication_namespace) GETSTRUCT(tup))->pntype;
+
+ /* Skip schemas publishing a different object type. */
+ if (pntype != objectType)
+ continue;
result = lappend_oid(result, pubid);
}
@@ -902,9 +1044,13 @@ GetSchemaPublications(Oid schemaid)
/*
* Get the list of publishable relation oids for a specified schema.
+ *
+ * objectType specifies whether this is FOR ALL TABLES IN SCHEMA or FOR ALL
+ * SEQUENCES IN SCHEMA
*/
List *
-GetSchemaPublicationRelations(Oid schemaid, PublicationPartOpt pub_partopt)
+GetSchemaPublicationRelations(Oid schemaid, char objectType,
+ PublicationPartOpt pub_partopt)
{
Relation classRel;
ScanKeyData key[1];
@@ -913,6 +1059,7 @@ GetSchemaPublicationRelations(Oid schemaid, PublicationPartOpt pub_partopt)
List *result = NIL;
Assert(OidIsValid(schemaid));
+ AssertObjectTypeValid(objectType);
classRel = table_open(RelationRelationId, AccessShareLock);
@@ -933,9 +1080,16 @@ GetSchemaPublicationRelations(Oid schemaid, PublicationPartOpt pub_partopt)
continue;
relkind = get_rel_relkind(relid);
- if (relkind == RELKIND_RELATION)
- result = lappend_oid(result, relid);
- else if (relkind == RELKIND_PARTITIONED_TABLE)
+
+ /* Skip if the relkind does not match FOR ALL TABLES / SEQUENCES. */
+ if (!pub_object_type_matches_relkind(objectType, relkind))
+ continue;
+
+ /*
+ * If the object is a partitioned table, lookup all the child relations
+ * (if requested). Otherwise just add the object to the list.
+ */
+ if (relkind == RELKIND_PARTITIONED_TABLE)
{
List *partitionrels = NIL;
@@ -948,7 +1102,11 @@ GetSchemaPublicationRelations(Oid schemaid, PublicationPartOpt pub_partopt)
pub_partopt,
relForm->oid);
result = list_concat_unique_oid(result, partitionrels);
+ continue;
}
+
+ /* non-partitioned tables and sequences */
+ result = lappend_oid(result, relid);
}
table_endscan(scan);
@@ -958,27 +1116,67 @@ GetSchemaPublicationRelations(Oid schemaid, PublicationPartOpt pub_partopt)
/*
* Gets the list of all relations published by FOR ALL TABLES IN SCHEMA
- * publication.
+ * or FOR ALL SEQUENCES IN SCHEMA publication.
*/
List *
-GetAllSchemaPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt)
+GetAllSchemaPublicationRelations(Oid pubid, char objectType,
+ PublicationPartOpt pub_partopt)
{
List *result = NIL;
- List *pubschemalist = GetPublicationSchemas(pubid);
+ List *pubschemalist = GetPublicationSchemas(pubid, objectType);
ListCell *cell;
+ AssertObjectTypeValid(objectType);
+
foreach(cell, pubschemalist)
{
Oid schemaid = lfirst_oid(cell);
List *schemaRels = NIL;
- schemaRels = GetSchemaPublicationRelations(schemaid, pub_partopt);
+ schemaRels = GetSchemaPublicationRelations(schemaid, objectType,
+ pub_partopt);
result = list_concat(result, schemaRels);
}
return result;
}
+/*
+ * Gets list of all relation published by FOR ALL SEQUENCES publication(s).
+ */
+List *
+GetAllSequencesPublicationRelations(void)
+{
+ Relation classRel;
+ ScanKeyData key[1];
+ TableScanDesc scan;
+ HeapTuple tuple;
+ List *result = NIL;
+
+ classRel = table_open(RelationRelationId, AccessShareLock);
+
+ ScanKeyInit(&key[0],
+ Anum_pg_class_relkind,
+ BTEqualStrategyNumber, F_CHAREQ,
+ CharGetDatum(RELKIND_SEQUENCE));
+
+ scan = table_beginscan_catalog(classRel, 1, key);
+
+ while ((tuple = heap_getnext(scan, ForwardScanDirection)) != NULL)
+ {
+ Form_pg_class relForm = (Form_pg_class) GETSTRUCT(tuple);
+ Oid relid = relForm->oid;
+
+ if (is_publishable_class(relid, relForm))
+ result = lappend_oid(result, relid);
+ }
+
+ table_endscan(scan);
+
+ table_close(classRel, AccessShareLock);
+ return result;
+}
+
/*
* Get publication using oid
*
@@ -1001,10 +1199,12 @@ GetPublication(Oid pubid)
pub->oid = pubid;
pub->name = pstrdup(NameStr(pubform->pubname));
pub->alltables = pubform->puballtables;
+ pub->allsequences = pubform->puballsequences;
pub->pubactions.pubinsert = pubform->pubinsert;
pub->pubactions.pubupdate = pubform->pubupdate;
pub->pubactions.pubdelete = pubform->pubdelete;
pub->pubactions.pubtruncate = pubform->pubtruncate;
+ pub->pubactions.pubsequence = pubform->pubsequence;
pub->pubviaroot = pubform->pubviaroot;
ReleaseSysCache(tup);
@@ -1115,10 +1315,12 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
*schemarelids;
relids = GetPublicationRelations(publication->oid,
+ PUB_OBJTYPE_TABLE,
publication->pubviaroot ?
PUBLICATION_PART_ROOT :
PUBLICATION_PART_LEAF);
schemarelids = GetAllSchemaPublicationRelations(publication->oid,
+ PUB_OBJTYPE_TABLE,
publication->pubviaroot ?
PUBLICATION_PART_ROOT :
PUBLICATION_PART_LEAF);
@@ -1154,3 +1356,71 @@ pg_get_publication_tables(PG_FUNCTION_ARGS)
SRF_RETURN_DONE(funcctx);
}
+
+/*
+ * Returns Oids of sequences in a publication.
+ */
+Datum
+pg_get_publication_sequences(PG_FUNCTION_ARGS)
+{
+ FuncCallContext *funcctx;
+ char *pubname = text_to_cstring(PG_GETARG_TEXT_PP(0));
+ Publication *publication;
+ List *sequences;
+
+ /* stuff done only on the first call of the function */
+ if (SRF_IS_FIRSTCALL())
+ {
+ MemoryContext oldcontext;
+
+ /* create a function context for cross-call persistence */
+ funcctx = SRF_FIRSTCALL_INIT();
+
+ /* switch to memory context appropriate for multiple function calls */
+ oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
+
+ publication = GetPublicationByName(pubname, false);
+
+ /*
+ * Publications support partitioned tables, although all changes are
+ * replicated using leaf partition identity and schema, so we only
+ * need those.
+ */
+ if (publication->allsequences)
+ sequences = GetAllSequencesPublicationRelations();
+ else
+ {
+ List *relids,
+ *schemarelids;
+
+ relids = GetPublicationRelations(publication->oid,
+ PUB_OBJTYPE_SEQUENCE,
+ publication->pubviaroot ?
+ PUBLICATION_PART_ROOT :
+ PUBLICATION_PART_LEAF);
+ schemarelids = GetAllSchemaPublicationRelations(publication->oid,
+ PUB_OBJTYPE_SEQUENCE,
+ publication->pubviaroot ?
+ PUBLICATION_PART_ROOT :
+ PUBLICATION_PART_LEAF);
+ sequences = list_concat_unique_oid(relids, schemarelids);
+ }
+
+ funcctx->user_fctx = (void *) sequences;
+
+ MemoryContextSwitchTo(oldcontext);
+ }
+
+ /* stuff done on every call of the function */
+ funcctx = SRF_PERCALL_SETUP();
+ sequences = (List *) funcctx->user_fctx;
+
+ if (funcctx->call_cntr < list_length(sequences))
+ {
+ Oid relid = list_nth_oid(sequences, funcctx->call_cntr);
+
+ SRF_RETURN_NEXT(funcctx, ObjectIdGetDatum(relid));
+ }
+
+ SRF_RETURN_DONE(funcctx);
+}
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 0fc614e32c0..b1a6df16ad3 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -374,6 +374,16 @@ CREATE VIEW pg_publication_tables AS
pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE C.oid = GPT.relid;
+CREATE VIEW pg_publication_sequences AS
+ SELECT
+ P.pubname AS pubname,
+ N.nspname AS schemaname,
+ C.relname AS sequencename
+ FROM pg_publication P,
+ LATERAL pg_get_publication_sequences(P.pubname) GPS,
+ pg_class C JOIN pg_namespace N ON (N.oid = C.relnamespace)
+ WHERE C.oid = GPS.relid;
+
CREATE VIEW pg_locks AS
SELECT * FROM pg_lock_status() AS L;
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 4fd1e6e7abb..84e37df783b 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -16,6 +16,7 @@
#include "access/genam.h"
#include "access/htup_details.h"
+#include "access/relation.h"
#include "access/table.h"
#include "access/xact.h"
#include "catalog/catalog.h"
@@ -67,15 +68,17 @@ typedef struct rf_context
} rf_context;
static List *OpenRelIdList(List *relids);
-static List *OpenTableList(List *tables);
-static void CloseTableList(List *rels);
+static List *OpenRelationList(List *rels, char objectType);
+static void CloseRelationList(List *rels);
static void LockSchemaList(List *schemalist);
-static void PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
+static void PublicationAddRelations(Oid pubid, List *rels, bool if_not_exists,
AlterPublicationStmt *stmt);
-static void PublicationDropTables(Oid pubid, List *rels, bool missing_ok);
-static void PublicationAddSchemas(Oid pubid, List *schemas, bool if_not_exists,
- AlterPublicationStmt *stmt);
-static void PublicationDropSchemas(Oid pubid, List *schemas, bool missing_ok);
+static void PublicationDropRelations(Oid pubid, List *rels, bool missing_ok);
+static void PublicationAddSchemas(Oid pubid, List *schemas, char objectType,
+ bool if_not_exists, AlterPublicationStmt *stmt);
+static void PublicationDropSchemas(Oid pubid, List *schemas, char objectType,
+ bool missing_ok);
+
static void
parse_publication_options(ParseState *pstate,
@@ -95,6 +98,7 @@ parse_publication_options(ParseState *pstate,
pubactions->pubupdate = true;
pubactions->pubdelete = true;
pubactions->pubtruncate = true;
+ pubactions->pubsequence = true;
*publish_via_partition_root = false;
/* Parse options */
@@ -119,6 +123,7 @@ parse_publication_options(ParseState *pstate,
pubactions->pubupdate = false;
pubactions->pubdelete = false;
pubactions->pubtruncate = false;
+ pubactions->pubsequence = false;
*publish_given = true;
publish = defGetString(defel);
@@ -141,6 +146,8 @@ parse_publication_options(ParseState *pstate,
pubactions->pubdelete = true;
else if (strcmp(publish_opt, "truncate") == 0)
pubactions->pubtruncate = true;
+ else if (strcmp(publish_opt, "sequence") == 0)
+ pubactions->pubsequence = true;
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -167,7 +174,8 @@ parse_publication_options(ParseState *pstate,
*/
static void
ObjectsInPublicationToOids(List *pubobjspec_list, ParseState *pstate,
- List **rels, List **schemas)
+ List **tables, List **sequences,
+ List **tables_schemas, List **sequences_schemas)
{
ListCell *cell;
PublicationObjSpec *pubobj;
@@ -185,13 +193,22 @@ ObjectsInPublicationToOids(List *pubobjspec_list, ParseState *pstate,
switch (pubobj->pubobjtype)
{
case PUBLICATIONOBJ_TABLE:
- *rels = lappend(*rels, pubobj->pubtable);
+ *tables = lappend(*tables, pubobj->pubtable);
+ break;
+ case PUBLICATIONOBJ_SEQUENCE:
+ *sequences = lappend(*sequences, pubobj->pubtable);
break;
case PUBLICATIONOBJ_TABLES_IN_SCHEMA:
schemaid = get_namespace_oid(pubobj->name, false);
/* Filter out duplicates if user specifies "sch1, sch1" */
- *schemas = list_append_unique_oid(*schemas, schemaid);
+ *tables_schemas = list_append_unique_oid(*tables_schemas, schemaid);
+ break;
+ case PUBLICATIONOBJ_SEQUENCES_IN_SCHEMA:
+ schemaid = get_namespace_oid(pubobj->name, false);
+
+ /* Filter out duplicates if user specifies "sch1, sch1" */
+ *sequences_schemas = list_append_unique_oid(*sequences_schemas, schemaid);
break;
case PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA:
search_path = fetch_search_path(false);
@@ -204,7 +221,20 @@ ObjectsInPublicationToOids(List *pubobjspec_list, ParseState *pstate,
list_free(search_path);
/* Filter out duplicates if user specifies "sch1, sch1" */
- *schemas = list_append_unique_oid(*schemas, schemaid);
+ *tables_schemas = list_append_unique_oid(*tables_schemas, schemaid);
+ break;
+ case PUBLICATIONOBJ_SEQUENCES_IN_CUR_SCHEMA:
+ search_path = fetch_search_path(false);
+ if (search_path == NIL) /* nothing valid in search_path? */
+ ereport(ERROR,
+ errcode(ERRCODE_UNDEFINED_SCHEMA),
+ errmsg("no schema has been selected for CURRENT_SCHEMA"));
+
+ schemaid = linitial_oid(search_path);
+ list_free(search_path);
+
+ /* Filter out duplicates if user specifies "sch1, sch1" */
+ *sequences_schemas = list_append_unique_oid(*sequences_schemas, schemaid);
break;
default:
/* shouldn't happen */
@@ -240,6 +270,14 @@ CheckObjSchemaNotAlreadyInPublication(List *rels, List *schemaidlist,
errdetail("Table \"%s\" in schema \"%s\" is already part of the publication, adding the same schema is not supported.",
RelationGetRelationName(rel),
get_namespace_name(relSchemaId)));
+ else if (checkobjtype == PUBLICATIONOBJ_SEQUENCES_IN_SCHEMA)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add schema \"%s\" to publication",
+ get_namespace_name(relSchemaId)),
+ errdetail("Sequence \"%s\" in schema \"%s\" is already part of the publication, adding the same schema is not supported.",
+ RelationGetRelationName(rel),
+ get_namespace_name(relSchemaId)));
else if (checkobjtype == PUBLICATIONOBJ_TABLE)
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
@@ -248,6 +286,14 @@ CheckObjSchemaNotAlreadyInPublication(List *rels, List *schemaidlist,
RelationGetRelationName(rel)),
errdetail("Table's schema \"%s\" is already part of the publication or part of the specified schema list.",
get_namespace_name(relSchemaId)));
+ else if (checkobjtype == PUBLICATIONOBJ_SEQUENCE)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("cannot add relation \"%s.%s\" to publication",
+ get_namespace_name(relSchemaId),
+ RelationGetRelationName(rel)),
+ errdetail("Sequence's schema \"%s\" is already part of the publication or part of the specified schema list.",
+ get_namespace_name(relSchemaId)));
}
}
}
@@ -753,6 +799,7 @@ CheckPubRelationColumnList(List *tables, const char *queryString,
ObjectAddress
CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
{
+ ListCell *lc;
Relation rel;
ObjectAddress myself;
Oid puboid;
@@ -764,8 +811,23 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
bool publish_via_partition_root_given;
bool publish_via_partition_root;
AclResult aclresult;
- List *relations = NIL;
- List *schemaidlist = NIL;
+ List *tables = NIL;
+ List *sequences = NIL;
+ List *tables_schemaidlist = NIL;
+ List *sequences_schemaidlist = NIL;
+
+ bool for_all_tables = false;
+ bool for_all_sequences = false;
+
+ /* Translate the list of object types (represented by strings) to bool flags. */
+ foreach (lc, stmt->for_all_objects)
+ {
+ char *val = strVal(lfirst(lc));
+ if (strcmp(val, "tables") == 0)
+ for_all_tables = true;
+ else if (strcmp(val, "sequences") == 0)
+ for_all_sequences = true;
+ }
/* must have CREATE privilege on database */
aclresult = pg_database_aclcheck(MyDatabaseId, GetUserId(), ACL_CREATE);
@@ -774,11 +836,17 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
get_database_name(MyDatabaseId));
/* FOR ALL TABLES requires superuser */
- if (stmt->for_all_tables && !superuser())
+ if (for_all_tables && !superuser())
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to create FOR ALL TABLES publication")));
+ /* FOR ALL SEQUENCES requires superuser */
+ if (for_all_sequences && !superuser())
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be superuser to create FOR ALL SEQUENCES publication")));
+
rel = table_open(PublicationRelationId, RowExclusiveLock);
/* Check if name is used */
@@ -810,7 +878,9 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
Anum_pg_publication_oid);
values[Anum_pg_publication_oid - 1] = ObjectIdGetDatum(puboid);
values[Anum_pg_publication_puballtables - 1] =
- BoolGetDatum(stmt->for_all_tables);
+ BoolGetDatum(for_all_tables);
+ values[Anum_pg_publication_puballsequences - 1] =
+ BoolGetDatum(for_all_sequences);
values[Anum_pg_publication_pubinsert - 1] =
BoolGetDatum(pubactions.pubinsert);
values[Anum_pg_publication_pubupdate - 1] =
@@ -819,6 +889,8 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
BoolGetDatum(pubactions.pubdelete);
values[Anum_pg_publication_pubtruncate - 1] =
BoolGetDatum(pubactions.pubtruncate);
+ values[Anum_pg_publication_pubsequence - 1] =
+ BoolGetDatum(pubactions.pubsequence);
values[Anum_pg_publication_pubviaroot - 1] =
BoolGetDatum(publish_via_partition_root);
@@ -836,28 +908,42 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
CommandCounterIncrement();
/* Associate objects with the publication. */
- if (stmt->for_all_tables)
+ if (for_all_tables || for_all_sequences)
{
/* Invalidate relcache so that publication info is rebuilt. */
CacheInvalidateRelcacheAll();
}
- else
+
+ /*
+ * If the publication might have either tables or sequences (directly or
+ * through a schema), process that.
+ */
+ if (!for_all_tables || !for_all_sequences)
{
- ObjectsInPublicationToOids(stmt->pubobjects, pstate, &relations,
- &schemaidlist);
+ ObjectsInPublicationToOids(stmt->pubobjects, pstate,
+ &tables, &sequences,
+ &tables_schemaidlist,
+ &sequences_schemaidlist);
/* FOR ALL TABLES IN SCHEMA requires superuser */
- if (list_length(schemaidlist) > 0 && !superuser())
+ if (list_length(tables_schemaidlist) > 0 && !superuser())
ereport(ERROR,
errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to create FOR ALL TABLES IN SCHEMA publication"));
- if (list_length(relations) > 0)
+ /* FOR ALL SEQUENCES IN SCHEMA requires superuser */
+ if (list_length(sequences_schemaidlist) > 0 && !superuser())
+ ereport(ERROR,
+ errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be superuser to create FOR ALL SEQUENCES IN SCHEMA publication"));
+
+ /* tables added directly */
+ if (list_length(tables) > 0)
{
List *rels;
- rels = OpenTableList(relations);
- CheckObjSchemaNotAlreadyInPublication(rels, schemaidlist,
+ rels = OpenRelationList(tables, PUB_OBJTYPE_TABLE);
+ CheckObjSchemaNotAlreadyInPublication(rels, tables_schemaidlist,
PUBLICATIONOBJ_TABLE);
TransformPubWhereClauses(rels, pstate->p_sourcetext,
@@ -866,18 +952,46 @@ CreatePublication(ParseState *pstate, CreatePublicationStmt *stmt)
CheckPubRelationColumnList(rels, pstate->p_sourcetext,
publish_via_partition_root);
- PublicationAddTables(puboid, rels, true, NULL);
- CloseTableList(rels);
+ PublicationAddRelations(puboid, rels, true, NULL);
+ CloseRelationList(rels);
}
- if (list_length(schemaidlist) > 0)
+ /* sequences added directly */
+ if (list_length(sequences) > 0)
+ {
+ List *rels;
+
+ rels = OpenRelationList(sequences, PUB_OBJTYPE_SEQUENCE);
+ CheckObjSchemaNotAlreadyInPublication(rels, sequences_schemaidlist,
+ PUBLICATIONOBJ_SEQUENCE);
+ PublicationAddRelations(puboid, rels, true, NULL);
+ CloseRelationList(rels);
+ }
+
+ /* tables added through a schema */
+ if (list_length(tables_schemaidlist) > 0)
{
/*
* Schema lock is held until the publication is created to prevent
* concurrent schema deletion.
*/
- LockSchemaList(schemaidlist);
- PublicationAddSchemas(puboid, schemaidlist, true, NULL);
+ LockSchemaList(tables_schemaidlist);
+ PublicationAddSchemas(puboid,
+ tables_schemaidlist, PUB_OBJTYPE_TABLE,
+ true, NULL);
+ }
+
+ /* sequences added through a schema */
+ if (list_length(sequences_schemaidlist) > 0)
+ {
+ /*
+ * Schema lock is held until the publication is created to prevent
+ * concurrent schema deletion.
+ */
+ LockSchemaList(sequences_schemaidlist);
+ PublicationAddSchemas(puboid,
+ sequences_schemaidlist, PUB_OBJTYPE_SEQUENCE,
+ true, NULL);
}
}
@@ -941,6 +1055,7 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
AccessShareLock);
root_relids = GetPublicationRelations(pubform->oid,
+ PUB_OBJTYPE_TABLE,
PUBLICATION_PART_ROOT);
foreach(lc, root_relids)
@@ -1020,6 +1135,9 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
values[Anum_pg_publication_pubtruncate - 1] = BoolGetDatum(pubactions.pubtruncate);
replaces[Anum_pg_publication_pubtruncate - 1] = true;
+
+ values[Anum_pg_publication_pubsequence - 1] = BoolGetDatum(pubactions.pubsequence);
+ replaces[Anum_pg_publication_pubsequence - 1] = true;
}
if (publish_via_partition_root_given)
@@ -1039,7 +1157,7 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
pubform = (Form_pg_publication) GETSTRUCT(tup);
/* Invalidate the relcache. */
- if (pubform->puballtables)
+ if (pubform->puballtables || pubform->puballsequences)
{
CacheInvalidateRelcacheAll();
}
@@ -1055,6 +1173,7 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
*/
if (root_relids == NIL)
relids = GetPublicationRelations(pubform->oid,
+ PUB_OBJTYPE_TABLE,
PUBLICATION_PART_ALL);
else
{
@@ -1068,7 +1187,20 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt,
lfirst_oid(lc));
}
+ /* tables */
+ schemarelids = GetAllSchemaPublicationRelations(pubform->oid,
+ PUB_OBJTYPE_TABLE,
+ PUBLICATION_PART_ALL);
+ relids = list_concat_unique_oid(relids, schemarelids);
+
+ /* sequences */
+ relids = list_concat_unique_oid(relids,
+ GetPublicationRelations(pubform->oid,
+ PUB_OBJTYPE_SEQUENCE,
+ PUBLICATION_PART_ALL));
+
schemarelids = GetAllSchemaPublicationRelations(pubform->oid,
+ PUB_OBJTYPE_SEQUENCE,
PUBLICATION_PART_ALL);
relids = list_concat_unique_oid(relids, schemarelids);
@@ -1123,7 +1255,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
if (!tables && stmt->action != AP_SetObjects)
return;
- rels = OpenTableList(tables);
+ rels = OpenRelationList(tables, PUB_OBJTYPE_TABLE);
if (stmt->action == AP_AddObjects)
{
@@ -1133,7 +1265,9 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
* Check if the relation is member of the existing schema in the
* publication or member of the schema list specified.
*/
- schemas = list_concat_copy(schemaidlist, GetPublicationSchemas(pubid));
+ schemas = list_concat_copy(schemaidlist,
+ GetPublicationSchemas(pubid,
+ PUB_OBJTYPE_TABLE));
CheckObjSchemaNotAlreadyInPublication(rels, schemas,
PUBLICATIONOBJ_TABLE);
@@ -1141,13 +1275,14 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
CheckPubRelationColumnList(rels, queryString, pubform->pubviaroot);
- PublicationAddTables(pubid, rels, false, stmt);
+ PublicationAddRelations(pubid, rels, false, stmt);
}
else if (stmt->action == AP_DropObjects)
- PublicationDropTables(pubid, rels, false);
+ PublicationDropRelations(pubid, rels, false);
else /* AP_SetObjects */
{
List *oldrelids = GetPublicationRelations(pubid,
+ PUB_OBJTYPE_TABLE,
PUBLICATION_PART_ROOT);
List *delrels = NIL;
ListCell *oldlc;
@@ -1266,18 +1401,18 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
}
/* And drop them. */
- PublicationDropTables(pubid, delrels, true);
+ PublicationDropRelations(pubid, delrels, true);
/*
* Don't bother calculating the difference for adding, we'll catch and
* skip existing ones when doing catalog update.
*/
- PublicationAddTables(pubid, rels, true, stmt);
+ PublicationAddRelations(pubid, rels, true, stmt);
- CloseTableList(delrels);
+ CloseRelationList(delrels);
}
- CloseTableList(rels);
+ CloseRelationList(rels);
}
/*
@@ -1287,7 +1422,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, HeapTuple tup,
*/
static void
AlterPublicationSchemas(AlterPublicationStmt *stmt,
- HeapTuple tup, List *schemaidlist)
+ HeapTuple tup, List *schemaidlist,
+ char objectType)
{
Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
@@ -1309,20 +1445,20 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
List *rels;
List *reloids;
- reloids = GetPublicationRelations(pubform->oid, PUBLICATION_PART_ROOT);
+ reloids = GetPublicationRelations(pubform->oid, objectType, PUBLICATION_PART_ROOT);
rels = OpenRelIdList(reloids);
CheckObjSchemaNotAlreadyInPublication(rels, schemaidlist,
PUBLICATIONOBJ_TABLES_IN_SCHEMA);
- CloseTableList(rels);
- PublicationAddSchemas(pubform->oid, schemaidlist, false, stmt);
+ CloseRelationList(rels);
+ PublicationAddSchemas(pubform->oid, schemaidlist, objectType, false, stmt);
}
else if (stmt->action == AP_DropObjects)
- PublicationDropSchemas(pubform->oid, schemaidlist, false);
+ PublicationDropSchemas(pubform->oid, schemaidlist, objectType, false);
else /* AP_SetObjects */
{
- List *oldschemaids = GetPublicationSchemas(pubform->oid);
+ List *oldschemaids = GetPublicationSchemas(pubform->oid, objectType);
List *delschemas = NIL;
/* Identify which schemas should be dropped */
@@ -1335,13 +1471,13 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
LockSchemaList(delschemas);
/* And drop them */
- PublicationDropSchemas(pubform->oid, delschemas, true);
+ PublicationDropSchemas(pubform->oid, delschemas, objectType, true);
/*
* Don't bother calculating the difference for adding, we'll catch and
* skip existing ones when doing catalog update.
*/
- PublicationAddSchemas(pubform->oid, schemaidlist, true, stmt);
+ PublicationAddSchemas(pubform->oid, schemaidlist, objectType, true, stmt);
}
}
@@ -1351,12 +1487,13 @@ AlterPublicationSchemas(AlterPublicationStmt *stmt,
*/
static void
CheckAlterPublication(AlterPublicationStmt *stmt, HeapTuple tup,
- List *tables, List *schemaidlist)
+ List *tables, List *tables_schemaidlist,
+ List *sequences, List *sequences_schemaidlist)
{
Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
if ((stmt->action == AP_AddObjects || stmt->action == AP_SetObjects) &&
- schemaidlist && !superuser())
+ (tables_schemaidlist || sequences_schemaidlist) && !superuser())
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to add or set schemas")));
@@ -1365,13 +1502,24 @@ CheckAlterPublication(AlterPublicationStmt *stmt, HeapTuple tup,
* Check that user is allowed to manipulate the publication tables in
* schema
*/
- if (schemaidlist && pubform->puballtables)
+ if (tables_schemaidlist && pubform->puballtables)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("publication \"%s\" is defined as FOR ALL TABLES",
NameStr(pubform->pubname)),
errdetail("Tables from schema cannot be added to, dropped from, or set on FOR ALL TABLES publications.")));
+ /*
+ * Check that user is allowed to manipulate the publication sequences in
+ * schema
+ */
+ if (sequences_schemaidlist && pubform->puballsequences)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("publication \"%s\" is defined as FOR ALL SEQUENCES",
+ NameStr(pubform->pubname)),
+ errdetail("Sequences from schema cannot be added to, dropped from, or set on FOR ALL SEQUENCES publications.")));
+
/* Check that user is allowed to manipulate the publication tables. */
if (tables && pubform->puballtables)
ereport(ERROR,
@@ -1379,6 +1527,108 @@ CheckAlterPublication(AlterPublicationStmt *stmt, HeapTuple tup,
errmsg("publication \"%s\" is defined as FOR ALL TABLES",
NameStr(pubform->pubname)),
errdetail("Tables cannot be added to or dropped from FOR ALL TABLES publications.")));
+
+ /* Check that user is allowed to manipulate the publication sequences. */
+ if (sequences && pubform->puballsequences)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("publication \"%s\" is defined as FOR ALL SEQUENCES",
+ NameStr(pubform->pubname)),
+ errdetail("Sequences cannot be added to or dropped from FOR ALL SEQUENCES publications.")));
+}
+
+/*
+ * Add or remove sequence to/from publication.
+ */
+static void
+AlterPublicationSequences(AlterPublicationStmt *stmt, HeapTuple tup,
+ List *sequences, List *schemaidlist)
+{
+ List *rels = NIL;
+ Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
+ Oid pubid = pubform->oid;
+
+ /*
+ * It is quite possible that for the SET case user has not specified any
+ * tables in which case we need to remove all the existing tables.
+ */
+ if (!sequences && stmt->action != AP_SetObjects)
+ return;
+
+ rels = OpenRelationList(sequences, PUB_OBJTYPE_SEQUENCE);
+
+ if (stmt->action == AP_AddObjects)
+ {
+ List *schemas = NIL;
+
+ /*
+ * Check if the relation is member of the existing schema in the
+ * publication or member of the schema list specified.
+ */
+ schemas = list_concat_copy(schemaidlist,
+ GetPublicationSchemas(pubid,
+ PUB_OBJTYPE_SEQUENCE));
+ CheckObjSchemaNotAlreadyInPublication(rels, schemas,
+ PUBLICATIONOBJ_SEQUENCE);
+ PublicationAddRelations(pubid, rels, false, stmt);
+ }
+ else if (stmt->action == AP_DropObjects)
+ PublicationDropRelations(pubid, rels, false);
+ else /* DEFELEM_SET */
+ {
+ List *oldrelids = GetPublicationRelations(pubid,
+ PUB_OBJTYPE_SEQUENCE,
+ PUBLICATION_PART_ROOT);
+ List *delrels = NIL;
+ ListCell *oldlc;
+
+ CheckObjSchemaNotAlreadyInPublication(rels, schemaidlist,
+ PUBLICATIONOBJ_SEQUENCE);
+
+ /* Calculate which relations to drop. */
+ foreach(oldlc, oldrelids)
+ {
+ Oid oldrelid = lfirst_oid(oldlc);
+ ListCell *newlc;
+ PublicationRelInfo *oldrel;
+ bool found = false;
+
+ foreach(newlc, rels)
+ {
+ PublicationRelInfo *newpubrel;
+
+ newpubrel = (PublicationRelInfo *) lfirst(newlc);
+ if (RelationGetRelid(newpubrel->relation) == oldrelid)
+ {
+ found = true;
+ break;
+ }
+ }
+ /* Not yet in the list, open it and add to the list */
+ if (!found)
+ {
+ oldrel = palloc(sizeof(PublicationRelInfo));
+ oldrel->whereClause = NULL;
+ oldrel->columns = NULL;
+ oldrel->relation = table_open(oldrelid,
+ ShareUpdateExclusiveLock);
+ delrels = lappend(delrels, oldrel);
+ }
+ }
+
+ /* And drop them. */
+ PublicationDropRelations(pubid, delrels, true);
+
+ /*
+ * Don't bother calculating the difference for adding, we'll catch and
+ * skip existing ones when doing catalog update.
+ */
+ PublicationAddRelations(pubid, rels, true, stmt);
+
+ CloseRelationList(delrels);
+ }
+
+ CloseRelationList(rels);
}
/*
@@ -1416,14 +1666,20 @@ AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt)
AlterPublicationOptions(pstate, stmt, rel, tup);
else
{
- List *relations = NIL;
- List *schemaidlist = NIL;
+ List *tables = NIL;
+ List *sequences = NIL;
+ List *tables_schemaidlist = NIL;
+ List *sequences_schemaidlist = NIL;
Oid pubid = pubform->oid;
- ObjectsInPublicationToOids(stmt->pubobjects, pstate, &relations,
- &schemaidlist);
+ ObjectsInPublicationToOids(stmt->pubobjects, pstate,
+ &tables, &sequences,
+ &tables_schemaidlist,
+ &sequences_schemaidlist);
- CheckAlterPublication(stmt, tup, relations, schemaidlist);
+ CheckAlterPublication(stmt, tup,
+ tables, tables_schemaidlist,
+ sequences, sequences_schemaidlist);
heap_freetuple(tup);
@@ -1451,9 +1707,16 @@ AlterPublication(ParseState *pstate, AlterPublicationStmt *stmt)
errmsg("publication \"%s\" does not exist",
stmt->pubname));
- AlterPublicationTables(stmt, tup, relations, schemaidlist,
+ AlterPublicationTables(stmt, tup, tables, tables_schemaidlist,
pstate->p_sourcetext);
- AlterPublicationSchemas(stmt, tup, schemaidlist);
+
+ AlterPublicationSequences(stmt, tup, sequences, sequences_schemaidlist);
+
+ AlterPublicationSchemas(stmt, tup, tables_schemaidlist,
+ PUB_OBJTYPE_TABLE);
+
+ AlterPublicationSchemas(stmt, tup, sequences_schemaidlist,
+ PUB_OBJTYPE_SEQUENCE);
}
/* Cleanup. */
@@ -1521,7 +1784,7 @@ RemovePublicationById(Oid pubid)
pubform = (Form_pg_publication) GETSTRUCT(tup);
/* Invalidate relcache so that publication info is rebuilt. */
- if (pubform->puballtables)
+ if (pubform->puballtables || pubform->puballsequences)
CacheInvalidateRelcacheAll();
CatalogTupleDelete(rel, &tup->t_self);
@@ -1557,6 +1820,7 @@ RemovePublicationSchemaById(Oid psoid)
* partitions.
*/
schemaRels = GetSchemaPublicationRelations(pubsch->pnnspid,
+ pubsch->pntype,
PUBLICATION_PART_ALL);
InvalidatePublicationRels(schemaRels);
@@ -1599,10 +1863,10 @@ OpenRelIdList(List *relids)
* add them to a publication.
*/
static List *
-OpenTableList(List *tables)
+OpenRelationList(List *rels, char objectType)
{
List *relids = NIL;
- List *rels = NIL;
+ List *result = NIL;
ListCell *lc;
List *relids_with_rf = NIL;
List *relids_with_collist = NIL;
@@ -1610,19 +1874,35 @@ OpenTableList(List *tables)
/*
* Open, share-lock, and check all the explicitly-specified relations
*/
- foreach(lc, tables)
+ foreach(lc, rels)
{
PublicationTable *t = lfirst_node(PublicationTable, lc);
bool recurse = t->relation->inh;
Relation rel;
Oid myrelid;
PublicationRelInfo *pub_rel;
+ char myrelkind;
/* Allow query cancel in case this takes a long time */
CHECK_FOR_INTERRUPTS();
rel = table_openrv(t->relation, ShareUpdateExclusiveLock);
myrelid = RelationGetRelid(rel);
+ myrelkind = get_rel_relkind(myrelid);
+
+ /*
+ * Make sure the relkind matches the expected object type. This may
+ * happen e.g. when adding a sequence using ADD TABLE or a table
+ * using ADD SEQUENCE).
+ *
+ * XXX We let through unsupported object types (views etc.). Those
+ * will be caught later in check_publication_add_relation.
+ */
+ if (pub_get_object_type_for_relkind(myrelkind) != PUB_OBJTYPE_UNSUPPORTED &&
+ pub_get_object_type_for_relkind(myrelkind) != objectType)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("object type does not match type expected by command"));
/*
* Filter out duplicates if user specifies "foo, foo".
@@ -1655,7 +1935,7 @@ OpenTableList(List *tables)
pub_rel->relation = rel;
pub_rel->whereClause = t->whereClause;
pub_rel->columns = t->columns;
- rels = lappend(rels, pub_rel);
+ result = lappend(result, pub_rel);
relids = lappend_oid(relids, myrelid);
if (t->whereClause)
@@ -1724,10 +2004,9 @@ OpenTableList(List *tables)
pub_rel->relation = rel;
/* child inherits WHERE clause from parent */
pub_rel->whereClause = t->whereClause;
-
/* child inherits column list from parent */
pub_rel->columns = t->columns;
- rels = lappend(rels, pub_rel);
+ result = lappend(result, pub_rel);
relids = lappend_oid(relids, childrelid);
if (t->whereClause)
@@ -1742,14 +2021,14 @@ OpenTableList(List *tables)
list_free(relids);
list_free(relids_with_rf);
- return rels;
+ return result;
}
/*
* Close all relations in the list.
*/
static void
-CloseTableList(List *rels)
+CloseRelationList(List *rels)
{
ListCell *lc;
@@ -1797,12 +2076,12 @@ LockSchemaList(List *schemalist)
* Add listed tables to the publication.
*/
static void
-PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
+PublicationAddRelations(Oid pubid, List *rels, bool if_not_exists,
AlterPublicationStmt *stmt)
{
ListCell *lc;
- Assert(!stmt || !stmt->for_all_tables);
+ Assert(!stmt || !stmt->for_all_objects);
foreach(lc, rels)
{
@@ -1831,7 +2110,7 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
* Remove listed tables from the publication.
*/
static void
-PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
+PublicationDropRelations(Oid pubid, List *rels, bool missing_ok)
{
ObjectAddress obj;
ListCell *lc;
@@ -1876,19 +2155,19 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
* Add listed schemas to the publication.
*/
static void
-PublicationAddSchemas(Oid pubid, List *schemas, bool if_not_exists,
- AlterPublicationStmt *stmt)
+PublicationAddSchemas(Oid pubid, List *schemas, char objectType,
+ bool if_not_exists, AlterPublicationStmt *stmt)
{
ListCell *lc;
- Assert(!stmt || !stmt->for_all_tables);
+ Assert(!stmt || !stmt->for_all_objects);
foreach(lc, schemas)
{
Oid schemaid = lfirst_oid(lc);
ObjectAddress obj;
- obj = publication_add_schema(pubid, schemaid, if_not_exists);
+ obj = publication_add_schema(pubid, schemaid, objectType, if_not_exists);
if (stmt)
{
EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
@@ -1904,7 +2183,7 @@ PublicationAddSchemas(Oid pubid, List *schemas, bool if_not_exists,
* Remove listed schemas from the publication.
*/
static void
-PublicationDropSchemas(Oid pubid, List *schemas, bool missing_ok)
+PublicationDropSchemas(Oid pubid, List *schemas, char objectType, bool missing_ok)
{
ObjectAddress obj;
ListCell *lc;
@@ -1914,10 +2193,11 @@ PublicationDropSchemas(Oid pubid, List *schemas, bool missing_ok)
{
Oid schemaid = lfirst_oid(lc);
- psid = GetSysCacheOid2(PUBLICATIONNAMESPACEMAP,
+ psid = GetSysCacheOid3(PUBLICATIONNAMESPACEMAP,
Anum_pg_publication_namespace_oid,
ObjectIdGetDatum(schemaid),
- ObjectIdGetDatum(pubid));
+ ObjectIdGetDatum(pubid),
+ CharGetDatum(objectType));
if (!OidIsValid(psid))
{
if (missing_ok)
@@ -1972,6 +2252,13 @@ AlterPublicationOwner_internal(Relation rel, HeapTuple tup, Oid newOwnerId)
NameStr(form->pubname)),
errhint("The owner of a FOR ALL TABLES publication must be a superuser.")));
+ if (form->puballsequences && !superuser_arg(newOwnerId))
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("permission denied to change owner of publication \"%s\"",
+ NameStr(form->pubname)),
+ errhint("The owner of a FOR ALL SEQUENCES publication must be a superuser.")));
+
if (!superuser_arg(newOwnerId) && is_schema_publication(form->oid))
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
diff --git a/src/backend/commands/sequence.c b/src/backend/commands/sequence.c
index ddf219b21f5..297245bd78c 100644
--- a/src/backend/commands/sequence.c
+++ b/src/backend/commands/sequence.c
@@ -38,6 +38,7 @@
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "parser/parse_type.h"
+#include "replication/message.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
#include "storage/smgr.h"
@@ -76,6 +77,7 @@ typedef struct SeqTableData
{
Oid relid; /* pg_class OID of this sequence (hash key) */
Oid filenode; /* last seen relfilenode of this sequence */
+ Oid tablespace; /* last seen tablespace of this sequence */
LocalTransactionId lxid; /* xact in which we last did a seq op */
bool last_valid; /* do we have a valid "last" value? */
int64 last; /* value last returned by nextval */
@@ -83,6 +85,7 @@ typedef struct SeqTableData
/* if last != cached, we have not used up all the cached values */
int64 increment; /* copy of sequence's increment field */
/* note that increment is zero until we first do nextval_internal() */
+ bool need_log; /* should be written to WAL at commit? */
} SeqTableData;
typedef SeqTableData *SeqTable;
@@ -332,6 +335,69 @@ ResetSequence(Oid seq_relid)
relation_close(seq_rel, NoLock);
}
+/*
+ * Update the sequence state by creating a new relfilenode.
+ *
+ * This creates a new relfilenode, to allow transactional behavior.
+ */
+void
+SetSequence(Oid seq_relid, int64 last_value, int64 log_cnt, bool is_called)
+{
+ SeqTable elm;
+ Relation seqrel;
+ Buffer buf;
+ HeapTupleData seqdatatuple;
+ Form_pg_sequence_data seq;
+ HeapTuple tuple;
+
+ /* open and lock sequence */
+ init_sequence(seq_relid, &elm, &seqrel);
+
+ /* lock page' buffer and read tuple */
+ seq = read_seq_tuple(seqrel, &buf, &seqdatatuple);
+
+ /* Copy the existing sequence tuple. */
+ tuple = heap_copytuple(&seqdatatuple);
+
+ /* Now we're done with the old page */
+ UnlockReleaseBuffer(buf);
+
+ /*
+ * Modify the copied tuple to update the sequence state (similar to what
+ * ResetSequence does).
+ */
+ seq = (Form_pg_sequence_data) GETSTRUCT(tuple);
+ seq->last_value = last_value;
+ seq->is_called = is_called;
+ seq->log_cnt = log_cnt;
+
+ /*
+ * Create a new storage file for the sequence - this is needed for the
+ * transactional behavior.
+ */
+ RelationSetNewRelfilenode(seqrel, seqrel->rd_rel->relpersistence);
+
+ /*
+ * Ensure sequence's relfrozenxid is at 0, since it won't contain any
+ * unfrozen XIDs. Same with relminmxid, since a sequence will never
+ * contain multixacts.
+ */
+ Assert(seqrel->rd_rel->relfrozenxid == InvalidTransactionId);
+ Assert(seqrel->rd_rel->relminmxid == InvalidMultiXactId);
+
+ /*
+ * Insert the modified tuple into the new storage file. This does all the
+ * necessary WAL-logging etc.
+ */
+ fill_seq_with_data(seqrel, tuple);
+
+ /* Clear local cache so that we don't think we have cached numbers */
+ /* Note that we do not change the currval() state */
+ elm->cached = elm->last;
+
+ relation_close(seqrel, NoLock);
+}
+
/*
* Initialize a sequence's relation with the specified tuple as content
*
@@ -396,10 +462,21 @@ fill_seq_fork_with_data(Relation rel, HeapTuple tuple, ForkNumber forkNum)
tuple->t_data->t_infomask |= HEAP_XMAX_INVALID;
ItemPointerSet(&tuple->t_data->t_ctid, 0, FirstOffsetNumber);
- /* check the comment above nextval_internal()'s equivalent call. */
+ /*
+ * Remember we need to write this sequence to WAL at commit, and make sure
+ * we have XID if we may need to decode this sequence.
+ *
+ * XXX Maybe we should not be doing this except when actually reading a
+ * value from the sequence?
+ */
if (RelationNeedsWAL(rel))
+ {
GetTopTransactionId();
+ if (XLogLogicalInfoActive())
+ GetCurrentTransactionId();
+ }
+
START_CRIT_SECTION();
MarkBufferDirty(buf);
@@ -644,6 +721,23 @@ nextval_internal(Oid relid, bool check_permissions)
/* open and lock sequence */
init_sequence(relid, &elm, &seqrel);
+ /*
+ * Remember we need to write this sequence to WAL at commit, and make sure
+ * we have XID if we may need to decode this sequence.
+ *
+ * XXX Maybe we should not be doing this except when actually reading a
+ * value from the sequence?
+ */
+ if (RelationNeedsWAL(seqrel))
+ {
+ elm->need_log = true;
+
+ GetTopTransactionId();
+
+ if (XLogLogicalInfoActive())
+ GetCurrentTransactionId();
+ }
+
if (check_permissions &&
pg_class_aclcheck(elm->relid, GetUserId(),
ACL_USAGE | ACL_UPDATE) != ACLCHECK_OK)
@@ -798,10 +892,28 @@ nextval_internal(Oid relid, bool check_permissions)
* It's sufficient to ensure the toplevel transaction has an xid, no need
* to assign xids subxacts, that'll already trigger an appropriate wait.
* (Have to do that here, so we're outside the critical section)
+ *
+ * We have to ensure we have a proper XID, which will be included in
+ * the XLOG record by XLogRecordAssemble. Otherwise the first nextval()
+ * in a subxact (without any preceding changes) would get XID 0, and it
+ * would then be impossible to decide which top xact it belongs to.
+ * It'd also trigger assert in DecodeSequence. We only do that with
+ * wal_level=logical, though.
+ *
+ * XXX This might seem unnecessary, because if there's no XID the xact
+ * couldn't have done anything important yet, e.g. it could not have
+ * created a sequence. But that's incorrect, because of subxacts. The
+ * current subtransaction might not have done anything yet (thus no XID),
+ * but an earlier one might have created the sequence.
*/
if (logit && RelationNeedsWAL(seqrel))
+ {
GetTopTransactionId();
+ if (XLogLogicalInfoActive())
+ GetCurrentTransactionId();
+ }
+
/* ready to change the on-disk (or really, in-buffer) tuple */
START_CRIT_SECTION();
@@ -955,6 +1067,23 @@ do_setval(Oid relid, int64 next, bool iscalled)
/* open and lock sequence */
init_sequence(relid, &elm, &seqrel);
+ /*
+ * Remember we need to write this sequence to WAL at commit, and make sure
+ * we have XID if we may need to decode this sequence.
+ *
+ * XXX Maybe we should not be doing this except when actually reading a
+ * value from the sequence?
+ */
+ if (RelationNeedsWAL(seqrel))
+ {
+ elm->need_log = true;
+
+ GetTopTransactionId();
+
+ if (XLogLogicalInfoActive())
+ GetCurrentTransactionId();
+ }
+
if (pg_class_aclcheck(elm->relid, GetUserId(), ACL_UPDATE) != ACLCHECK_OK)
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
@@ -1002,8 +1131,13 @@ do_setval(Oid relid, int64 next, bool iscalled)
/* check the comment above nextval_internal()'s equivalent call. */
if (RelationNeedsWAL(seqrel))
+ {
GetTopTransactionId();
+ if (XLogLogicalInfoActive())
+ GetCurrentTransactionId();
+ }
+
/* ready to change the on-disk (or really, in-buffer) tuple */
START_CRIT_SECTION();
@@ -1172,6 +1306,7 @@ init_sequence(Oid relid, SeqTable *p_elm, Relation *p_rel)
if (seqrel->rd_rel->relfilenode != elm->filenode)
{
elm->filenode = seqrel->rd_rel->relfilenode;
+ elm->tablespace = seqrel->rd_rel->reltablespace;
elm->cached = elm->last;
}
@@ -1907,3 +2042,121 @@ seq_mask(char *page, BlockNumber blkno)
mask_unused_space(page);
}
+
+
+static void
+read_sequence_info(Relation seqrel, int64 *last_value, int64 *log_cnt, bool *is_called)
+{
+ Buffer buf;
+ Form_pg_sequence_data seq;
+ HeapTupleData seqdatatuple;
+
+ seq = read_seq_tuple(seqrel, &buf, &seqdatatuple);
+
+ *last_value = seq->last_value;
+ *is_called = seq->is_called;
+ *log_cnt = seq->log_cnt;
+
+ UnlockReleaseBuffer(buf);
+}
+
+/* XXX Do this only for wal_level = logical, probably? */
+void
+AtEOXact_Sequences(bool isCommit)
+{
+ SeqTable entry;
+ HASH_SEQ_STATUS scan;
+
+ if (!seqhashtab)
+ return;
+
+ /* only do this with wal_level=logical */
+ if (!XLogLogicalInfoActive())
+ return;
+
+ /*
+ * XXX Maybe we could enforce having XID here? Or is it too late?
+ */
+ // Assert(GetTopTransactionId() != InvalidTransactionId);
+ // Assert(GetCurrentTransactionId() != InvalidTransactionId);
+
+ hash_seq_init(&scan, seqhashtab);
+
+ while ((entry = (SeqTable) hash_seq_search(&scan)))
+ {
+ Relation rel;
+ RelFileNode rnode;
+ xl_logical_sequence xlrec;
+
+ /* not commit, we don't guarantee any data to be durable */
+ if (!isCommit)
+ entry->need_log = false;
+
+ /*
+ * If not touched in the current transaction, don't log anything.
+ * We leave needs_log set, so that if future transactions touch
+ * the sequence we'll log it properly.
+ */
+ if (!entry->need_log)
+ continue;
+
+ /* if this is commit, we'll log the */
+ entry->need_log = false;
+
+ /*
+ * does the relation still exist?
+ *
+ * XXX We need to make sure another transaction can't jump ahead of
+ * this one, otherwise the ordering of sequence changes could change.
+ * Imagine T1 and T2, where T1 writes sequence state first, but then
+ * T2 does it too and commits first:
+ *
+ * T1: log sequence state
+ * T2: increment sequence
+ * T2: log sequence state
+ * T2: commit
+ * T1: commit
+ *
+ * If we apply the sequences in this order, it'd be broken as the
+ * value might go backwars. ShareUpdateExclusive protects against
+ * that, but it's also restricting commit throughtput, probably.
+ *
+ * XXX This might be an issue for deadlocks, too, if two xacts try
+ * to write sequences in different ordering. We may need to sort
+ * the OIDs first, to enforce the same lock ordering.
+ */
+ rel = try_relation_open(entry->relid, ShareUpdateExclusiveLock);
+
+ /* XXX relation might have been dropped, maybe we should have logged
+ * the last change? */
+ if (!rel)
+ continue;
+
+ /* tablespace */
+ if (OidIsValid(entry->tablespace))
+ rnode.spcNode = entry->tablespace;
+ else
+ rnode.spcNode = MyDatabaseTableSpace;
+
+ rnode.dbNode = MyDatabaseId; /* database */
+ rnode.relNode = entry->filenode; /* relation */
+
+ xlrec.node = rnode;
+ xlrec.reloid = entry->relid;
+
+ /* XXX is it good enough to log values we have in cache? seems
+ * wrong and we may need to re-read that. */
+ read_sequence_info(rel, &xlrec.last, &xlrec.log_cnt, &xlrec.is_called);
+
+ XLogBeginInsert();
+ XLogRegisterData((char *) &xlrec, SizeOfLogicalSequence);
+
+ /* allow origin filtering */
+ XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN);
+
+ (void) XLogInsert(RM_LOGICALMSG_ID, XLOG_LOGICAL_SEQUENCE);
+
+ /* hold the sequence lock until the end of the transaction */
+ relation_close(rel, NoLock);
+ }
+}
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 2e8d8afead8..057ab4b6a3f 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -90,6 +90,7 @@ typedef struct SubOpts
} SubOpts;
static List *fetch_table_list(WalReceiverConn *wrconn, List *publications);
+static List *fetch_sequence_list(WalReceiverConn *wrconn, List *publications);
static void check_duplicates_in_publist(List *publist, Datum *datums);
static List *merge_publications(List *oldpublist, List *newpublist, bool addpub, const char *subname);
static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err);
@@ -638,9 +639,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
{
char *err;
WalReceiverConn *wrconn;
- List *tables;
+ List *relations;
ListCell *lc;
- char table_state;
+ char sync_state;
/* Try to connect to the publisher. */
wrconn = walrcv_connect(conninfo, true, stmt->subname, &err);
@@ -657,14 +658,17 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
* Set sync state based on if we were asked to do data copy or
* not.
*/
- table_state = opts.copy_data ? SUBREL_STATE_INIT : SUBREL_STATE_READY;
+ sync_state = opts.copy_data ? SUBREL_STATE_INIT : SUBREL_STATE_READY;
/*
- * Get the table list from publisher and build local table status
- * info.
+ * Get the table and sequence list from publisher and build
+ * local relation sync status info.
*/
- tables = fetch_table_list(wrconn, publications);
- foreach(lc, tables)
+ relations = fetch_table_list(wrconn, publications);
+ relations = list_concat(relations,
+ fetch_sequence_list(wrconn, publications));
+
+ foreach(lc, relations)
{
RangeVar *rv = (RangeVar *) lfirst(lc);
Oid relid;
@@ -675,7 +679,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
CheckSubscriptionRelkind(get_rel_relkind(relid),
rv->schemaname, rv->relname);
- AddSubscriptionRelState(subid, relid, table_state,
+ AddSubscriptionRelState(subid, relid, sync_state,
InvalidXLogRecPtr);
}
@@ -701,12 +705,12 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
*
* Note that if tables were specified but copy_data is false
* then it is safe to enable two_phase up-front because those
- * tables are already initially in READY state. When the
- * subscription has no tables, we leave the twophase state as
- * PENDING, to allow ALTER SUBSCRIPTION ... REFRESH
+ * relations are already initially in READY state. When the
+ * subscription has no relations, we leave the twophase state
+ * as PENDING, to allow ALTER SUBSCRIPTION ... REFRESH
* PUBLICATION to work.
*/
- if (opts.twophase && !opts.copy_data && tables != NIL)
+ if (opts.twophase && !opts.copy_data && relations != NIL)
twophase_enabled = true;
walrcv_create_slot(wrconn, opts.slot_name, false, twophase_enabled,
@@ -782,8 +786,10 @@ AlterSubscription_refresh(Subscription *sub, bool copy_data,
if (validate_publications)
check_publications(wrconn, validate_publications);
- /* Get the table list from publisher. */
+ /* Get the list of relations from publisher. */
pubrel_names = fetch_table_list(wrconn, sub->publications);
+ pubrel_names = list_concat(pubrel_names,
+ fetch_sequence_list(wrconn, sub->publications));
/* Get local table list. */
subrel_states = GetSubscriptionRelations(sub->oid);
@@ -1807,6 +1813,75 @@ fetch_table_list(WalReceiverConn *wrconn, List *publications)
return tablelist;
}
+/*
+ * Get the list of sequences which belong to specified publications on the
+ * publisher connection.
+ */
+static List *
+fetch_sequence_list(WalReceiverConn *wrconn, List *publications)
+{
+ WalRcvExecResult *res;
+ StringInfoData cmd;
+ TupleTableSlot *slot;
+ Oid tableRow[2] = {TEXTOID, TEXTOID};
+ ListCell *lc;
+ bool first;
+ List *tablelist = NIL;
+
+ Assert(list_length(publications) > 0);
+
+ initStringInfo(&cmd);
+ appendStringInfoString(&cmd, "SELECT DISTINCT s.schemaname, s.sequencename\n"
+ " FROM pg_catalog.pg_publication_sequences s\n"
+ " WHERE s.pubname IN (");
+ first = true;
+ foreach(lc, publications)
+ {
+ char *pubname = strVal(lfirst(lc));
+
+ if (first)
+ first = false;
+ else
+ appendStringInfoString(&cmd, ", ");
+
+ appendStringInfoString(&cmd, quote_literal_cstr(pubname));
+ }
+ appendStringInfoChar(&cmd, ')');
+
+ res = walrcv_exec(wrconn, cmd.data, 2, tableRow);
+ pfree(cmd.data);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not receive list of replicated sequences from the publisher: %s",
+ res->err)));
+
+ /* Process sequences. */
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ {
+ char *nspname;
+ char *relname;
+ bool isnull;
+ RangeVar *rv;
+
+ nspname = TextDatumGetCString(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+ relname = TextDatumGetCString(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ rv = makeRangeVar(nspname, relname, -1);
+ tablelist = lappend(tablelist, rv);
+
+ ExecClearTuple(slot);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+
+ walrcv_clear_result(res);
+
+ return tablelist;
+}
+
/*
* This is to report the connection failure while dropping replication slots.
* Here, we report the WARNING for all tablesync slots so that user can drop
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 90edd0bb97d..4dd545cdd20 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -42,6 +42,7 @@
#include "catalog/pg_inherits.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_opclass.h"
+#include "catalog/pg_publication_namespace.h"
#include "catalog/pg_statistic_ext.h"
#include "catalog/pg_tablespace.h"
#include "catalog/pg_trigger.h"
@@ -16408,11 +16409,14 @@ AlterTableNamespace(AlterObjectSchemaStmt *stmt, Oid *oldschema)
* Check that setting the relation to a different schema won't result in a
* publication having both a schema and the same schema's table, as this
* is not supported.
+ *
+ * XXX We do this for tables and sequences, but it's better to keep the two
+ * blocks separate, to make the strings easier to translate.
*/
if (stmt->objectType == OBJECT_TABLE)
{
ListCell *lc;
- List *schemaPubids = GetSchemaPublications(nspOid);
+ List *schemaPubids = GetSchemaPublications(nspOid, PUB_OBJTYPE_TABLE);
List *relPubids = GetRelationPublications(RelationGetRelid(rel));
foreach(lc, relPubids)
@@ -16430,6 +16434,27 @@ AlterTableNamespace(AlterObjectSchemaStmt *stmt, Oid *oldschema)
get_publication_name(pubid, false)));
}
}
+ else if (stmt->objectType == OBJECT_SEQUENCE)
+ {
+ ListCell *lc;
+ List *schemaPubids = GetSchemaPublications(nspOid, PUB_OBJTYPE_SEQUENCE);
+ List *relPubids = GetRelationPublications(RelationGetRelid(rel));
+
+ foreach(lc, relPubids)
+ {
+ Oid pubid = lfirst_oid(lc);
+
+ if (list_member_oid(schemaPubids, pubid))
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot move sequence \"%s\" to schema \"%s\"",
+ RelationGetRelationName(rel), stmt->newschema),
+ errdetail("The schema \"%s\" and same schema's sequence \"%s\" cannot be part of the same publication \"%s\".",
+ stmt->newschema,
+ RelationGetRelationName(rel),
+ get_publication_name(pubid, false)));
+ }
+ }
/* common checks on switching namespaces */
CheckSetNamespace(oldNspOid, nspOid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 27989bd723d..228e3547012 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -649,7 +649,9 @@ void
CheckSubscriptionRelkind(char relkind, const char *nspname,
const char *relname)
{
- if (relkind != RELKIND_RELATION && relkind != RELKIND_PARTITIONED_TABLE)
+ if (relkind != RELKIND_RELATION &&
+ relkind != RELKIND_PARTITIONED_TABLE &&
+ relkind != RELKIND_SEQUENCE)
ereport(ERROR,
(errcode(ERRCODE_WRONG_OBJECT_TYPE),
errmsg("cannot use relation \"%s.%s\" as logical replication target",
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1585cf2d58c..46a1943d97a 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -5390,7 +5390,7 @@ _copyCreatePublicationStmt(const CreatePublicationStmt *from)
COPY_STRING_FIELD(pubname);
COPY_NODE_FIELD(options);
COPY_NODE_FIELD(pubobjects);
- COPY_SCALAR_FIELD(for_all_tables);
+ COPY_NODE_FIELD(for_all_objects);
return newnode;
}
@@ -5403,7 +5403,7 @@ _copyAlterPublicationStmt(const AlterPublicationStmt *from)
COPY_STRING_FIELD(pubname);
COPY_NODE_FIELD(options);
COPY_NODE_FIELD(pubobjects);
- COPY_SCALAR_FIELD(for_all_tables);
+ COPY_NODE_FIELD(for_all_objects);
COPY_SCALAR_FIELD(action);
return newnode;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index caad20e0478..1f765f42c91 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2688,7 +2688,7 @@ _equalCreatePublicationStmt(const CreatePublicationStmt *a,
COMPARE_STRING_FIELD(pubname);
COMPARE_NODE_FIELD(options);
COMPARE_NODE_FIELD(pubobjects);
- COMPARE_SCALAR_FIELD(for_all_tables);
+ COMPARE_NODE_FIELD(for_all_objects);
return true;
}
@@ -2700,7 +2700,7 @@ _equalAlterPublicationStmt(const AlterPublicationStmt *a,
COMPARE_STRING_FIELD(pubname);
COMPARE_NODE_FIELD(options);
COMPARE_NODE_FIELD(pubobjects);
- COMPARE_SCALAR_FIELD(for_all_tables);
+ COMPARE_NODE_FIELD(for_all_objects);
COMPARE_SCALAR_FIELD(action);
return true;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index c9941d9cb4f..2cc92a89432 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -455,7 +455,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
transform_element_list transform_type_list
TriggerTransitions TriggerReferencing
vacuum_relation_list opt_vacuum_relation_list
- drop_option_list pub_obj_list
+ drop_option_list pub_obj_list pub_obj_type_list
%type <node> opt_routine_body
%type <groupclause> group_clause
@@ -588,6 +588,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <node> var_value zone_value
%type <rolespec> auth_ident RoleSpec opt_granted_by
%type <publicationobjectspec> PublicationObjSpec
+%type <node> pub_obj_type
%type <keyword> unreserved_keyword type_func_name_keyword
%type <keyword> col_name_keyword reserved_keyword
@@ -9862,12 +9863,9 @@ AlterOwnerStmt: ALTER AGGREGATE aggregate_with_argtypes OWNER TO RoleSpec
*
* CREATE PUBLICATION FOR ALL TABLES [WITH options]
*
- * CREATE PUBLICATION FOR pub_obj [, ...] [WITH options]
- *
- * pub_obj is one of:
+ * CREATE PUBLICATION FOR ALL SEQUENCES [WITH options]
*
- * TABLE table [, ...]
- * ALL TABLES IN SCHEMA schema [, ...]
+ * CREATE PUBLICATION FOR pub_obj [, ...] [WITH options]
*
*****************************************************************************/
@@ -9879,12 +9877,12 @@ CreatePublicationStmt:
n->options = $4;
$$ = (Node *)n;
}
- | CREATE PUBLICATION name FOR ALL TABLES opt_definition
+ | CREATE PUBLICATION name FOR ALL pub_obj_type_list opt_definition
{
CreatePublicationStmt *n = makeNode(CreatePublicationStmt);
n->pubname = $3;
n->options = $7;
- n->for_all_tables = true;
+ n->for_all_objects = $6;
$$ = (Node *)n;
}
| CREATE PUBLICATION name FOR pub_obj_list opt_definition
@@ -9934,6 +9932,26 @@ PublicationObjSpec:
$$->pubobjtype = PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA;
$$->location = @5;
}
+ | SEQUENCE relation_expr
+ {
+ $$ = makeNode(PublicationObjSpec);
+ $$->pubobjtype = PUBLICATIONOBJ_SEQUENCE;
+ $$->pubtable = makeNode(PublicationTable);
+ $$->pubtable->relation = $2;
+ }
+ | ALL SEQUENCES IN_P SCHEMA ColId
+ {
+ $$ = makeNode(PublicationObjSpec);
+ $$->pubobjtype = PUBLICATIONOBJ_SEQUENCES_IN_SCHEMA;
+ $$->name = $5;
+ $$->location = @5;
+ }
+ | ALL SEQUENCES IN_P SCHEMA CURRENT_SCHEMA
+ {
+ $$ = makeNode(PublicationObjSpec);
+ $$->pubobjtype = PUBLICATIONOBJ_SEQUENCES_IN_CUR_SCHEMA;
+ $$->location = @5;
+ }
| ColId opt_column_list OptWhereClause
{
$$ = makeNode(PublicationObjSpec);
@@ -9995,6 +10013,19 @@ pub_obj_list: PublicationObjSpec
{ $$ = lappend($1, $3); }
;
+pub_obj_type: TABLES
+ { $$ = (Node *) makeString("tables"); }
+ | SEQUENCES
+ { $$ = (Node *) makeString("sequences"); }
+ ;
+
+pub_obj_type_list: pub_obj_type
+ { $$ = list_make1($1); }
+ | pub_obj_type_list ',' pub_obj_type
+ { $$ = lappend($1, $3); }
+ ;
+
+
/*****************************************************************************
*
* ALTER PUBLICATION name SET ( options )
@@ -10005,11 +10036,6 @@ pub_obj_list: PublicationObjSpec
*
* ALTER PUBLICATION name SET pub_obj [, ...]
*
- * pub_obj is one of:
- *
- * TABLE table_name [, ...]
- * ALL TABLES IN SCHEMA schema_name [, ...]
- *
*****************************************************************************/
AlterPublicationStmt:
@@ -18731,7 +18757,8 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
if (pubobj->pubobjtype == PUBLICATIONOBJ_CONTINUATION)
pubobj->pubobjtype = prevobjtype;
- if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLE)
+ if (pubobj->pubobjtype == PUBLICATIONOBJ_TABLE ||
+ pubobj->pubobjtype == PUBLICATIONOBJ_SEQUENCE)
{
/* relation name or pubtable must be set for this type of object */
if (!pubobj->name && !pubobj->pubtable)
@@ -18782,6 +18809,30 @@ preprocess_pubobj_list(List *pubobjspec_list, core_yyscan_t yyscanner)
errmsg("invalid schema name at or near"),
parser_errposition(pubobj->location));
}
+ else if (pubobj->pubobjtype == PUBLICATIONOBJ_SEQUENCES_IN_SCHEMA ||
+ pubobj->pubobjtype == PUBLICATIONOBJ_SEQUENCES_IN_CUR_SCHEMA)
+ {
+ /* WHERE clause is not allowed on a schema object */
+ if (pubobj->pubtable && pubobj->pubtable->whereClause)
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("WHERE clause not allowed for schema"),
+ parser_errposition(pubobj->location));
+
+ /*
+ * We can distinguish between the different type of schema
+ * objects based on whether name and pubtable is set.
+ */
+ if (pubobj->name)
+ pubobj->pubobjtype = PUBLICATIONOBJ_SEQUENCES_IN_SCHEMA;
+ else if (!pubobj->name && !pubobj->pubtable)
+ pubobj->pubobjtype = PUBLICATIONOBJ_SEQUENCES_IN_CUR_SCHEMA;
+ else
+ ereport(ERROR,
+ errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("invalid schema name at or near"),
+ parser_errposition(pubobj->location));
+ }
prevobjtype = pubobj->pubobjtype;
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 6303647fe0f..cf6464fa57d 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -42,6 +42,10 @@
#include "replication/reorderbuffer.h"
#include "replication/snapbuild.h"
#include "storage/standby.h"
+#include "commands/sequence.h"
+
+static void DecodeLogicalMessage(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
+static void DecodeLogicalSequence(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
/* individual record(group)'s handlers */
static void DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
@@ -558,6 +562,27 @@ FilterByOrigin(LogicalDecodingContext *ctx, RepOriginId origin_id)
*/
void
logicalmsg_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
+{
+ XLogReaderState *r = buf->record;
+ uint8 info = XLogRecGetInfo(r) & ~XLR_INFO_MASK;
+
+ switch (info)
+ {
+ case XLOG_LOGICAL_MESSAGE:
+ DecodeLogicalMessage(ctx, buf);
+ break;
+
+ case XLOG_LOGICAL_SEQUENCE:
+ DecodeLogicalSequence(ctx, buf);
+ break;
+
+ default:
+ elog(ERROR, "unexpected RM_LOGICALMSG_ID record type: %u", info);
+ }
+}
+
+static void
+DecodeLogicalMessage(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
{
SnapBuild *builder = ctx->snapshot_builder;
XLogReaderState *r = buf->record;
@@ -1250,3 +1275,67 @@ DecodeTXNNeedSkip(LogicalDecodingContext *ctx, XLogRecordBuffer *buf,
(txn_dbid != InvalidOid && txn_dbid != ctx->slot->data.database) ||
ctx->fast_forward || FilterByOrigin(ctx, origin_id));
}
+
+/*
+ * Handle sequence decode
+ *
+ * Decoding sequences is a bit tricky, because while most sequence actions
+ * are non-transactional (not subject to rollback), some need to be handled
+ * as transactional.
+ *
+ * By default, a sequence increment is non-transactional - we must not queue
+ * it in a transaction as other changes, because the transaction might get
+ * rolled back and we'd discard the increment. The downstream would not be
+ * notified about the increment, which is wrong.
+ *
+ * On the other hand, the sequence may be created in a transaction. In this
+ * case we *should* queue the change as other changes in the transaction,
+ * because we don't want to send the increments for unknown sequence to the
+ * plugin - it might get confused about which sequence it's related to etc.
+ */
+void
+DecodeLogicalSequence(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
+{
+ SnapBuild *builder = ctx->snapshot_builder;
+ RelFileNode target_node;
+ XLogReaderState *r = buf->record;
+ TransactionId xid = XLogRecGetXid(r);
+ uint8 info = XLogRecGetInfo(buf->record) & ~XLR_INFO_MASK;
+ xl_logical_sequence *xlrec;
+ RepOriginId origin_id = XLogRecGetOrigin(r);
+
+ /* only decode changes flagged with XLOG_SEQ_LOG */
+ if (info != XLOG_LOGICAL_SEQUENCE)
+ elog(ERROR, "unexpected RM_SEQ_ID record type: %u", info);
+
+ ReorderBufferProcessXid(ctx->reorder, XLogRecGetXid(r), buf->origptr);
+
+ /*
+ * If we don't have snapshot or we are just fast-forwarding, there is no
+ * point in decoding messages.
+ */
+ if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT ||
+ ctx->fast_forward)
+ return;
+
+ /* extract the WAL record, with "created" flag */
+ xlrec = (xl_logical_sequence *) XLogRecGetData(r);
+
+ /* only interested in our database */
+ target_node = xlrec->node;
+ if (target_node.dbNode != ctx->slot->data.database)
+ return;
+
+ /* output plugin doesn't look for this origin, no need to queue */
+ if (FilterByOrigin(ctx, XLogRecGetOrigin(r)))
+ return;
+
+ /* Skip the change if already processed (per the snapshot). */
+ if (!SnapBuildProcessChange(builder, xid, buf->origptr))
+ return;
+
+ /* Queue the increment (or send immediately if not transactional). */
+ ReorderBufferQueueSequence(ctx->reorder, xid, buf->endptr,
+ origin_id, xlrec->reloid, target_node,
+ xlrec->last, xlrec->log_cnt, xlrec->is_called);
+}
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 788769dd738..94b94e4be0c 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -73,6 +73,9 @@ static void truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+static void sequence_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr sequence_lsn, Relation rel,
+ int64 last_value, int64 log_cnt, bool is_called);
/* streaming callbacks */
static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
@@ -90,6 +93,9 @@ static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn
static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+static void stream_sequence_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr sequence_lsn, Relation rel,
+ int64 last_value, int64 log_cnt, bool is_called);
static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
int nrelations, Relation relations[], ReorderBufferChange *change);
@@ -218,6 +224,7 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->apply_truncate = truncate_cb_wrapper;
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ ctx->reorder->sequence = sequence_cb_wrapper;
/*
* To support streaming, we require start/stop/abort/commit/change
@@ -234,6 +241,7 @@ StartupDecodingContext(List *output_plugin_options,
(ctx->callbacks.stream_commit_cb != NULL) ||
(ctx->callbacks.stream_change_cb != NULL) ||
(ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_sequence_cb != NULL) ||
(ctx->callbacks.stream_truncate_cb != NULL);
/*
@@ -251,6 +259,7 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->stream_commit = stream_commit_cb_wrapper;
ctx->reorder->stream_change = stream_change_cb_wrapper;
ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_sequence = stream_sequence_cb_wrapper;
ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
@@ -1205,6 +1214,42 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+sequence_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr sequence_lsn, Relation rel,
+ int64 last_value, int64 log_cnt, bool is_called)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ if (ctx->callbacks.sequence_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "sequence";
+ state.report_location = sequence_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = sequence_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.sequence_cb(ctx, txn, sequence_lsn, rel,
+ last_value, log_cnt, is_called);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
static void
stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr first_lsn)
@@ -1510,6 +1555,46 @@ stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_sequence_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr sequence_lsn, Relation rel,
+ int64 last_value, int64 log_cnt, bool is_called)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_sequence_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_sequence";
+ state.report_location = sequence_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = sequence_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.sequence_cb(ctx, txn, sequence_lsn, rel,
+ last_value, log_cnt, is_called);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
static void
stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
int nrelations, Relation relations[],
diff --git a/src/backend/replication/logical/message.c b/src/backend/replication/logical/message.c
index 1c34912610e..69178dc1cfb 100644
--- a/src/backend/replication/logical/message.c
+++ b/src/backend/replication/logical/message.c
@@ -82,7 +82,8 @@ logicalmsg_redo(XLogReaderState *record)
{
uint8 info = XLogRecGetInfo(record) & ~XLR_INFO_MASK;
- if (info != XLOG_LOGICAL_MESSAGE)
+ if (info != XLOG_LOGICAL_MESSAGE &&
+ info != XLOG_LOGICAL_SEQUENCE)
elog(PANIC, "logicalmsg_redo: unknown op code %u", info);
/* This is only interesting for logical decoding, see decode.c. */
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index ff8513e2d29..ae078eed048 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -662,6 +662,54 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
pq_sendbytes(out, message, sz);
}
+/*
+ * Write SEQUENCE to stream
+ */
+void
+logicalrep_write_sequence(StringInfo out, Relation rel, TransactionId xid,
+ XLogRecPtr lsn,
+ int64 last_value, int64 log_cnt, bool is_called)
+{
+ uint8 flags = 0;
+ char *relname;
+
+ pq_sendbyte(out, LOGICAL_REP_MSG_SEQUENCE);
+
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
+ pq_sendint8(out, flags);
+ pq_sendint64(out, lsn);
+
+ logicalrep_write_namespace(out, RelationGetNamespace(rel));
+ relname = RelationGetRelationName(rel);
+ pq_sendstring(out, relname);
+
+ pq_sendint64(out, last_value);
+ pq_sendint64(out, log_cnt);
+ pq_sendint8(out, is_called);
+}
+
+/*
+ * Read SEQUENCE from the stream.
+ */
+void
+logicalrep_read_sequence(StringInfo in, LogicalRepSequence *seqdata)
+{
+ /* XXX skipping flags and lsn */
+ pq_getmsgint(in, 1);
+ pq_getmsgint64(in);
+
+ /* Read relation name from stream */
+ seqdata->nspname = pstrdup(logicalrep_read_namespace(in));
+ seqdata->seqname = pstrdup(pq_getmsgstring(in));
+
+ seqdata->last_value = pq_getmsgint64(in);
+ seqdata->log_cnt = pq_getmsgint64(in);
+ seqdata->is_called = pq_getmsgint(in, 1);
+}
+
/*
* Write relation description to the output stream.
*/
@@ -1236,6 +1284,8 @@ logicalrep_message_type(LogicalRepMsgType action)
return "STREAM ABORT";
case LOGICAL_REP_MSG_STREAM_PREPARE:
return "STREAM PREPARE";
+ case LOGICAL_REP_MSG_SEQUENCE:
+ return "SEQUENCE";
}
elog(ERROR, "invalid logical replication message type \"%c\"", action);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 5adc016d449..0cc7c8d3bc4 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -77,6 +77,8 @@
* a bit more memory to the oldest subtransactions, because it's likely
* they are the source for the next sequence of changes.
*
+ * FIXME
+ *
* -------------------------------------------------------------------------
*/
#include "postgres.h"
@@ -91,6 +93,7 @@
#include "access/xact.h"
#include "access/xlog_internal.h"
#include "catalog/catalog.h"
+#include "commands/sequence.h"
#include "lib/binaryheap.h"
#include "miscadmin.h"
#include "pgstat.h"
@@ -536,6 +539,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_SEQUENCE:
break;
}
@@ -866,6 +870,45 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
}
}
+/*
+ * A transactional sequence increment is queued to be processed upon commit
+ * and a non-transactional increment gets processed immediately.
+ *
+ * A sequence update may be both transactional and non-transactional. When
+ * created in a running transaction, treat it as transactional and queue
+ * the change in it. Otherwise treat it as non-transactional, so that we
+ * don't forget the increment in case of a rollback.
+ */
+void
+ReorderBufferQueueSequence(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, RepOriginId origin_id,
+ Oid reloid, RelFileNode rnode,
+ int64 last, int64 log_cnt, bool is_called)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* OK, allocate and queue the change */
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+
+ change->action = REORDER_BUFFER_CHANGE_SEQUENCE;
+ change->origin_id = origin_id;
+
+ memcpy(&change->data.sequence.relnode, &rnode, sizeof(RelFileNode));
+
+ change->data.sequence.reloid = reloid;
+ change->data.sequence.last = last;
+ change->data.sequence.log_cnt = log_cnt;
+ change->data.sequence.is_called = is_called;
+
+ /* add it to the same subxact that created the sequence */
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
/*
* AssertTXNLsnOrder
* Verify LSN ordering of transaction lists in the reorderbuffer
@@ -1957,6 +2000,33 @@ ReorderBufferApplyMessage(ReorderBuffer *rb, ReorderBufferTXN *txn,
change->data.msg.message);
}
+/*
+ * Helper function for ReorderBufferProcessTXN for applying sequences.
+ */
+static inline void
+ReorderBufferApplySequence(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change,
+ bool streaming)
+{
+ int64 last_value, log_cnt;
+ bool is_called;
+/*
+ * FIXME is it possible that we write sequence state for T1 before T2, but
+ * then end up committing T2 first? If the sequence could be incremented
+ * in between, that might cause data corruption result.
+ */
+ last_value = change->data.sequence.last;
+ log_cnt = change->data.sequence.log_cnt;
+ is_called = change->data.sequence.is_called;
+
+ if (streaming)
+ rb->stream_sequence(rb, txn, change->lsn, relation,
+ last_value, log_cnt, is_called);
+ else
+ rb->sequence(rb, txn, change->lsn, relation,
+ last_value, log_cnt, is_called);
+}
+
/*
* Function to store the command id and snapshot at the end of the current
* stream so that we can reuse the same while sending the next stream.
@@ -2399,6 +2469,33 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
+
+ case REORDER_BUFFER_CHANGE_SEQUENCE:
+ Assert(snapshot_now);
+
+ /*
+ reloid = RelidByRelfilenode(change->data.sequence.relnode.spcNode,
+ change->data.sequence.relnode.relNode);
+
+ if (reloid == InvalidOid)
+ elog(ERROR, "zz could not map filenode \"%s\" to relation OID",
+ relpathperm(change->data.sequence.relnode,
+ MAIN_FORKNUM));
+
+ */
+ relation = RelationIdGetRelation(change->data.sequence.reloid);
+
+ if (!RelationIsValid(relation))
+ elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
+ reloid,
+ relpathperm(change->data.sequence.relnode,
+ MAIN_FORKNUM));
+
+ if (RelationIsLogicallyLogged(relation))
+ ReorderBufferApplySequence(rb, txn, relation, change, streaming);
+
+ RelationClose(relation);
+ break;
}
}
@@ -3789,6 +3886,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_SEQUENCE:
/* ReorderBufferChange contains everything important */
break;
}
@@ -4053,6 +4151,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_SEQUENCE:
/* ReorderBufferChange contains everything important */
break;
}
@@ -4352,6 +4451,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_SEQUENCE:
break;
}
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 49ceec3bdc8..f767820cb6c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -100,6 +100,7 @@
#include "catalog/pg_subscription_rel.h"
#include "catalog/pg_type.h"
#include "commands/copy.h"
+#include "commands/sequence.h"
#include "miscadmin.h"
#include "parser/parse_relation.h"
#include "pgstat.h"
@@ -1136,6 +1137,94 @@ copy_table(Relation rel)
logicalrep_rel_close(relmapentry, NoLock);
}
+/*
+ * Fetch sequence data (current state) from the remote node.
+ */
+static void
+fetch_sequence_data(char *nspname, char *relname,
+ int64 *last_value, int64 *log_cnt, bool *is_called)
+{
+ WalRcvExecResult *res;
+ StringInfoData cmd;
+ TupleTableSlot *slot;
+ Oid tableRow[3] = {INT8OID, INT8OID, BOOLOID};
+
+ initStringInfo(&cmd);
+ appendStringInfo(&cmd, "SELECT last_value, log_cnt, is_called\n"
+ " FROM %s", quote_qualified_identifier(nspname, relname));
+
+ res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 3, tableRow);
+ pfree(cmd.data);
+
+ if (res->status != WALRCV_OK_TUPLES)
+ ereport(ERROR,
+ (errmsg("could not receive list of replicated tables from the publisher: %s",
+ res->err)));
+
+ /* Process the sequence. */
+ slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+ while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+ {
+ bool isnull;
+
+ *last_value = DatumGetInt64(slot_getattr(slot, 1, &isnull));
+ Assert(!isnull);
+
+ *log_cnt = DatumGetInt64(slot_getattr(slot, 2, &isnull));
+ Assert(!isnull);
+
+ *is_called = DatumGetBool(slot_getattr(slot, 3, &isnull));
+ Assert(!isnull);
+
+ ExecClearTuple(slot);
+ }
+ ExecDropSingleTupleTableSlot(slot);
+
+ walrcv_clear_result(res);
+}
+
+/*
+ * Copy existing data of a sequence from publisher.
+ *
+ * Caller is responsible for locking the local relation.
+ */
+static void
+copy_sequence(Relation rel)
+{
+ LogicalRepRelMapEntry *relmapentry;
+ LogicalRepRelation lrel;
+ List *qual = NIL;
+ StringInfoData cmd;
+ int64 last_value = 0,
+ log_cnt = 0;
+ bool is_called = 0;
+
+ /* Get the publisher relation info. */
+ fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
+ RelationGetRelationName(rel), &lrel, &qual);
+
+ /* sequences don't have row filters */
+ Assert(!qual);
+
+ /* Put the relation into relmap. */
+ logicalrep_relmap_update(&lrel);
+
+ /* Map the publisher relation to local one. */
+ relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
+ Assert(rel == relmapentry->localrel);
+
+ /* Start copy on the publisher. */
+ initStringInfo(&cmd);
+
+ Assert(lrel.relkind == RELKIND_SEQUENCE);
+
+ fetch_sequence_data(lrel.nspname, lrel.relname, &last_value, &log_cnt, &is_called);
+
+ SetSequence(RelationGetRelid(rel), last_value, log_cnt, is_called);
+
+ logicalrep_rel_close(relmapentry, NoLock);
+}
+
/*
* Determine the tablesync slot name.
*
@@ -1397,10 +1486,21 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos)
originname)));
}
- /* Now do the initial data copy */
- PushActiveSnapshot(GetTransactionSnapshot());
- copy_table(rel);
- PopActiveSnapshot();
+ /* Do the right action depending on the relation kind. */
+ if (get_rel_relkind(RelationGetRelid(rel)) == RELKIND_SEQUENCE)
+ {
+ /* Now do the initial sequence copy */
+ PushActiveSnapshot(GetTransactionSnapshot());
+ copy_sequence(rel);
+ PopActiveSnapshot();
+ }
+ else
+ {
+ /* Now do the initial data copy */
+ PushActiveSnapshot(GetTransactionSnapshot());
+ copy_table(rel);
+ PopActiveSnapshot();
+ }
res = walrcv_exec(LogRepWorkerWalRcvConn, "COMMIT", 0, NULL);
if (res->status != WALRCV_OK_COMMAND)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 9181d3e8636..71fe9390f0d 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -143,6 +143,7 @@
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
#include "catalog/pg_tablespace.h"
+#include "commands/sequence.h"
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
#include "commands/trigger.h"
@@ -1143,6 +1144,47 @@ apply_handle_origin(StringInfo s)
errmsg_internal("ORIGIN message sent out of order")));
}
+/*
+ * Handle SEQUENCE message.
+ */
+static void
+apply_handle_sequence(StringInfo s)
+{
+ LogicalRepSequence seq;
+ Oid relid;
+
+ if (handle_streamed_transaction(LOGICAL_REP_MSG_SEQUENCE, s))
+ return;
+
+ logicalrep_read_sequence(s, &seq);
+
+ /*
+ * We should be in remote transaction.
+ */
+ Assert(in_remote_transaction);
+
+ /*
+ * Make sure we're in a transaction (needed by SetSequence). For
+ * non-transactional updates we're guaranteed to start a new one,
+ * and we'll commit it at the end.
+ */
+ if (!IsTransactionState())
+ {
+ StartTransactionCommand();
+ maybe_reread_subscription();
+ }
+
+ relid = RangeVarGetRelid(makeRangeVar(seq.nspname,
+ seq.seqname, -1),
+ RowExclusiveLock, false);
+
+ /* lock the sequence in AccessExclusiveLock, as expected by SetSequence */
+ LockRelationOid(relid, AccessExclusiveLock);
+
+ /* apply the sequence change */
+ SetSequence(relid, seq.last_value, seq.log_cnt, seq.is_called);
+}
+
/*
* Handle STREAM START message.
*/
@@ -2511,6 +2553,10 @@ apply_dispatch(StringInfo s)
*/
break;
+ case LOGICAL_REP_MSG_SEQUENCE:
+ apply_handle_sequence(s);
+ return;
+
case LOGICAL_REP_MSG_STREAM_START:
apply_handle_stream_start(s);
break;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index fe5accca576..31be8064ebe 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,6 +15,7 @@
#include "access/tupconvert.h"
#include "catalog/partition.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_namespace.h"
#include "catalog/pg_publication_rel.h"
#include "commands/defrem.h"
#include "executor/executor.h"
@@ -54,6 +55,10 @@ static void pgoutput_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pgoutput_sequence(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr sequence_lsn,
+ Relation relation,
+ int64 last_value, int64 log_cnt, bool is_called);
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx,
@@ -255,6 +260,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->change_cb = pgoutput_change;
cb->truncate_cb = pgoutput_truncate;
cb->message_cb = pgoutput_message;
+ cb->sequence_cb = pgoutput_sequence;
cb->commit_cb = pgoutput_commit_txn;
cb->begin_prepare_cb = pgoutput_begin_prepare_txn;
@@ -271,6 +277,7 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->stream_commit_cb = pgoutput_stream_commit;
cb->stream_change_cb = pgoutput_change;
cb->stream_message_cb = pgoutput_message;
+ cb->stream_sequence_cb = pgoutput_sequence;
cb->stream_truncate_cb = pgoutput_truncate;
/* transaction streaming - two-phase commit */
cb->stream_prepare_cb = pgoutput_stream_prepare_txn;
@@ -284,6 +291,7 @@ parse_output_parameters(List *options, PGOutputData *data)
bool publication_names_given = false;
bool binary_option_given = false;
bool messages_option_given = false;
+ bool sequences_option_given = false;
bool streaming_given = false;
bool two_phase_option_given = false;
@@ -291,6 +299,7 @@ parse_output_parameters(List *options, PGOutputData *data)
data->streaming = false;
data->messages = false;
data->two_phase = false;
+ data->sequences = true;
foreach(lc, options)
{
@@ -359,6 +368,16 @@ parse_output_parameters(List *options, PGOutputData *data)
data->messages = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "sequences") == 0)
+ {
+ if (sequences_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sequences_option_given = true;
+
+ data->sequences = defGetBoolean(defel);
+ }
else if (strcmp(defel->defname, "streaming") == 0)
{
if (streaming_given)
@@ -1690,6 +1709,55 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginWrite(ctx, true);
}
+static void
+pgoutput_sequence(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr sequence_lsn,
+ Relation relation,
+ int64 last_value, int64 log_cnt, bool is_called)
+{
+ PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
+ TransactionId xid = InvalidTransactionId;
+ RelationSyncEntry *relentry;
+ PGOutputTxnData *txndata = (PGOutputTxnData *) txn->output_plugin_private;
+
+ if (!data->sequences)
+ return;
+
+ if (!is_publishable_relation(relation))
+ return;
+
+ /*
+ * Remember the xid for the message in streaming mode. See
+ * pgoutput_change.
+ */
+ if (in_streaming)
+ xid = txn->xid;
+
+ relentry = get_rel_sync_entry(data, relation);
+
+ /*
+ * First check the sequence filter.
+ *
+ * We handle just REORDER_BUFFER_CHANGE_SEQUENCE here.
+ */
+ if (!relentry->pubactions.pubsequence)
+ return;
+
+ /* Send BEGIN if we haven't yet */
+ if (txndata && !txndata->sent_begin_txn)
+ pgoutput_send_begin(ctx, txn);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_sequence(ctx->out,
+ relation,
+ xid,
+ sequence_lsn,
+ last_value,
+ log_cnt,
+ is_called);
+ OutputPluginWrite(ctx, true);
+}
+
/*
* Currently we always forward.
*/
@@ -1975,7 +2043,8 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->schema_sent = false;
entry->streamed_txns = NIL;
entry->pubactions.pubinsert = entry->pubactions.pubupdate =
- entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
+ entry->pubactions.pubdelete = entry->pubactions.pubtruncate =
+ entry->pubactions.pubsequence = false;
entry->new_slot = NULL;
entry->old_slot = NULL;
memset(entry->exprstate, 0, sizeof(entry->exprstate));
@@ -1990,18 +2059,18 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
{
Oid schemaId = get_rel_namespace(relid);
List *pubids = GetRelationPublications(relid);
-
+ char relkind = get_rel_relkind(relid);
+ char objectType = pub_get_object_type_for_relkind(relkind);
/*
* We don't acquire a lock on the namespace system table as we build
* the cache entry using a historic snapshot and all the later changes
* are absorbed while decoding WAL.
*/
- List *schemaPubids = GetSchemaPublications(schemaId);
+ List *schemaPubids = GetSchemaPublications(schemaId, objectType);
ListCell *lc;
Oid publish_as_relid = relid;
int publish_ancestor_level = 0;
bool am_partition = get_rel_relispartition(relid);
- char relkind = get_rel_relkind(relid);
List *rel_publications = NIL;
/* Reload publications if needed before use. */
@@ -2033,6 +2102,7 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->pubactions.pubupdate = false;
entry->pubactions.pubdelete = false;
entry->pubactions.pubtruncate = false;
+ entry->pubactions.pubsequence = false;
/*
* Tuple slots cleanups. (Will be rebuilt later if needed).
@@ -2080,9 +2150,11 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
/*
* If this is a FOR ALL TABLES publication, pick the partition root
- * and set the ancestor level accordingly.
+ * and set the ancestor level accordingly. If this is a FOR ALL
+ * SEQUENCES publication, we publish it too but we don't need to
+ * pick the partition root etc.
*/
- if (pub->alltables)
+ if (pub->alltables || pub->allsequences)
{
publish = true;
if (pub->pubviaroot && am_partition)
@@ -2146,6 +2218,7 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
entry->pubactions.pubupdate |= pub->pubactions.pubupdate;
entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
+ entry->pubactions.pubsequence |= pub->pubactions.pubsequence;
/*
* We want to publish the changes as the top-most ancestor
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 43f14c233d6..1f29670a131 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -56,6 +56,7 @@
#include "catalog/pg_opclass.h"
#include "catalog/pg_proc.h"
#include "catalog/pg_publication.h"
+#include "catalog/pg_publication_namespace.h"
#include "catalog/pg_rewrite.h"
#include "catalog/pg_shseclabel.h"
#include "catalog/pg_statistic_ext.h"
@@ -5567,6 +5568,8 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
Oid schemaid;
List *ancestors = NIL;
Oid relid = RelationGetRelid(relation);
+ char relkind = relation->rd_rel->relkind;
+ char objType;
/*
* If not publishable, it publishes no actions. (pgoutput_change() will
@@ -5597,8 +5600,15 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
/* Fetch the publication membership info. */
puboids = GetRelationPublications(relid);
schemaid = RelationGetNamespace(relation);
- puboids = list_concat_unique_oid(puboids, GetSchemaPublications(schemaid));
+ objType = pub_get_object_type_for_relkind(relkind);
+ puboids = list_concat_unique_oid(puboids,
+ GetSchemaPublications(schemaid, objType));
+
+ /*
+ * If this is a partion (and thus a table), lookup all ancestors and track
+ * all publications them too.
+ */
if (relation->rd_rel->relispartition)
{
/* Add publications that the ancestors are in too. */
@@ -5610,12 +5620,23 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
puboids = list_concat_unique_oid(puboids,
GetRelationPublications(ancestor));
+
+ /* include all publications publishing schema of all ancestors */
schemaid = get_rel_namespace(ancestor);
puboids = list_concat_unique_oid(puboids,
- GetSchemaPublications(schemaid));
+ GetSchemaPublications(schemaid,
+ PUB_OBJTYPE_TABLE));
}
}
- puboids = list_concat_unique_oid(puboids, GetAllTablesPublications());
+
+ /*
+ * Consider also FOR ALL TABLES and FOR ALL SEQUENCES publications,
+ * depending on the relkind of the relation.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ puboids = list_concat_unique_oid(puboids, GetAllSequencesPublications());
+ else
+ puboids = list_concat_unique_oid(puboids, GetAllTablesPublications());
foreach(lc, puboids)
{
@@ -5634,6 +5655,7 @@ RelationBuildPublicationDesc(Relation relation, PublicationDesc *pubdesc)
pubdesc->pubactions.pubupdate |= pubform->pubupdate;
pubdesc->pubactions.pubdelete |= pubform->pubdelete;
pubdesc->pubactions.pubtruncate |= pubform->pubtruncate;
+ pubdesc->pubactions.pubsequence |= pubform->pubsequence;
/*
* Check if all columns referenced in the filter expression are part of
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index 1912b121463..8d265f2d23c 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -653,12 +653,12 @@ static const struct cachedesc cacheinfo[] = {
64
},
{PublicationNamespaceRelationId, /* PUBLICATIONNAMESPACEMAP */
- PublicationNamespacePnnspidPnpubidIndexId,
- 2,
+ PublicationNamespacePnnspidPnpubidPntypeIndexId,
+ 3,
{
Anum_pg_publication_namespace_pnnspid,
Anum_pg_publication_namespace_pnpubid,
- 0,
+ Anum_pg_publication_namespace_pntype,
0
},
64
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 196f6d23a3e..3c2201a725f 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -3814,10 +3814,12 @@ getPublications(Archive *fout, int *numPublications)
int i_pubname;
int i_pubowner;
int i_puballtables;
+ int i_puballsequences;
int i_pubinsert;
int i_pubupdate;
int i_pubdelete;
int i_pubtruncate;
+ int i_pubsequence;
int i_pubviaroot;
int i,
ntups;
@@ -3833,23 +3835,29 @@ getPublications(Archive *fout, int *numPublications)
resetPQExpBuffer(query);
/* Get the publications. */
- if (fout->remoteVersion >= 130000)
+ if (fout->remoteVersion >= 150000)
+ appendPQExpBuffer(query,
+ "SELECT p.tableoid, p.oid, p.pubname, "
+ "p.pubowner, "
+ "p.puballtables, p.puballsequences, p.pubinsert, p.pubupdate, p.pubdelete, p.pubtruncate, p.pubsequence, p.pubviaroot "
+ "FROM pg_publication p");
+ else if (fout->remoteVersion >= 130000)
appendPQExpBuffer(query,
"SELECT p.tableoid, p.oid, p.pubname, "
"p.pubowner, "
- "p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete, p.pubtruncate, p.pubviaroot "
+ "p.puballtables, false AS puballsequences, p.pubinsert, p.pubupdate, p.pubdelete, p.pubtruncate, false AS pubsequence, p.pubviaroot "
"FROM pg_publication p");
else if (fout->remoteVersion >= 110000)
appendPQExpBuffer(query,
"SELECT p.tableoid, p.oid, p.pubname, "
"p.pubowner, "
- "p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete, p.pubtruncate, false AS pubviaroot "
+ "p.puballtables, false AS puballsequences, p.pubinsert, p.pubupdate, p.pubdelete, p.pubtruncate, false AS pubsequence, false AS pubviaroot "
"FROM pg_publication p");
else
appendPQExpBuffer(query,
"SELECT p.tableoid, p.oid, p.pubname, "
"p.pubowner, "
- "p.puballtables, p.pubinsert, p.pubupdate, p.pubdelete, false AS pubtruncate, false AS pubviaroot "
+ "p.puballtables, false AS puballsequences, p.pubinsert, p.pubupdate, p.pubdelete, false AS pubtruncate, false AS pubsequence, false AS pubviaroot "
"FROM pg_publication p");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
@@ -3861,10 +3869,12 @@ getPublications(Archive *fout, int *numPublications)
i_pubname = PQfnumber(res, "pubname");
i_pubowner = PQfnumber(res, "pubowner");
i_puballtables = PQfnumber(res, "puballtables");
+ i_puballsequences = PQfnumber(res, "puballsequences");
i_pubinsert = PQfnumber(res, "pubinsert");
i_pubupdate = PQfnumber(res, "pubupdate");
i_pubdelete = PQfnumber(res, "pubdelete");
i_pubtruncate = PQfnumber(res, "pubtruncate");
+ i_pubsequence = PQfnumber(res, "pubsequence");
i_pubviaroot = PQfnumber(res, "pubviaroot");
pubinfo = pg_malloc(ntups * sizeof(PublicationInfo));
@@ -3880,6 +3890,8 @@ getPublications(Archive *fout, int *numPublications)
pubinfo[i].rolname = getRoleName(PQgetvalue(res, i, i_pubowner));
pubinfo[i].puballtables =
(strcmp(PQgetvalue(res, i, i_puballtables), "t") == 0);
+ pubinfo[i].puballsequences =
+ (strcmp(PQgetvalue(res, i, i_puballsequences), "t") == 0);
pubinfo[i].pubinsert =
(strcmp(PQgetvalue(res, i, i_pubinsert), "t") == 0);
pubinfo[i].pubupdate =
@@ -3888,6 +3900,8 @@ getPublications(Archive *fout, int *numPublications)
(strcmp(PQgetvalue(res, i, i_pubdelete), "t") == 0);
pubinfo[i].pubtruncate =
(strcmp(PQgetvalue(res, i, i_pubtruncate), "t") == 0);
+ pubinfo[i].pubsequence =
+ (strcmp(PQgetvalue(res, i, i_pubsequence), "t") == 0);
pubinfo[i].pubviaroot =
(strcmp(PQgetvalue(res, i, i_pubviaroot), "t") == 0);
@@ -3933,6 +3947,9 @@ dumpPublication(Archive *fout, const PublicationInfo *pubinfo)
if (pubinfo->puballtables)
appendPQExpBufferStr(query, " FOR ALL TABLES");
+ if (pubinfo->puballsequences)
+ appendPQExpBufferStr(query, " FOR ALL SEQUENCES");
+
appendPQExpBufferStr(query, " WITH (publish = '");
if (pubinfo->pubinsert)
{
@@ -3967,6 +3984,15 @@ dumpPublication(Archive *fout, const PublicationInfo *pubinfo)
first = false;
}
+ if (pubinfo->pubsequence)
+ {
+ if (!first)
+ appendPQExpBufferStr(query, ", ");
+
+ appendPQExpBufferStr(query, "sequence");
+ first = false;
+ }
+
appendPQExpBufferStr(query, "'");
if (pubinfo->pubviaroot)
@@ -4013,6 +4039,7 @@ getPublicationNamespaces(Archive *fout)
int i_oid;
int i_pnpubid;
int i_pnnspid;
+ int i_pntype;
int i,
j,
ntups;
@@ -4024,7 +4051,7 @@ getPublicationNamespaces(Archive *fout)
/* Collect all publication membership info. */
appendPQExpBufferStr(query,
- "SELECT tableoid, oid, pnpubid, pnnspid "
+ "SELECT tableoid, oid, pnpubid, pnnspid, pntype "
"FROM pg_catalog.pg_publication_namespace");
res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
@@ -4034,6 +4061,7 @@ getPublicationNamespaces(Archive *fout)
i_oid = PQfnumber(res, "oid");
i_pnpubid = PQfnumber(res, "pnpubid");
i_pnnspid = PQfnumber(res, "pnnspid");
+ i_pntype = PQfnumber(res, "pntype");
/* this allocation may be more than we need */
pubsinfo = pg_malloc(ntups * sizeof(PublicationSchemaInfo));
@@ -4043,6 +4071,7 @@ getPublicationNamespaces(Archive *fout)
{
Oid pnpubid = atooid(PQgetvalue(res, i, i_pnpubid));
Oid pnnspid = atooid(PQgetvalue(res, i, i_pnnspid));
+ char pntype = PQgetvalue(res, i, i_pntype)[0];
PublicationInfo *pubinfo;
NamespaceInfo *nspinfo;
@@ -4074,6 +4103,7 @@ getPublicationNamespaces(Archive *fout)
pubsinfo[j].dobj.name = nspinfo->dobj.name;
pubsinfo[j].publication = pubinfo;
pubsinfo[j].pubschema = nspinfo;
+ pubsinfo[j].pubtype = pntype;
/* Decide whether we want to dump it */
selectDumpablePublicationObject(&(pubsinfo[j].dobj), fout);
@@ -4239,7 +4269,11 @@ dumpPublicationNamespace(Archive *fout, const PublicationSchemaInfo *pubsinfo)
query = createPQExpBuffer();
appendPQExpBuffer(query, "ALTER PUBLICATION %s ", fmtId(pubinfo->dobj.name));
- appendPQExpBuffer(query, "ADD ALL TABLES IN SCHEMA %s;\n", fmtId(schemainfo->dobj.name));
+
+ if (pubsinfo->pubtype == 't')
+ appendPQExpBuffer(query, "ADD ALL TABLES IN SCHEMA %s;\n", fmtId(schemainfo->dobj.name));
+ else
+ appendPQExpBuffer(query, "ADD ALL SEQUENCES IN SCHEMA %s;\n", fmtId(schemainfo->dobj.name));
/*
* There is no point in creating drop query as the drop is done by schema
@@ -4272,6 +4306,7 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
TableInfo *tbinfo = pubrinfo->pubtable;
PQExpBuffer query;
char *tag;
+ char *description;
/* Do nothing in data-only dump */
if (dopt->dataOnly)
@@ -4281,8 +4316,19 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
query = createPQExpBuffer();
- appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
- fmtId(pubinfo->dobj.name));
+ if (tbinfo->relkind == RELKIND_SEQUENCE)
+ {
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD SEQUENCE",
+ fmtId(pubinfo->dobj.name));
+ description = "PUBLICATION SEQUENCE";
+ }
+ else
+ {
+ appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
+ fmtId(pubinfo->dobj.name));
+ description = "PUBLICATION TABLE";
+ }
+
appendPQExpBuffer(query, " %s",
fmtQualifiedDumpable(tbinfo));
@@ -4311,7 +4357,7 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
ARCHIVE_OPTS(.tag = tag,
.namespace = tbinfo->dobj.namespace->dobj.name,
.owner = pubinfo->rolname,
- .description = "PUBLICATION TABLE",
+ .description = description,
.section = SECTION_POST_DATA,
.createStmt = query->data));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1d21c2906f1..688093c55e0 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -615,10 +615,12 @@ typedef struct _PublicationInfo
DumpableObject dobj;
const char *rolname;
bool puballtables;
+ bool puballsequences;
bool pubinsert;
bool pubupdate;
bool pubdelete;
bool pubtruncate;
+ bool pubsequence;
bool pubviaroot;
} PublicationInfo;
@@ -644,6 +646,7 @@ typedef struct _PublicationSchemaInfo
DumpableObject dobj;
PublicationInfo *publication;
NamespaceInfo *pubschema;
+ char pubtype;
} PublicationSchemaInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index c65c92bfb0e..75b754a4202 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2420,7 +2420,7 @@ my %tests = (
create_order => 50,
create_sql => 'CREATE PUBLICATION pub1;',
regexp => qr/^
- \QCREATE PUBLICATION pub1 WITH (publish = 'insert, update, delete, truncate');\E
+ \QCREATE PUBLICATION pub1 WITH (publish = 'insert, update, delete, truncate, sequence');\E
/xm,
like => { %full_runs, section_post_data => 1, },
},
@@ -2440,16 +2440,27 @@ my %tests = (
create_order => 50,
create_sql => 'CREATE PUBLICATION pub3;',
regexp => qr/^
- \QCREATE PUBLICATION pub3 WITH (publish = 'insert, update, delete, truncate');\E
+ \QCREATE PUBLICATION pub3 WITH (publish = 'insert, update, delete, truncate, sequence');\E
/xm,
like => { %full_runs, section_post_data => 1, },
},
'CREATE PUBLICATION pub4' => {
create_order => 50,
- create_sql => 'CREATE PUBLICATION pub4;',
+ create_sql => 'CREATE PUBLICATION pub4
+ FOR ALL SEQUENCES
+ WITH (publish = \'\');',
regexp => qr/^
- \QCREATE PUBLICATION pub4 WITH (publish = 'insert, update, delete, truncate');\E
+ \QCREATE PUBLICATION pub4 FOR ALL SEQUENCES WITH (publish = '');\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ },
+
+ 'CREATE PUBLICATION pub5' => {
+ create_order => 50,
+ create_sql => 'CREATE PUBLICATION pub5;',
+ regexp => qr/^
+ \QCREATE PUBLICATION pub5 WITH (publish = 'insert, update, delete, truncate, sequence');\E
/xm,
like => { %full_runs, section_post_data => 1, },
},
@@ -2558,6 +2569,27 @@ my %tests = (
unlike => { exclude_dump_test_schema => 1, },
},
+ 'ALTER PUBLICATION pub3 ADD ALL SEQUENCES IN SCHEMA dump_test' => {
+ create_order => 51,
+ create_sql =>
+ 'ALTER PUBLICATION pub3 ADD ALL SEQUENCES IN SCHEMA dump_test;',
+ regexp => qr/^
+ \QALTER PUBLICATION pub3 ADD ALL SEQUENCES IN SCHEMA dump_test;\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ unlike => { exclude_dump_test_schema => 1, },
+ },
+
+ 'ALTER PUBLICATION pub3 ADD ALL SEQUENCES IN SCHEMA public' => {
+ create_order => 52,
+ create_sql =>
+ 'ALTER PUBLICATION pub3 ADD ALL SEQUENCES IN SCHEMA public;',
+ regexp => qr/^
+ \QALTER PUBLICATION pub3 ADD ALL SEQUENCES IN SCHEMA public;\E
+ /xm,
+ like => { %full_runs, section_post_data => 1, },
+ },
+
'CREATE SCHEMA public' => {
regexp => qr/^CREATE SCHEMA public;/m,
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 3f1b3802c22..73bbbe2eb40 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -1633,28 +1633,19 @@ describeOneTableDetails(const char *schemaname,
if (tableinfo.relkind == RELKIND_SEQUENCE)
{
PGresult *result = NULL;
- printQueryOpt myopt = pset.popt;
- char *footers[2] = {NULL, NULL};
if (pset.sversion >= 100000)
{
printfPQExpBuffer(&buf,
- "SELECT pg_catalog.format_type(seqtypid, NULL) AS \"%s\",\n"
- " seqstart AS \"%s\",\n"
- " seqmin AS \"%s\",\n"
- " seqmax AS \"%s\",\n"
- " seqincrement AS \"%s\",\n"
- " CASE WHEN seqcycle THEN '%s' ELSE '%s' END AS \"%s\",\n"
- " seqcache AS \"%s\"\n",
- gettext_noop("Type"),
- gettext_noop("Start"),
- gettext_noop("Minimum"),
- gettext_noop("Maximum"),
- gettext_noop("Increment"),
+ "SELECT pg_catalog.format_type(seqtypid, NULL),\n"
+ " seqstart,\n"
+ " seqmin,\n"
+ " seqmax,\n"
+ " seqincrement,\n"
+ " CASE WHEN seqcycle THEN '%s' ELSE '%s' END,\n"
+ " seqcache\n",
gettext_noop("yes"),
- gettext_noop("no"),
- gettext_noop("Cycles?"),
- gettext_noop("Cache"));
+ gettext_noop("no"));
appendPQExpBuffer(&buf,
"FROM pg_catalog.pg_sequence\n"
"WHERE seqrelid = '%s';",
@@ -1663,22 +1654,15 @@ describeOneTableDetails(const char *schemaname,
else
{
printfPQExpBuffer(&buf,
- "SELECT 'bigint' AS \"%s\",\n"
- " start_value AS \"%s\",\n"
- " min_value AS \"%s\",\n"
- " max_value AS \"%s\",\n"
- " increment_by AS \"%s\",\n"
- " CASE WHEN is_cycled THEN '%s' ELSE '%s' END AS \"%s\",\n"
- " cache_value AS \"%s\"\n",
- gettext_noop("Type"),
- gettext_noop("Start"),
- gettext_noop("Minimum"),
- gettext_noop("Maximum"),
- gettext_noop("Increment"),
+ "SELECT 'bigint',\n"
+ " start_value,\n"
+ " min_value,\n"
+ " max_value,\n"
+ " increment_by,\n"
+ " CASE WHEN is_cycled THEN '%s' ELSE '%s' END,\n"
+ " cache_value\n",
gettext_noop("yes"),
- gettext_noop("no"),
- gettext_noop("Cycles?"),
- gettext_noop("Cache"));
+ gettext_noop("no"));
appendPQExpBuffer(&buf, "FROM %s", fmtId(schemaname));
/* must be separate because fmtId isn't reentrant */
appendPQExpBuffer(&buf, ".%s;", fmtId(relationname));
@@ -1688,6 +1672,57 @@ describeOneTableDetails(const char *schemaname,
if (!res)
goto error_return;
+ numrows = PQntuples(res);
+
+ /* XXX reset to use expanded output for sequences (maybe we should
+ * keep this disabled, just like for tables?) */
+ myopt.expanded = pset.popt.topt.expanded;
+
+ printTableInit(&cont, &myopt, title.data, 7, numrows);
+ printTableInitialized = true;
+
+ if (tableinfo.relpersistence == 'u')
+ printfPQExpBuffer(&title, _("Unlogged sequence \"%s.%s\""),
+ schemaname, relationname);
+ else
+ printfPQExpBuffer(&title, _("Sequence \"%s.%s\""),
+ schemaname, relationname);
+
+ printTableAddHeader(&cont, gettext_noop("Type"), true, 'l');
+ printTableAddHeader(&cont, gettext_noop("Start"), true, 'r');
+ printTableAddHeader(&cont, gettext_noop("Minimum"), true, 'r');
+ printTableAddHeader(&cont, gettext_noop("Maximum"), true, 'r');
+ printTableAddHeader(&cont, gettext_noop("Increment"), true, 'r');
+ printTableAddHeader(&cont, gettext_noop("Cycles?"), true, 'l');
+ printTableAddHeader(&cont, gettext_noop("Cache"), true, 'r');
+
+ /* Generate table cells to be printed */
+ for (i = 0; i < numrows; i++)
+ {
+ /* Type */
+ printTableAddCell(&cont, PQgetvalue(res, i, 0), false, false);
+
+ /* Start */
+ printTableAddCell(&cont, PQgetvalue(res, i, 1), false, false);
+
+ /* Minimum */
+ printTableAddCell(&cont, PQgetvalue(res, i, 2), false, false);
+
+ /* Maximum */
+ printTableAddCell(&cont, PQgetvalue(res, i, 3), false, false);
+
+ /* Increment */
+ printTableAddCell(&cont, PQgetvalue(res, i, 4), false, false);
+
+ /* Cycles? */
+ printTableAddCell(&cont, PQgetvalue(res, i, 5), false, false);
+
+ /* Cache */
+ printTableAddCell(&cont, PQgetvalue(res, i, 6), false, false);
+ }
+
+ /* Footer information about a sequence */
+
/* Get the column that owns this sequence */
printfPQExpBuffer(&buf, "SELECT pg_catalog.quote_ident(nspname) || '.' ||"
"\n pg_catalog.quote_ident(relname) || '.' ||"
@@ -1719,33 +1754,63 @@ describeOneTableDetails(const char *schemaname,
switch (PQgetvalue(result, 0, 1)[0])
{
case 'a':
- footers[0] = psprintf(_("Owned by: %s"),
- PQgetvalue(result, 0, 0));
+ printTableAddFooter(&cont,
+ psprintf(_("Owned by: %s"),
+ PQgetvalue(result, 0, 0)));
break;
case 'i':
- footers[0] = psprintf(_("Sequence for identity column: %s"),
- PQgetvalue(result, 0, 0));
+ printTableAddFooter(&cont,
+ psprintf(_("Sequence for identity column: %s"),
+ PQgetvalue(result, 0, 0)));
break;
}
}
PQclear(result);
- if (tableinfo.relpersistence == 'u')
- printfPQExpBuffer(&title, _("Unlogged sequence \"%s.%s\""),
- schemaname, relationname);
- else
- printfPQExpBuffer(&title, _("Sequence \"%s.%s\""),
- schemaname, relationname);
+ /* print any publications */
+ if (pset.sversion >= 150000)
+ {
+ int tuples = 0;
- myopt.footers = footers;
- myopt.topt.default_footer = false;
- myopt.title = title.data;
- myopt.translate_header = true;
+ printfPQExpBuffer(&buf,
+ "SELECT pubname\n"
+ "FROM pg_catalog.pg_publication p\n"
+ " JOIN pg_catalog.pg_publication_namespace pn ON p.oid = pn.pnpubid\n"
+ " JOIN pg_catalog.pg_class pc ON pc.relnamespace = pn.pnnspid\n"
+ "WHERE pc.oid ='%s' and pn.pntype = 's' and pg_catalog.pg_relation_is_publishable('%s')\n"
+ "UNION\n"
+ "SELECT pubname\n"
+ "FROM pg_catalog.pg_publication p\n"
+ " JOIN pg_catalog.pg_publication_rel pr ON p.oid = pr.prpubid\n"
+ "WHERE pr.prrelid = '%s'\n"
+ "UNION\n"
+ "SELECT pubname\n"
+ "FROM pg_catalog.pg_publication p\n"
+ "WHERE p.puballsequences AND pg_catalog.pg_relation_is_publishable('%s')\n"
+ "ORDER BY 1;",
+ oid, oid, oid, oid);
- printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+ result = PSQLexec(buf.data);
+ if (!result)
+ goto error_return;
+ else
+ tuples = PQntuples(result);
+
+ if (tuples > 0)
+ printTableAddFooter(&cont, _("Publications:"));
+
+ /* Might be an empty set - that's ok */
+ for (i = 0; i < tuples; i++)
+ {
+ printfPQExpBuffer(&buf, " \"%s\"",
+ PQgetvalue(result, i, 0));
+
+ printTableAddFooter(&cont, buf.data);
+ }
+ PQclear(result);
+ }
- if (footers[0])
- free(footers[0]);
+ printTable(&cont, pset.queryFout, false, pset.logfile);
retval = true;
goto error_return; /* not an error, just return early */
@@ -1972,6 +2037,11 @@ describeOneTableDetails(const char *schemaname,
for (i = 0; i < cols; i++)
printTableAddHeader(&cont, headers[i], true, 'l');
+ res = PSQLexec(buf.data);
+ if (!res)
+ goto error_return;
+ numrows = PQntuples(res);
+
/* Generate table cells to be printed */
for (i = 0; i < numrows; i++)
{
@@ -2898,7 +2968,7 @@ describeOneTableDetails(const char *schemaname,
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_namespace pn ON p.oid = pn.pnpubid\n"
" JOIN pg_catalog.pg_class pc ON pc.relnamespace = pn.pnnspid\n"
- "WHERE pc.oid ='%s' and pg_catalog.pg_relation_is_publishable('%s')\n"
+ "WHERE pc.oid ='%s' and pn.pntype = 't' and pg_catalog.pg_relation_is_publishable('%s')\n"
"UNION\n"
"SELECT pubname\n"
" , pg_get_expr(pr.prqual, c.oid)\n"
@@ -4802,7 +4872,7 @@ listSchemas(const char *pattern, bool verbose, bool showSystem)
int i;
printfPQExpBuffer(&buf,
- "SELECT pubname \n"
+ "SELECT pubname, (CASE WHEN pntype = 't' THEN 'tables' ELSE 'sequences' END) AS pubtype\n"
"FROM pg_catalog.pg_publication p\n"
" JOIN pg_catalog.pg_publication_namespace pn ON p.oid = pn.pnpubid\n"
" JOIN pg_catalog.pg_namespace n ON n.oid = pn.pnnspid \n"
@@ -4831,8 +4901,9 @@ listSchemas(const char *pattern, bool verbose, bool showSystem)
/* Might be an empty set - that's ok */
for (i = 0; i < pub_schema_tuples; i++)
{
- printfPQExpBuffer(&buf, " \"%s\"",
- PQgetvalue(result, i, 0));
+ printfPQExpBuffer(&buf, " \"%s\" (%s)",
+ PQgetvalue(result, i, 0),
+ PQgetvalue(result, i, 1));
footers[i + 1] = pg_strdup(buf.data);
}
@@ -5837,7 +5908,7 @@ listPublications(const char *pattern)
PQExpBufferData buf;
PGresult *res;
printQueryOpt myopt = pset.popt;
- static const bool translate_columns[] = {false, false, false, false, false, false, false, false};
+ static const bool translate_columns[] = {false, false, false, false, false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -5851,23 +5922,45 @@ listPublications(const char *pattern)
initPQExpBuffer(&buf);
- printfPQExpBuffer(&buf,
- "SELECT pubname AS \"%s\",\n"
- " pg_catalog.pg_get_userbyid(pubowner) AS \"%s\",\n"
- " puballtables AS \"%s\",\n"
- " pubinsert AS \"%s\",\n"
- " pubupdate AS \"%s\",\n"
- " pubdelete AS \"%s\"",
- gettext_noop("Name"),
- gettext_noop("Owner"),
- gettext_noop("All tables"),
- gettext_noop("Inserts"),
- gettext_noop("Updates"),
- gettext_noop("Deletes"));
+ if (pset.sversion >= 150000)
+ printfPQExpBuffer(&buf,
+ "SELECT pubname AS \"%s\",\n"
+ " pg_catalog.pg_get_userbyid(pubowner) AS \"%s\",\n"
+ " puballtables AS \"%s\",\n"
+ " puballsequences AS \"%s\",\n"
+ " pubinsert AS \"%s\",\n"
+ " pubupdate AS \"%s\",\n"
+ " pubdelete AS \"%s\"",
+ gettext_noop("Name"),
+ gettext_noop("Owner"),
+ gettext_noop("All tables"),
+ gettext_noop("All sequences"),
+ gettext_noop("Inserts"),
+ gettext_noop("Updates"),
+ gettext_noop("Deletes"));
+ else
+ printfPQExpBuffer(&buf,
+ "SELECT pubname AS \"%s\",\n"
+ " pg_catalog.pg_get_userbyid(pubowner) AS \"%s\",\n"
+ " puballtables AS \"%s\",\n"
+ " pubinsert AS \"%s\",\n"
+ " pubupdate AS \"%s\",\n"
+ " pubdelete AS \"%s\"",
+ gettext_noop("Name"),
+ gettext_noop("Owner"),
+ gettext_noop("All tables"),
+ gettext_noop("Inserts"),
+ gettext_noop("Updates"),
+ gettext_noop("Deletes"));
+
if (pset.sversion >= 110000)
appendPQExpBuffer(&buf,
",\n pubtruncate AS \"%s\"",
gettext_noop("Truncates"));
+ if (pset.sversion >= 150000)
+ appendPQExpBuffer(&buf,
+ ",\n pubsequence AS \"%s\"",
+ gettext_noop("Sequences"));
if (pset.sversion >= 130000)
appendPQExpBuffer(&buf,
",\n pubviaroot AS \"%s\"",
@@ -5957,6 +6050,7 @@ describePublications(const char *pattern)
PGresult *res;
bool has_pubtruncate;
bool has_pubviaroot;
+ bool has_pubsequence;
PQExpBufferData title;
printTableContent cont;
@@ -5973,6 +6067,7 @@ describePublications(const char *pattern)
has_pubtruncate = (pset.sversion >= 110000);
has_pubviaroot = (pset.sversion >= 130000);
+ has_pubsequence = (pset.sversion >= 150000);
initPQExpBuffer(&buf);
@@ -5980,12 +6075,17 @@ describePublications(const char *pattern)
"SELECT oid, pubname,\n"
" pg_catalog.pg_get_userbyid(pubowner) AS owner,\n"
" puballtables, pubinsert, pubupdate, pubdelete");
+
if (has_pubtruncate)
appendPQExpBufferStr(&buf,
", pubtruncate");
if (has_pubviaroot)
appendPQExpBufferStr(&buf,
", pubviaroot");
+ if (has_pubsequence)
+ appendPQExpBufferStr(&buf,
+ ", puballsequences, pubsequence");
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_publication\n");
@@ -6026,6 +6126,7 @@ describePublications(const char *pattern)
char *pubid = PQgetvalue(res, i, 0);
char *pubname = PQgetvalue(res, i, 1);
bool puballtables = strcmp(PQgetvalue(res, i, 3), "t") == 0;
+ bool puballsequences = strcmp(PQgetvalue(res, i, 9), "t") == 0;
printTableOpt myopt = pset.popt.topt;
if (has_pubtruncate)
@@ -6033,29 +6134,43 @@ describePublications(const char *pattern)
if (has_pubviaroot)
ncols++;
+ /* sequences have two extra columns (puballsequences, pubsequences) */
+ if (has_pubsequence)
+ ncols += 2;
+
initPQExpBuffer(&title);
printfPQExpBuffer(&title, _("Publication %s"), pubname);
printTableInit(&cont, &myopt, title.data, ncols, nrows);
printTableAddHeader(&cont, gettext_noop("Owner"), true, align);
printTableAddHeader(&cont, gettext_noop("All tables"), true, align);
+ if (has_pubsequence)
+ printTableAddHeader(&cont, gettext_noop("All sequences"), true, align);
printTableAddHeader(&cont, gettext_noop("Inserts"), true, align);
printTableAddHeader(&cont, gettext_noop("Updates"), true, align);
printTableAddHeader(&cont, gettext_noop("Deletes"), true, align);
if (has_pubtruncate)
printTableAddHeader(&cont, gettext_noop("Truncates"), true, align);
+ if (has_pubsequence)
+ printTableAddHeader(&cont, gettext_noop("Sequences"), true, align);
if (has_pubviaroot)
printTableAddHeader(&cont, gettext_noop("Via root"), true, align);
- printTableAddCell(&cont, PQgetvalue(res, i, 2), false, false);
- printTableAddCell(&cont, PQgetvalue(res, i, 3), false, false);
- printTableAddCell(&cont, PQgetvalue(res, i, 4), false, false);
- printTableAddCell(&cont, PQgetvalue(res, i, 5), false, false);
- printTableAddCell(&cont, PQgetvalue(res, i, 6), false, false);
+ printTableAddCell(&cont, PQgetvalue(res, i, 2), false, false); /* owner */
+ printTableAddCell(&cont, PQgetvalue(res, i, 3), false, false); /* all tables */
+
+ if (has_pubsequence)
+ printTableAddCell(&cont, PQgetvalue(res, i, 9), false, false); /* all sequences */
+
+ printTableAddCell(&cont, PQgetvalue(res, i, 4), false, false); /* insert */
+ printTableAddCell(&cont, PQgetvalue(res, i, 5), false, false); /* update */
+ printTableAddCell(&cont, PQgetvalue(res, i, 6), false, false); /* delete */
if (has_pubtruncate)
- printTableAddCell(&cont, PQgetvalue(res, i, 7), false, false);
+ printTableAddCell(&cont, PQgetvalue(res, i, 7), false, false); /* truncate */
+ if (has_pubsequence)
+ printTableAddCell(&cont, PQgetvalue(res, i, 10), false, false); /* sequence */
if (has_pubviaroot)
- printTableAddCell(&cont, PQgetvalue(res, i, 8), false, false);
+ printTableAddCell(&cont, PQgetvalue(res, i, 8), false, false); /* via root */
if (!puballtables)
{
@@ -6086,6 +6201,7 @@ describePublications(const char *pattern)
"WHERE c.relnamespace = n.oid\n"
" AND c.oid = pr.prrelid\n"
" AND pr.prpubid = '%s'\n"
+ " AND c.relkind != 'S'\n" /* exclude sequences */
"ORDER BY 1,2", pubid);
if (!addFooterToPublicationDesc(&buf, "Tables:", false, &cont))
goto error_return;
@@ -6097,7 +6213,7 @@ describePublications(const char *pattern)
"SELECT n.nspname\n"
"FROM pg_catalog.pg_namespace n\n"
" JOIN pg_catalog.pg_publication_namespace pn ON n.oid = pn.pnnspid\n"
- "WHERE pn.pnpubid = '%s'\n"
+ "WHERE pn.pnpubid = '%s' AND pn.pntype = 't'\n"
"ORDER BY 1", pubid);
if (!addFooterToPublicationDesc(&buf, "Tables from schemas:",
true, &cont))
@@ -6105,6 +6221,37 @@ describePublications(const char *pattern)
}
}
+ if (!puballsequences)
+ {
+ /* Get the sequences for the specified publication */
+ printfPQExpBuffer(&buf,
+ "SELECT n.nspname, c.relname, NULL, NULL\n"
+ "FROM pg_catalog.pg_class c,\n"
+ " pg_catalog.pg_namespace n,\n"
+ " pg_catalog.pg_publication_rel pr\n"
+ "WHERE c.relnamespace = n.oid\n"
+ " AND c.oid = pr.prrelid\n"
+ " AND pr.prpubid = '%s'\n"
+ " AND c.relkind = 'S'\n" /* only sequences */
+ "ORDER BY 1,2", pubid);
+ if (!addFooterToPublicationDesc(&buf, "Sequences:", false, &cont))
+ goto error_return;
+
+ if (pset.sversion >= 150000)
+ {
+ /* Get the schemas for the specified publication */
+ printfPQExpBuffer(&buf,
+ "SELECT n.nspname\n"
+ "FROM pg_catalog.pg_namespace n\n"
+ " JOIN pg_catalog.pg_publication_namespace pn ON n.oid = pn.pnnspid\n"
+ "WHERE pn.pnpubid = '%s' AND pn.pntype = 's'\n"
+ "ORDER BY 1", pubid);
+ if (!addFooterToPublicationDesc(&buf, "Sequences from schemas:",
+ true, &cont))
+ goto error_return;
+ }
+ }
+
printTable(&cont, pset.queryFout, false, pset.logfile);
printTableCleanup(&cont);
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index a2df39d0c1b..025d3f71a11 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1827,11 +1827,15 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("ADD", "DROP", "OWNER TO", "RENAME TO", "SET");
/* ALTER PUBLICATION <name> ADD */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD"))
- COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ COMPLETE_WITH("ALL TABLES IN SCHEMA", "ALL SEQUENCES IN SCHEMA", "TABLE", "SEQUENCE");
else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD|SET", "TABLE") ||
(HeadMatches("ALTER", "PUBLICATION", MatchAny, "ADD|SET", "TABLE") &&
ends_with(prev_wd, ',')))
COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables);
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD|SET", "SEQUENCE") ||
+ (HeadMatches("ALTER", "PUBLICATION", MatchAny, "ADD|SET", "SEQUENCE") &&
+ ends_with(prev_wd, ',')))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_sequences);
/*
* "ALTER PUBLICATION <name> SET TABLE <name> WHERE (" - complete with
* table attributes
@@ -1850,11 +1854,11 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH(",");
/* ALTER PUBLICATION <name> DROP */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "DROP"))
- COMPLETE_WITH("ALL TABLES IN SCHEMA", "TABLE");
+ COMPLETE_WITH("ALL TABLES IN SCHEMA", "ALL SEQUENCES IN SCHEMA", "TABLE", "SEQUENCE");
/* ALTER PUBLICATION <name> SET */
else if (Matches("ALTER", "PUBLICATION", MatchAny, "SET"))
- COMPLETE_WITH("(", "ALL TABLES IN SCHEMA", "TABLE");
- else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD|DROP|SET", "ALL", "TABLES", "IN", "SCHEMA"))
+ COMPLETE_WITH("(", "ALL TABLES IN SCHEMA", "ALL SEQUENCES IN SCHEMA", "TABLE", "SEQUENCE");
+ else if (Matches("ALTER", "PUBLICATION", MatchAny, "ADD|DROP|SET", "ALL", "TABLES|SEQUENCES", "IN", "SCHEMA"))
COMPLETE_WITH_QUERY_PLUS(Query_for_list_of_schemas
" AND nspname NOT LIKE E'pg\\\\_%%'",
"CURRENT_SCHEMA");
@@ -2984,21 +2988,27 @@ psql_completion(const char *text, int start, int end)
/* CREATE PUBLICATION */
else if (Matches("CREATE", "PUBLICATION", MatchAny))
- COMPLETE_WITH("FOR TABLE", "FOR ALL TABLES", "FOR ALL TABLES IN SCHEMA", "WITH (");
+ COMPLETE_WITH("FOR TABLE", "FOR ALL TABLES", "FOR ALL TABLES IN SCHEMA",
+ "FOR SEQUENCE", "FOR ALL SEQUENCES", "FOR ALL SEQUENCES IN SCHEMA",
+ "WITH (");
else if (Matches("CREATE", "PUBLICATION", MatchAny, "FOR"))
- COMPLETE_WITH("TABLE", "ALL TABLES", "ALL TABLES IN SCHEMA");
+ COMPLETE_WITH("TABLE", "ALL TABLES", "ALL TABLES IN SCHEMA",
+ "SEQUENCE", "ALL SEQUENCES", "ALL SEQUENCES IN SCHEMA");
else if (Matches("CREATE", "PUBLICATION", MatchAny, "FOR", "ALL"))
- COMPLETE_WITH("TABLES", "TABLES IN SCHEMA");
- else if (Matches("CREATE", "PUBLICATION", MatchAny, "FOR", "ALL", "TABLES"))
+ COMPLETE_WITH("TABLES", "TABLES IN SCHEMA", "SEQUENCES", "SEQUENCES IN SCHEMA");
+ else if (Matches("CREATE", "PUBLICATION", MatchAny, "FOR", "ALL", "TABLES|SEQUENCES"))
COMPLETE_WITH("IN SCHEMA", "WITH (");
- else if (Matches("CREATE", "PUBLICATION", MatchAny, "FOR", "TABLE", MatchAny) && !ends_with(prev_wd, ','))
+ else if (Matches("CREATE", "PUBLICATION", MatchAny, "FOR", "TABLE|SEQUENCE", MatchAny) && !ends_with(prev_wd, ','))
COMPLETE_WITH("WHERE (", "WITH (");
/* Complete "CREATE PUBLICATION <name> FOR TABLE" with "<table>, ..." */
else if (Matches("CREATE", "PUBLICATION", MatchAny, "FOR", "TABLE"))
COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_tables);
+ /* Complete "CREATE PUBLICATION <name> FOR SEQUENCE" with "<sequence>, ..." */
+ else if (Matches("CREATE", "PUBLICATION", MatchAny, "FOR", "SEQUENCE"))
+ COMPLETE_WITH_SCHEMA_QUERY(Query_for_list_of_sequences);
/*
- * "CREATE PUBLICATION <name> FOR TABLE <name> WHERE (" - complete with
+ * "CREATE PUBLICATION <name> FOR TABLE|SEQUENCE <name> WHERE (" - complete with
* table attributes
*/
else if (HeadMatches("CREATE", "PUBLICATION", MatchAny) && TailMatches("WHERE"))
@@ -3009,14 +3019,14 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH(" WITH (");
/*
- * Complete "CREATE PUBLICATION <name> FOR ALL TABLES IN SCHEMA <schema>,
+ * Complete "CREATE PUBLICATION <name> FOR ALL TABLES|SEQUENCES IN SCHEMA <schema>,
* ..."
*/
- else if (Matches("CREATE", "PUBLICATION", MatchAny, "FOR", "ALL", "TABLES", "IN", "SCHEMA"))
+ else if (Matches("CREATE", "PUBLICATION", MatchAny, "FOR", "ALL", "TABLES|SEQUENCES", "IN", "SCHEMA"))
COMPLETE_WITH_QUERY_PLUS(Query_for_list_of_schemas
" AND nspname NOT LIKE E'pg\\\\_%%'",
"CURRENT_SCHEMA");
- else if (Matches("CREATE", "PUBLICATION", MatchAny, "FOR", "ALL", "TABLES", "IN", "SCHEMA", MatchAny) && (!ends_with(prev_wd, ',')))
+ else if (Matches("CREATE", "PUBLICATION", MatchAny, "FOR", "ALL", "TABLES|SEQUENCES", "IN", "SCHEMA", MatchAny) && (!ends_with(prev_wd, ',')))
COMPLETE_WITH("WITH (");
/* Complete "CREATE PUBLICATION <name> [...] WITH" */
else if (HeadMatches("CREATE", "PUBLICATION") && TailMatches("WITH", "("))
diff --git a/src/include/catalog/catversion.h b/src/include/catalog/catversion.h
index cec70caae1f..67f3d8526cd 100644
--- a/src/include/catalog/catversion.h
+++ b/src/include/catalog/catversion.h
@@ -53,6 +53,6 @@
*/
/* yyyymmddN */
-#define CATALOG_VERSION_NO 202204075
+#define CATALOG_VERSION_NO 202204074
#endif
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 52f56cf5b1b..61876c4e808 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -11658,6 +11658,11 @@
provolatile => 's', prorettype => 'oid', proargtypes => 'text',
proallargtypes => '{text,oid}', proargmodes => '{i,o}',
proargnames => '{pubname,relid}', prosrc => 'pg_get_publication_tables' },
+{ oid => '8000', descr => 'get OIDs of sequences in a publication',
+ proname => 'pg_get_publication_sequences', prorows => '1000', proretset => 't',
+ provolatile => 's', prorettype => 'oid', proargtypes => 'text',
+ proallargtypes => '{text,oid}', proargmodes => '{i,o}',
+ proargnames => '{pubname,relid}', prosrc => 'pg_get_publication_sequences' },
{ oid => '6121',
descr => 'returns whether a relation can be part of a publication',
proname => 'pg_relation_is_publishable', provolatile => 's',
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 29b18566657..186d8ea74b6 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -40,6 +40,12 @@ CATALOG(pg_publication,6104,PublicationRelationId)
*/
bool puballtables;
+ /*
+ * indicates that this is special publication which should encompass all
+ * sequences in the database (except for the unlogged and temp ones)
+ */
+ bool puballsequences;
+
/* true if inserts are published */
bool pubinsert;
@@ -52,6 +58,9 @@ CATALOG(pg_publication,6104,PublicationRelationId)
/* true if truncates are published */
bool pubtruncate;
+ /* true if sequences are published */
+ bool pubsequence;
+
/* true if partition changes are published using root schema */
bool pubviaroot;
} FormData_pg_publication;
@@ -72,6 +81,7 @@ typedef struct PublicationActions
bool pubupdate;
bool pubdelete;
bool pubtruncate;
+ bool pubsequence;
} PublicationActions;
typedef struct PublicationDesc
@@ -99,6 +109,7 @@ typedef struct Publication
Oid oid;
char *name;
bool alltables;
+ bool allsequences;
bool pubviaroot;
PublicationActions pubactions;
} Publication;
@@ -130,14 +141,16 @@ typedef enum PublicationPartOpt
PUBLICATION_PART_ALL,
} PublicationPartOpt;
-extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetPublicationRelations(Oid pubid, char objectType,
+ PublicationPartOpt pub_partopt);
extern List *GetAllTablesPublications(void);
extern List *GetAllTablesPublicationRelations(bool pubviaroot);
-extern List *GetPublicationSchemas(Oid pubid);
-extern List *GetSchemaPublications(Oid schemaid);
-extern List *GetSchemaPublicationRelations(Oid schemaid,
+extern void GetActionsInPublication(Oid pubid, PublicationActions *actions);
+extern List *GetPublicationSchemas(Oid pubid, char objectType);
+extern List *GetSchemaPublications(Oid schemaid, char objectType);
+extern List *GetSchemaPublicationRelations(Oid schemaid, char objectType,
PublicationPartOpt pub_partopt);
-extern List *GetAllSchemaPublicationRelations(Oid puboid,
+extern List *GetAllSchemaPublicationRelations(Oid puboid, char objectType,
PublicationPartOpt pub_partopt);
extern List *GetPubPartitionOptionRelations(List *result,
PublicationPartOpt pub_partopt,
@@ -145,11 +158,15 @@ extern List *GetPubPartitionOptionRelations(List *result,
extern Oid GetTopMostAncestorInPublication(Oid puboid, List *ancestors,
int *ancestor_level);
+extern List *GetAllSequencesPublications(void);
+extern List *GetAllSequencesPublicationRelations(void);
+
extern bool is_publishable_relation(Relation rel);
extern bool is_schema_publication(Oid pubid);
extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelInfo *pri,
bool if_not_exists);
extern ObjectAddress publication_add_schema(Oid pubid, Oid schemaid,
+ char objectType,
bool if_not_exists);
extern Bitmapset *pub_collist_to_bitmapset(Bitmapset *columns, Datum pubcols,
diff --git a/src/include/catalog/pg_publication_namespace.h b/src/include/catalog/pg_publication_namespace.h
index e4306da02e7..7340a1ec646 100644
--- a/src/include/catalog/pg_publication_namespace.h
+++ b/src/include/catalog/pg_publication_namespace.h
@@ -32,6 +32,7 @@ CATALOG(pg_publication_namespace,8901,PublicationNamespaceRelationId)
Oid oid; /* oid */
Oid pnpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
Oid pnnspid BKI_LOOKUP(pg_namespace); /* Oid of the schema */
+ char pntype; /* object type to include */
} FormData_pg_publication_namespace;
/* ----------------
@@ -42,6 +43,13 @@ CATALOG(pg_publication_namespace,8901,PublicationNamespaceRelationId)
typedef FormData_pg_publication_namespace *Form_pg_publication_namespace;
DECLARE_UNIQUE_INDEX_PKEY(pg_publication_namespace_oid_index, 8902, PublicationNamespaceObjectIndexId, on pg_publication_namespace using btree(oid oid_ops));
-DECLARE_UNIQUE_INDEX(pg_publication_namespace_pnnspid_pnpubid_index, 8903, PublicationNamespacePnnspidPnpubidIndexId, on pg_publication_namespace using btree(pnnspid oid_ops, pnpubid oid_ops));
+DECLARE_UNIQUE_INDEX(pg_publication_namespace_pnnspid_pnpubid_pntype_index, 8903, PublicationNamespacePnnspidPnpubidPntypeIndexId, on pg_publication_namespace using btree(pnnspid oid_ops, pnpubid oid_ops, pntype char_ops));
+
+/* object type to include from a schema, maps to relkind */
+#define PUB_OBJTYPE_TABLE 't' /* table (regular or partitioned) */
+#define PUB_OBJTYPE_SEQUENCE 's' /* sequence object */
+#define PUB_OBJTYPE_UNSUPPORTED 'u' /* used for non-replicated types */
+
+extern char pub_get_object_type_for_relkind(char relkind);
#endif /* PG_PUBLICATION_NAMESPACE_H */
diff --git a/src/include/commands/sequence.h b/src/include/commands/sequence.h
index 9da23008101..57807874d76 100644
--- a/src/include/commands/sequence.h
+++ b/src/include/commands/sequence.h
@@ -60,8 +60,11 @@ extern ObjectAddress AlterSequence(ParseState *pstate, AlterSeqStmt *stmt);
extern void SequenceChangePersistence(Oid relid, char newrelpersistence);
extern void DeleteSequenceTuple(Oid relid);
extern void ResetSequence(Oid seq_relid);
+extern void SetSequence(Oid seq_relid, int64 last_value, int64 log_cnt, bool is_called);
extern void ResetSequenceCaches(void);
+extern void AtEOXact_Sequences(bool isCommit);
+
extern void seq_redo(XLogReaderState *rptr);
extern void seq_desc(StringInfo buf, XLogReaderState *rptr);
extern const char *seq_identify(uint8 info);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1bdeb9eb3d3..8998d345605 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -4013,6 +4013,10 @@ typedef enum PublicationObjSpecType
PUBLICATIONOBJ_TABLES_IN_SCHEMA, /* All tables in schema */
PUBLICATIONOBJ_TABLES_IN_CUR_SCHEMA, /* All tables in first element of
* search_path */
+ PUBLICATIONOBJ_SEQUENCE, /* Sequence type */
+ PUBLICATIONOBJ_SEQUENCES_IN_SCHEMA, /* Sequences in schema type */
+ PUBLICATIONOBJ_SEQUENCES_IN_CUR_SCHEMA, /* Get the first element of
+ * search_path */
PUBLICATIONOBJ_CONTINUATION /* Continuation of previous type */
} PublicationObjSpecType;
@@ -4031,7 +4035,7 @@ typedef struct CreatePublicationStmt
char *pubname; /* Name of the publication */
List *options; /* List of DefElem nodes */
List *pubobjects; /* Optional list of publication objects */
- bool for_all_tables; /* Special publication for all tables in db */
+ List *for_all_objects; /* Special publication for all objects in db */
} CreatePublicationStmt;
typedef enum AlterPublicationAction
@@ -4054,7 +4058,7 @@ typedef struct AlterPublicationStmt
* objects.
*/
List *pubobjects; /* Optional list of publication objects */
- bool for_all_tables; /* Special publication for all tables in db */
+ List *for_all_objects; /* Special publication for all objects in db */
AlterPublicationAction action; /* What action to perform with the given
* objects */
} AlterPublicationStmt;
diff --git a/src/include/replication/decode.h b/src/include/replication/decode.h
index a33c2a718a7..8e07bb7409a 100644
--- a/src/include/replication/decode.h
+++ b/src/include/replication/decode.h
@@ -27,6 +27,7 @@ extern void heap2_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
extern void xact_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
extern void standby_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
extern void logicalmsg_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
+extern void sequence_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf);
extern void LogicalDecodingProcessRecord(LogicalDecodingContext *ctx,
XLogReaderState *record);
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index a771ab8ff33..a466db394b8 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -61,6 +61,7 @@ typedef enum LogicalRepMsgType
LOGICAL_REP_MSG_RELATION = 'R',
LOGICAL_REP_MSG_TYPE = 'Y',
LOGICAL_REP_MSG_MESSAGE = 'M',
+ LOGICAL_REP_MSG_SEQUENCE = 'Q',
LOGICAL_REP_MSG_BEGIN_PREPARE = 'b',
LOGICAL_REP_MSG_PREPARE = 'P',
LOGICAL_REP_MSG_COMMIT_PREPARED = 'K',
@@ -118,6 +119,17 @@ typedef struct LogicalRepTyp
char *typname; /* name of the remote type */
} LogicalRepTyp;
+/* Sequence info */
+typedef struct LogicalRepSequence
+{
+ Oid remoteid; /* unique id of the remote sequence */
+ char *nspname; /* schema name of remote sequence */
+ char *seqname; /* name of the remote sequence */
+ int64 last_value;
+ int64 log_cnt;
+ bool is_called;
+} LogicalRepSequence;
+
/* Transaction info */
typedef struct LogicalRepBeginData
{
@@ -230,6 +242,11 @@ extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn,
bool transactional, const char *prefix, Size sz, const char *message);
+extern void logicalrep_write_sequence(StringInfo out, Relation rel,
+ TransactionId xid, XLogRecPtr lsn,
+ int64 last_value, int64 log_cnt,
+ bool is_called);
+extern void logicalrep_read_sequence(StringInfo in, LogicalRepSequence *seqdata);
extern void logicalrep_write_rel(StringInfo out, TransactionId xid,
Relation rel, Bitmapset *columns);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
diff --git a/src/include/replication/message.h b/src/include/replication/message.h
index 7d7785292f1..a084acc06ab 100644
--- a/src/include/replication/message.h
+++ b/src/include/replication/message.h
@@ -27,13 +27,28 @@ typedef struct xl_logical_message
char message[FLEXIBLE_ARRAY_MEMBER];
} xl_logical_message;
+/*
+ * Generic logical decoding sequence wal record.
+ */
+typedef struct xl_logical_sequence
+{
+ RelFileNode node;
+ Oid reloid;
+ int64 last; /* last value emitted for sequence */
+ int64 log_cnt; /* last value cached for sequence */
+ bool is_called;
+} xl_logical_sequence;
+
#define SizeOfLogicalMessage (offsetof(xl_logical_message, message))
+#define SizeOfLogicalSequence (sizeof(xl_logical_sequence))
extern XLogRecPtr LogLogicalMessage(const char *prefix, const char *message,
size_t size, bool transactional);
/* RMGR API*/
#define XLOG_LOGICAL_MESSAGE 0x00
+#define XLOG_LOGICAL_SEQUENCE 0x10
+
void logicalmsg_redo(XLogReaderState *record);
void logicalmsg_desc(StringInfo buf, XLogReaderState *record);
const char *logicalmsg_identify(uint8 info);
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 539dc8e6974..abc05644516 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -88,6 +88,17 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
Size message_size,
const char *message);
+/*
+ * Called for the generic logical decoding sequences.
+ */
+typedef void (*LogicalDecodeSequenceCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr sequence_lsn,
+ Relation rel,
+ int64 last_value,
+ int64 log_cnt,
+ bool is_called);
+
/*
* Filter changes by origin.
*/
@@ -199,6 +210,18 @@ typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx
Size message_size,
const char *message);
+/*
+ * Called for the streaming generic logical decoding sequences from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamSequenceCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr sequence_lsn,
+ Relation rel,
+ int64 last_value,
+ int64 log_cnt,
+ bool is_called);
+
/*
* Callback for streaming truncates from in-progress transactions.
*/
@@ -219,6 +242,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeTruncateCB truncate_cb;
LogicalDecodeCommitCB commit_cb;
LogicalDecodeMessageCB message_cb;
+ LogicalDecodeSequenceCB sequence_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
@@ -237,6 +261,7 @@ typedef struct OutputPluginCallbacks
LogicalDecodeStreamCommitCB stream_commit_cb;
LogicalDecodeStreamChangeCB stream_change_cb;
LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamSequenceCB stream_sequence_cb;
LogicalDecodeStreamTruncateCB stream_truncate_cb;
} OutputPluginCallbacks;
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index eafedd610a5..f4e9f35d09d 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -29,6 +29,7 @@ typedef struct PGOutputData
bool streaming;
bool messages;
bool two_phase;
+ bool sequences;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index f12e75d69be..260630c2ba5 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -64,7 +64,8 @@ typedef enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT,
REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM,
REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT,
- REORDER_BUFFER_CHANGE_TRUNCATE
+ REORDER_BUFFER_CHANGE_TRUNCATE,
+ REORDER_BUFFER_CHANGE_SEQUENCE
} ReorderBufferChangeType;
/* forward declaration */
@@ -158,6 +159,16 @@ typedef struct ReorderBufferChange
uint32 ninvalidations; /* Number of messages */
SharedInvalidationMessage *invalidations; /* invalidation message */
} inval;
+
+ /* Context data for Sequence changes */
+ struct
+ {
+ RelFileNode relnode;
+ Oid reloid;
+ int64 last;
+ int64 log_cnt;
+ bool is_called;
+ } sequence;
} data;
/*
@@ -430,6 +441,14 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* sequence callback signature */
+typedef void (*ReorderBufferSequenceCB) (ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr sequence_lsn,
+ Relation rel,
+ int64 last_value, int64 log_cnt,
+ bool is_called);
+
/* begin prepare callback signature */
typedef void (*ReorderBufferBeginPrepareCB) (ReorderBuffer *rb,
ReorderBufferTXN *txn);
@@ -496,6 +515,14 @@ typedef void (*ReorderBufferStreamMessageCB) (
const char *prefix, Size sz,
const char *message);
+/* stream sequence callback signature */
+typedef void (*ReorderBufferStreamSequenceCB) (ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr sequence_lsn,
+ Relation rel,
+ int64 last_value, int64 log_cnt,
+ bool is_called);
+
/* stream truncate callback signature */
typedef void (*ReorderBufferStreamTruncateCB) (
ReorderBuffer *rb,
@@ -511,6 +538,12 @@ struct ReorderBuffer
*/
HTAB *by_txn;
+ /*
+ * relfilenode => XID lookup table for sequences created in a transaction
+ * (also includes altered sequences, which assigns new relfilenode)
+ */
+ HTAB *sequences;
+
/*
* Transactions that could be a toplevel xact, ordered by LSN of the first
* record bearing that xid.
@@ -541,6 +574,7 @@ struct ReorderBuffer
ReorderBufferApplyTruncateCB apply_truncate;
ReorderBufferCommitCB commit;
ReorderBufferMessageCB message;
+ ReorderBufferSequenceCB sequence;
/*
* Callbacks to be called when streaming a transaction at prepare time.
@@ -560,6 +594,7 @@ struct ReorderBuffer
ReorderBufferStreamCommitCB stream_commit;
ReorderBufferStreamChangeCB stream_change;
ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamSequenceCB stream_sequence;
ReorderBufferStreamTruncateCB stream_truncate;
/*
@@ -635,6 +670,10 @@ void ReorderBufferQueueChange(ReorderBuffer *, TransactionId,
void ReorderBufferQueueMessage(ReorderBuffer *, TransactionId, Snapshot snapshot, XLogRecPtr lsn,
bool transactional, const char *prefix,
Size message_size, const char *message);
+void ReorderBufferQueueSequence(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, RepOriginId origin_id,
+ Oid reloid, RelFileNode rnode,
+ int64 last, int64 log_cnt, bool is_called);
void ReorderBufferCommit(ReorderBuffer *, TransactionId,
XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
TimestampTz commit_time, RepOriginId origin_id, XLogRecPtr origin_lsn);
diff --git a/src/test/regress/expected/object_address.out b/src/test/regress/expected/object_address.out
index 4117fc27c9a..c95d44b3db9 100644
--- a/src/test/regress/expected/object_address.out
+++ b/src/test/regress/expected/object_address.out
@@ -46,6 +46,7 @@ CREATE TRANSFORM FOR int LANGUAGE SQL (
SET client_min_messages = 'ERROR';
CREATE PUBLICATION addr_pub FOR TABLE addr_nsp.gentable;
CREATE PUBLICATION addr_pub_schema FOR ALL TABLES IN SCHEMA addr_nsp;
+CREATE PUBLICATION addr_pub_schema2 FOR ALL SEQUENCES IN SCHEMA addr_nsp;
RESET client_min_messages;
CREATE SUBSCRIPTION regress_addr_sub CONNECTION '' PUBLICATION bar WITH (connect = false, slot_name = NONE);
WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
@@ -428,7 +429,8 @@ WITH objects (type, name, args) AS (VALUES
('transform', '{int}', '{sql}'),
('access method', '{btree}', '{}'),
('publication', '{addr_pub}', '{}'),
- ('publication namespace', '{addr_nsp}', '{addr_pub_schema}'),
+ ('publication namespace', '{addr_nsp}', '{addr_pub_schema,t}'),
+ ('publication namespace', '{addr_nsp}', '{addr_pub_schema2,s}'),
('publication relation', '{addr_nsp, gentable}', '{addr_pub}'),
('subscription', '{regress_addr_sub}', '{}'),
('statistics object', '{addr_nsp, gentable_stat}', '{}')
@@ -492,8 +494,9 @@ SELECT (pg_identify_object(addr1.classid, addr1.objid, addr1.objsubid)).*,
subscription | | regress_addr_sub | regress_addr_sub | t
publication | | addr_pub | addr_pub | t
publication relation | | | addr_nsp.gentable in publication addr_pub | t
- publication namespace | | | addr_nsp in publication addr_pub_schema | t
-(50 rows)
+ publication namespace | | | addr_nsp in publication addr_pub_schema type t | t
+ publication namespace | | | addr_nsp in publication addr_pub_schema2 type s | t
+(51 rows)
---
--- Cleanup resources
@@ -506,6 +509,7 @@ drop cascades to server integer
drop cascades to user mapping for regress_addr_user on server integer
DROP PUBLICATION addr_pub;
DROP PUBLICATION addr_pub_schema;
+DROP PUBLICATION addr_pub_schema2;
DROP SUBSCRIPTION regress_addr_sub;
DROP SCHEMA addr_nsp CASCADE;
NOTICE: drop cascades to 14 other objects
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 4d24d772bde..0308e40ba6c 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -30,20 +30,20 @@ ERROR: conflicting or redundant options
LINE 1: ...ub_xxx WITH (publish_via_partition_root = 'true', publish_vi...
^
\dRp
- List of publications
- Name | Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------+--------------------------+------------+---------+---------+---------+-----------+----------
- testpib_ins_trunct | regress_publication_user | f | t | f | f | f | f
- testpub_default | regress_publication_user | f | f | t | f | f | f
+ List of publications
+ Name | Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ testpib_ins_trunct | regress_publication_user | f | f | t | f | f | f | f | f
+ testpub_default | regress_publication_user | f | f | f | t | f | f | f | f
(2 rows)
-ALTER PUBLICATION testpub_default SET (publish = 'insert, update, delete');
+ALTER PUBLICATION testpub_default SET (publish = 'insert, update, delete, sequence');
\dRp
- List of publications
- Name | Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------+--------------------------+------------+---------+---------+---------+-----------+----------
- testpib_ins_trunct | regress_publication_user | f | t | f | f | f | f
- testpub_default | regress_publication_user | f | t | t | t | f | f
+ List of publications
+ Name | Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ testpib_ins_trunct | regress_publication_user | f | f | t | f | f | f | f | f
+ testpub_default | regress_publication_user | f | f | t | t | t | f | t | f
(2 rows)
--- adding tables
@@ -61,6 +61,9 @@ CREATE TABLE testpub_tbl2 (id serial primary key, data text);
ALTER PUBLICATION testpub_foralltables ADD TABLE testpub_tbl2;
ERROR: publication "testpub_foralltables" is defined as FOR ALL TABLES
DETAIL: Tables cannot be added to or dropped from FOR ALL TABLES publications.
+-- fail - can't add a table using ADD SEQUENCE command
+ALTER PUBLICATION testpub_foralltables ADD SEQUENCE testpub_tbl2;
+ERROR: object type does not match type expected by command
-- fail - can't drop from all tables publication
ALTER PUBLICATION testpub_foralltables DROP TABLE testpub_tbl2;
ERROR: publication "testpub_foralltables" is defined as FOR ALL TABLES
@@ -87,10 +90,10 @@ RESET client_min_messages;
-- should be able to add schema to 'FOR TABLE' publication
ALTER PUBLICATION testpub_fortable ADD ALL TABLES IN SCHEMA pub_test;
\dRp+ testpub_fortable
- Publication testpub_fortable
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub_fortable
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables:
"public.testpub_tbl1"
Tables from schemas:
@@ -99,20 +102,20 @@ Tables from schemas:
-- should be able to drop schema from 'FOR TABLE' publication
ALTER PUBLICATION testpub_fortable DROP ALL TABLES IN SCHEMA pub_test;
\dRp+ testpub_fortable
- Publication testpub_fortable
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub_fortable
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables:
"public.testpub_tbl1"
-- should be able to set schema to 'FOR TABLE' publication
ALTER PUBLICATION testpub_fortable SET ALL TABLES IN SCHEMA pub_test;
\dRp+ testpub_fortable
- Publication testpub_fortable
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub_fortable
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test"
@@ -134,10 +137,10 @@ ERROR: relation "testpub_nopk" is not part of the publication
-- should be able to set table to schema publication
ALTER PUBLICATION testpub_forschema SET TABLE pub_test.testpub_nopk;
\dRp+ testpub_forschema
- Publication testpub_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables:
"pub_test.testpub_nopk"
@@ -159,10 +162,10 @@ Publications:
"testpub_foralltables"
\dRp+ testpub_foralltables
- Publication testpub_foralltables
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | t | t | t | f | f | f
+ Publication testpub_foralltables
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | t | f | t | t | f | f | f | f
(1 row)
DROP TABLE testpub_tbl2;
@@ -174,24 +177,545 @@ CREATE PUBLICATION testpub3 FOR TABLE testpub_tbl3;
CREATE PUBLICATION testpub4 FOR TABLE ONLY testpub_tbl3;
RESET client_min_messages;
\dRp+ testpub3
- Publication testpub3
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub3
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables:
"public.testpub_tbl3"
"public.testpub_tbl3a"
\dRp+ testpub4
- Publication testpub4
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub4
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables:
"public.testpub_tbl3"
DROP TABLE testpub_tbl3, testpub_tbl3a;
DROP PUBLICATION testpub3, testpub4;
+--- adding sequences
+CREATE SEQUENCE testpub_seq0;
+CREATE SEQUENCE pub_test.testpub_seq1;
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_forallsequences FOR ALL SEQUENCES WITH (publish = 'sequence');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_forallsequences SET (publish = 'insert, sequence');
+CREATE SEQUENCE testpub_seq2;
+-- fail - can't add to for all sequences publication
+ALTER PUBLICATION testpub_forallsequences ADD SEQUENCE testpub_seq2;
+ERROR: publication "testpub_forallsequences" is defined as FOR ALL SEQUENCES
+DETAIL: Sequences cannot be added to or dropped from FOR ALL SEQUENCES publications.
+-- fail - can't drop from all sequences publication
+ALTER PUBLICATION testpub_forallsequences DROP SEQUENCE testpub_seq2;
+ERROR: publication "testpub_forallsequences" is defined as FOR ALL SEQUENCES
+DETAIL: Sequences cannot be added to or dropped from FOR ALL SEQUENCES publications.
+-- fail - can't add to for all sequences publication
+ALTER PUBLICATION testpub_forallsequences SET SEQUENCE pub_test.testpub_seq1;
+ERROR: publication "testpub_forallsequences" is defined as FOR ALL SEQUENCES
+DETAIL: Sequences cannot be added to or dropped from FOR ALL SEQUENCES publications.
+-- fail - can't add schema to 'FOR ALL SEQUENCES' publication
+ALTER PUBLICATION testpub_forallsequences ADD ALL SEQUENCES IN SCHEMA pub_test;
+ERROR: publication "testpub_forallsequences" is defined as FOR ALL SEQUENCES
+DETAIL: Sequences from schema cannot be added to, dropped from, or set on FOR ALL SEQUENCES publications.
+-- fail - can't drop schema from 'FOR ALL SEQUENCES' publication
+ALTER PUBLICATION testpub_forallsequences DROP ALL SEQUENCES IN SCHEMA pub_test;
+ERROR: publication "testpub_forallsequences" is defined as FOR ALL SEQUENCES
+DETAIL: Sequences from schema cannot be added to, dropped from, or set on FOR ALL SEQUENCES publications.
+-- fail - can't set schema to 'FOR ALL SEQUENCES' publication
+ALTER PUBLICATION testpub_forallsequences SET ALL SEQUENCES IN SCHEMA pub_test;
+ERROR: publication "testpub_forallsequences" is defined as FOR ALL SEQUENCES
+DETAIL: Sequences from schema cannot be added to, dropped from, or set on FOR ALL SEQUENCES publications.
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_forsequence FOR SEQUENCE testpub_seq0;
+RESET client_min_messages;
+-- should be able to add schema to 'FOR SEQUENCE' publication
+ALTER PUBLICATION testpub_forsequence ADD ALL SEQUENCES IN SCHEMA pub_test;
+\dRp+ testpub_forsequence
+ Publication testpub_forsequence
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Sequences:
+ "public.testpub_seq0"
+Sequences from schemas:
+ "pub_test"
+
+-- fail - can't add sequence from the schema we already added
+ALTER PUBLICATION testpub_forsequence ADD SEQUENCE pub_test.testpub_seq1;
+ERROR: cannot add relation "pub_test.testpub_seq1" to publication
+DETAIL: Sequence's schema "pub_test" is already part of the publication or part of the specified schema list.
+-- fail - can't add sequence using ADD TABLE command
+ALTER PUBLICATION testpub_forsequence ADD TABLE pub_test.testpub_seq1;
+ERROR: object type does not match type expected by command
+-- should be able to drop schema from 'FOR SEQUENCE' publication
+ALTER PUBLICATION testpub_forsequence DROP ALL SEQUENCES IN SCHEMA pub_test;
+\dRp+ testpub_forsequence
+ Publication testpub_forsequence
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Sequences:
+ "public.testpub_seq0"
+
+-- should be able to set schema to 'FOR SEQUENCE' publication
+ALTER PUBLICATION testpub_forsequence SET ALL SEQUENCES IN SCHEMA pub_test;
+\dRp+ testpub_forsequence
+ Publication testpub_forsequence
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Sequences from schemas:
+ "pub_test"
+
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_forschema FOR ALL SEQUENCES IN SCHEMA pub_test;
+RESET client_min_messages;
+-- fail - can't create publication with schema and sequence of the same schema
+CREATE PUBLICATION testpub_for_seq_schema FOR ALL SEQUENCES IN SCHEMA pub_test, SEQUENCE pub_test.testpub_seq1;
+ERROR: cannot add relation "pub_test.testpub_seq1" to publication
+DETAIL: Sequence's schema "pub_test" is already part of the publication or part of the specified schema list.
+-- fail - can't add a sequence of the same schema to the schema publication
+ALTER PUBLICATION testpub_forschema ADD SEQUENCE pub_test.testpub_seq1;
+ERROR: cannot add relation "pub_test.testpub_seq1" to publication
+DETAIL: Sequence's schema "pub_test" is already part of the publication or part of the specified schema list.
+-- fail - can't drop a sequence from the schema publication which isn't in the
+-- publication
+ALTER PUBLICATION testpub_forschema DROP SEQUENCE pub_test.testpub_seq1;
+ERROR: relation "testpub_seq1" is not part of the publication
+-- should be able to set sequence to schema publication
+ALTER PUBLICATION testpub_forschema SET SEQUENCE pub_test.testpub_seq1;
+\dRp+ testpub_forschema
+ Publication testpub_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Sequences:
+ "pub_test.testpub_seq1"
+
+SELECT pubname, puballtables, puballsequences FROM pg_publication WHERE pubname = 'testpub_forallsequences';
+ pubname | puballtables | puballsequences
+-------------------------+--------------+-----------------
+ testpub_forallsequences | f | t
+(1 row)
+
+\d+ pub_test.testpub_seq1
+ Sequence "pub_test.testpub_seq1"
+ Type | Start | Minimum | Maximum | Increment | Cycles? | Cache
+--------+-------+---------+---------------------+-----------+---------+-------
+ bigint | 1 | 1 | 9223372036854775807 | 1 | no | 1
+Publications:
+ "testpub_forallsequences"
+ "testpub_forschema"
+ "testpub_forsequence"
+
+\dRp+ testpub_forallsequences
+ Publication testpub_forallsequences
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | t | t | f | f | f | t | f
+(1 row)
+
+DROP SEQUENCE testpub_seq0, pub_test.testpub_seq1, testpub_seq2;
+DROP PUBLICATION testpub_forallsequences, testpub_forsequence, testpub_forschema;
+-- publication testing multiple sequences at the same time
+CREATE SEQUENCE testpub_seq1;
+CREATE SEQUENCE testpub_seq2;
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_multi FOR SEQUENCE testpub_seq1, testpub_seq2;
+RESET client_min_messages;
+\dRp+ testpub_multi
+ Publication testpub_multi
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Sequences:
+ "public.testpub_seq1"
+ "public.testpub_seq2"
+
+DROP PUBLICATION testpub_multi;
+DROP SEQUENCE testpub_seq1;
+DROP SEQUENCE testpub_seq2;
+-- Publication mixing tables and sequences
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_mix;
+RESET client_min_messages;
+CREATE SEQUENCE testpub_seq1;
+CREATE SEQUENCE pub_test.testpub_seq2;
+ALTER PUBLICATION testpub_mix ADD SEQUENCE testpub_seq1, TABLE testpub_tbl1;
+\dRp+ testpub_mix
+ Publication testpub_mix
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Tables:
+ "public.testpub_tbl1"
+Sequences:
+ "public.testpub_seq1"
+
+ALTER PUBLICATION testpub_mix ADD ALL SEQUENCES IN SCHEMA pub_test, ALL TABLES IN SCHEMA pub_test;
+\dRp+ testpub_mix
+ Publication testpub_mix
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Tables:
+ "public.testpub_tbl1"
+Tables from schemas:
+ "pub_test"
+Sequences:
+ "public.testpub_seq1"
+Sequences from schemas:
+ "pub_test"
+
+ALTER PUBLICATION testpub_mix DROP ALL SEQUENCES IN SCHEMA pub_test;
+\dRp+ testpub_mix
+ Publication testpub_mix
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Tables:
+ "public.testpub_tbl1"
+Tables from schemas:
+ "pub_test"
+Sequences:
+ "public.testpub_seq1"
+
+ALTER PUBLICATION testpub_mix DROP ALL TABLES IN SCHEMA pub_test;
+\dRp+ testpub_mix
+ Publication testpub_mix
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Tables:
+ "public.testpub_tbl1"
+Sequences:
+ "public.testpub_seq1"
+
+DROP PUBLICATION testpub_mix;
+DROP SEQUENCE testpub_seq1;
+DROP SEQUENCE pub_test.testpub_seq2;
+-- make sure we replicate only the correct relation type
+CREATE SCHEMA pub_test1;
+CREATE SEQUENCE pub_test1.test_seq1;
+CREATE TABLE pub_test1.test_tbl1 (a int primary key, b int);
+CREATE SCHEMA pub_test2;
+CREATE SEQUENCE pub_test2.test_seq2;
+CREATE TABLE pub_test2.test_tbl2 (a int primary key, b int);
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_schemas;
+RESET client_min_messages;
+-- add tables from one schema, sequences from the other
+ALTER PUBLICATION testpub_schemas ADD ALL TABLES IN SCHEMA pub_test2;
+ALTER PUBLICATION testpub_schemas ADD ALL SEQUENCES IN SCHEMA pub_test1;
+\dRp+ testpub_schemas
+ Publication testpub_schemas
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Tables from schemas:
+ "pub_test2"
+Sequences from schemas:
+ "pub_test1"
+
+\dn+ pub_test1
+ List of schemas
+ Name | Owner | Access privileges | Description
+-----------+--------------------------+-------------------+-------------
+ pub_test1 | regress_publication_user | |
+Publications:
+ "testpub_schemas" (sequences)
+
+\dn+ pub_test2
+ List of schemas
+ Name | Owner | Access privileges | Description
+-----------+--------------------------+-------------------+-------------
+ pub_test2 | regress_publication_user | |
+Publications:
+ "testpub_schemas" (tables)
+
+\d+ pub_test1.test_seq1;
+ Sequence "pub_test1.test_seq1"
+ Type | Start | Minimum | Maximum | Increment | Cycles? | Cache
+--------+-------+---------+---------------------+-----------+---------+-------
+ bigint | 1 | 1 | 9223372036854775807 | 1 | no | 1
+Publications:
+ "testpub_schemas"
+
+\d+ pub_test1.test_tbl1;
+ Table "pub_test1.test_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+Indexes:
+ "test_tbl1_pkey" PRIMARY KEY, btree (a)
+
+\d+ pub_test2.test_seq2;
+ Sequence "pub_test2.test_seq2"
+ Type | Start | Minimum | Maximum | Increment | Cycles? | Cache
+--------+-------+---------+---------------------+-----------+---------+-------
+ bigint | 1 | 1 | 9223372036854775807 | 1 | no | 1
+
+\d+ pub_test2.test_tbl2;
+ Table "pub_test2.test_tbl2"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+Indexes:
+ "test_tbl2_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_schemas"
+
+-- add the other object type from each schema
+ALTER PUBLICATION testpub_schemas ADD ALL TABLES IN SCHEMA pub_test1;
+ALTER PUBLICATION testpub_schemas ADD ALL SEQUENCES IN SCHEMA pub_test2;
+\dRp+ testpub_schemas
+ Publication testpub_schemas
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Tables from schemas:
+ "pub_test1"
+ "pub_test2"
+Sequences from schemas:
+ "pub_test1"
+ "pub_test2"
+
+\dn+ pub_test1
+ List of schemas
+ Name | Owner | Access privileges | Description
+-----------+--------------------------+-------------------+-------------
+ pub_test1 | regress_publication_user | |
+Publications:
+ "testpub_schemas" (sequences)
+ "testpub_schemas" (tables)
+
+\dn+ pub_test2
+ List of schemas
+ Name | Owner | Access privileges | Description
+-----------+--------------------------+-------------------+-------------
+ pub_test2 | regress_publication_user | |
+Publications:
+ "testpub_schemas" (tables)
+ "testpub_schemas" (sequences)
+
+\d+ pub_test1.test_seq1;
+ Sequence "pub_test1.test_seq1"
+ Type | Start | Minimum | Maximum | Increment | Cycles? | Cache
+--------+-------+---------+---------------------+-----------+---------+-------
+ bigint | 1 | 1 | 9223372036854775807 | 1 | no | 1
+Publications:
+ "testpub_schemas"
+
+\d+ pub_test1.test_tbl1;
+ Table "pub_test1.test_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+Indexes:
+ "test_tbl1_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_schemas"
+
+\d+ pub_test2.test_seq2;
+ Sequence "pub_test2.test_seq2"
+ Type | Start | Minimum | Maximum | Increment | Cycles? | Cache
+--------+-------+---------+---------------------+-----------+---------+-------
+ bigint | 1 | 1 | 9223372036854775807 | 1 | no | 1
+Publications:
+ "testpub_schemas"
+
+\d+ pub_test2.test_tbl2;
+ Table "pub_test2.test_tbl2"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+Indexes:
+ "test_tbl2_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_schemas"
+
+-- now drop the object type added first
+ALTER PUBLICATION testpub_schemas DROP ALL TABLES IN SCHEMA pub_test2;
+ALTER PUBLICATION testpub_schemas DROP ALL SEQUENCES IN SCHEMA pub_test1;
+\dRp+ testpub_schemas
+ Publication testpub_schemas
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Tables from schemas:
+ "pub_test1"
+Sequences from schemas:
+ "pub_test2"
+
+\dn+ pub_test1
+ List of schemas
+ Name | Owner | Access privileges | Description
+-----------+--------------------------+-------------------+-------------
+ pub_test1 | regress_publication_user | |
+Publications:
+ "testpub_schemas" (tables)
+
+\dn+ pub_test2
+ List of schemas
+ Name | Owner | Access privileges | Description
+-----------+--------------------------+-------------------+-------------
+ pub_test2 | regress_publication_user | |
+Publications:
+ "testpub_schemas" (sequences)
+
+\d+ pub_test1.test_seq1;
+ Sequence "pub_test1.test_seq1"
+ Type | Start | Minimum | Maximum | Increment | Cycles? | Cache
+--------+-------+---------+---------------------+-----------+---------+-------
+ bigint | 1 | 1 | 9223372036854775807 | 1 | no | 1
+
+\d+ pub_test1.test_tbl1;
+ Table "pub_test1.test_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+Indexes:
+ "test_tbl1_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_schemas"
+
+\d+ pub_test2.test_seq2;
+ Sequence "pub_test2.test_seq2"
+ Type | Start | Minimum | Maximum | Increment | Cycles? | Cache
+--------+-------+---------+---------------------+-----------+---------+-------
+ bigint | 1 | 1 | 9223372036854775807 | 1 | no | 1
+Publications:
+ "testpub_schemas"
+
+\d+ pub_test2.test_tbl2;
+ Table "pub_test2.test_tbl2"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+Indexes:
+ "test_tbl2_pkey" PRIMARY KEY, btree (a)
+
+-- should fail (publication contains the whole schema)
+ALTER PUBLICATION testpub_schemas ADD TABLE pub_test1.test_tbl1;
+ERROR: cannot add relation "pub_test1.test_tbl1" to publication
+DETAIL: Table's schema "pub_test1" is already part of the publication or part of the specified schema list.
+ALTER PUBLICATION testpub_schemas ADD SEQUENCE pub_test2.test_seq2;
+ERROR: cannot add relation "pub_test2.test_seq2" to publication
+DETAIL: Sequence's schema "pub_test2" is already part of the publication or part of the specified schema list.
+-- should work (different schema)
+ALTER PUBLICATION testpub_schemas ADD TABLE pub_test2.test_tbl2;
+ALTER PUBLICATION testpub_schemas ADD SEQUENCE pub_test1.test_seq1;
+\dRp+ testpub_schemas
+ Publication testpub_schemas
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Tables:
+ "pub_test2.test_tbl2"
+Tables from schemas:
+ "pub_test1"
+Sequences:
+ "pub_test1.test_seq1"
+Sequences from schemas:
+ "pub_test2"
+
+\d+ pub_test1.test_seq1;
+ Sequence "pub_test1.test_seq1"
+ Type | Start | Minimum | Maximum | Increment | Cycles? | Cache
+--------+-------+---------+---------------------+-----------+---------+-------
+ bigint | 1 | 1 | 9223372036854775807 | 1 | no | 1
+Publications:
+ "testpub_schemas"
+
+\d+ pub_test1.test_tbl1;
+ Table "pub_test1.test_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+Indexes:
+ "test_tbl1_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_schemas"
+
+\d+ pub_test2.test_seq2;
+ Sequence "pub_test2.test_seq2"
+ Type | Start | Minimum | Maximum | Increment | Cycles? | Cache
+--------+-------+---------+---------------------+-----------+---------+-------
+ bigint | 1 | 1 | 9223372036854775807 | 1 | no | 1
+Publications:
+ "testpub_schemas"
+
+\d+ pub_test2.test_tbl2;
+ Table "pub_test2.test_tbl2"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+Indexes:
+ "test_tbl2_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_schemas"
+
+-- now drop the explicitly added objects again
+ALTER PUBLICATION testpub_schemas DROP TABLE pub_test2.test_tbl2;
+ALTER PUBLICATION testpub_schemas DROP SEQUENCE pub_test1.test_seq1;
+\dRp+ testpub_schemas
+ Publication testpub_schemas
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
+Tables from schemas:
+ "pub_test1"
+Sequences from schemas:
+ "pub_test2"
+
+\d+ pub_test1.test_seq1;
+ Sequence "pub_test1.test_seq1"
+ Type | Start | Minimum | Maximum | Increment | Cycles? | Cache
+--------+-------+---------+---------------------+-----------+---------+-------
+ bigint | 1 | 1 | 9223372036854775807 | 1 | no | 1
+
+\d+ pub_test1.test_tbl1;
+ Table "pub_test1.test_tbl1"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+Indexes:
+ "test_tbl1_pkey" PRIMARY KEY, btree (a)
+Publications:
+ "testpub_schemas"
+
+\d+ pub_test2.test_seq2;
+ Sequence "pub_test2.test_seq2"
+ Type | Start | Minimum | Maximum | Increment | Cycles? | Cache
+--------+-------+---------+---------------------+-----------+---------+-------
+ bigint | 1 | 1 | 9223372036854775807 | 1 | no | 1
+Publications:
+ "testpub_schemas"
+
+\d+ pub_test2.test_tbl2;
+ Table "pub_test2.test_tbl2"
+ Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
+--------+---------+-----------+----------+---------+---------+--------------+-------------
+ a | integer | | not null | | plain | |
+ b | integer | | | | plain | |
+Indexes:
+ "test_tbl2_pkey" PRIMARY KEY, btree (a)
+
+DROP PUBLICATION testpub_schemas;
+DROP TABLE pub_test1.test_tbl1, pub_test2.test_tbl2;
+DROP SEQUENCE pub_test1.test_seq1, pub_test2.test_seq2;
+DROP SCHEMA pub_test1, pub_test2;
-- Tests for partitioned tables
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_forparted;
@@ -207,10 +731,10 @@ UPDATE testpub_parted1 SET a = 1;
-- only parent is listed as being in publication, not the partition
ALTER PUBLICATION testpub_forparted ADD TABLE testpub_parted;
\dRp+ testpub_forparted
- Publication testpub_forparted
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub_forparted
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables:
"public.testpub_parted"
@@ -223,10 +747,10 @@ ALTER TABLE testpub_parted DETACH PARTITION testpub_parted1;
UPDATE testpub_parted1 SET a = 1;
ALTER PUBLICATION testpub_forparted SET (publish_via_partition_root = true);
\dRp+ testpub_forparted
- Publication testpub_forparted
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | t
+ Publication testpub_forparted
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | t
Tables:
"public.testpub_parted"
@@ -255,10 +779,10 @@ SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub5 FOR TABLE testpub_rf_tbl1, testpub_rf_tbl2 WHERE (c <> 'test' AND d < 5) WITH (publish = 'insert');
RESET client_min_messages;
\dRp+ testpub5
- Publication testpub5
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | f | f | f | f
+ Publication testpub5
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | f | f | f | f | f
Tables:
"public.testpub_rf_tbl1"
"public.testpub_rf_tbl2" WHERE ((c <> 'test'::text) AND (d < 5))
@@ -271,10 +795,10 @@ Tables:
ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl3 WHERE (e > 1000 AND e < 2000);
\dRp+ testpub5
- Publication testpub5
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | f | f | f | f
+ Publication testpub5
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | f | f | f | f | f
Tables:
"public.testpub_rf_tbl1"
"public.testpub_rf_tbl2" WHERE ((c <> 'test'::text) AND (d < 5))
@@ -290,10 +814,10 @@ Publications:
ALTER PUBLICATION testpub5 DROP TABLE testpub_rf_tbl2;
\dRp+ testpub5
- Publication testpub5
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | f | f | f | f
+ Publication testpub5
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | f | f | f | f | f
Tables:
"public.testpub_rf_tbl1"
"public.testpub_rf_tbl3" WHERE ((e > 1000) AND (e < 2000))
@@ -301,10 +825,10 @@ Tables:
-- remove testpub_rf_tbl1 and add testpub_rf_tbl3 again (another WHERE expression)
ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl3 WHERE (e > 300 AND e < 500);
\dRp+ testpub5
- Publication testpub5
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | f | f | f | f
+ Publication testpub5
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | f | f | f | f | f
Tables:
"public.testpub_rf_tbl3" WHERE ((e > 300) AND (e < 500))
@@ -337,10 +861,10 @@ SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_syntax1 FOR TABLE testpub_rf_tbl1, ONLY testpub_rf_tbl3 WHERE (e < 999) WITH (publish = 'insert');
RESET client_min_messages;
\dRp+ testpub_syntax1
- Publication testpub_syntax1
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | f | f | f | f
+ Publication testpub_syntax1
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | f | f | f | f | f
Tables:
"public.testpub_rf_tbl1"
"public.testpub_rf_tbl3" WHERE (e < 999)
@@ -350,10 +874,10 @@ SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_syntax2 FOR TABLE testpub_rf_tbl1, testpub_rf_schema1.testpub_rf_tbl5 WHERE (h < 999) WITH (publish = 'insert');
RESET client_min_messages;
\dRp+ testpub_syntax2
- Publication testpub_syntax2
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | f | f | f | f
+ Publication testpub_syntax2
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | f | f | f | f | f
Tables:
"public.testpub_rf_tbl1"
"testpub_rf_schema1.testpub_rf_tbl5" WHERE (h < 999)
@@ -672,10 +1196,10 @@ CREATE PUBLICATION testpub_table_ins WITH (publish = 'insert, truncate');
RESET client_min_messages;
ALTER PUBLICATION testpub_table_ins ADD TABLE testpub_tbl5 (a); -- ok
\dRp+ testpub_table_ins
- Publication testpub_table_ins
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | f | f | t | f
+ Publication testpub_table_ins
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | f | f | t | f | f
Tables:
"public.testpub_tbl5" (a)
@@ -817,10 +1341,10 @@ CREATE TABLE testpub_tbl_both_filters (a int, b int, c int, PRIMARY KEY (a,c));
ALTER TABLE testpub_tbl_both_filters REPLICA IDENTITY USING INDEX testpub_tbl_both_filters_pkey;
ALTER PUBLICATION testpub_both_filters ADD TABLE testpub_tbl_both_filters (a,c) WHERE (c != 1);
\dRp+ testpub_both_filters
- Publication testpub_both_filters
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub_both_filters
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables:
"public.testpub_tbl_both_filters" (a, c) WHERE (c <> 1)
@@ -1021,10 +1545,10 @@ ERROR: relation "testpub_tbl1" is already member of publication "testpub_fortbl
CREATE PUBLICATION testpub_fortbl FOR TABLE testpub_tbl1;
ERROR: publication "testpub_fortbl" already exists
\dRp+ testpub_fortbl
- Publication testpub_fortbl
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub_fortbl
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables:
"pub_test.testpub_nopk"
"public.testpub_tbl1"
@@ -1062,10 +1586,10 @@ Publications:
"testpub_fortbl"
\dRp+ testpub_default
- Publication testpub_default
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | f | f
+ Publication testpub_default
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | f | t | f
Tables:
"pub_test.testpub_nopk"
"public.testpub_tbl1"
@@ -1143,10 +1667,10 @@ REVOKE CREATE ON DATABASE regression FROM regress_publication_user2;
DROP TABLE testpub_parted;
DROP TABLE testpub_tbl1;
\dRp+ testpub_default
- Publication testpub_default
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | f | f
+ Publication testpub_default
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | f | t | f
(1 row)
-- fail - must be owner of publication
@@ -1156,20 +1680,20 @@ ERROR: must be owner of publication testpub_default
RESET ROLE;
ALTER PUBLICATION testpub_default RENAME TO testpub_foo;
\dRp testpub_foo
- List of publications
- Name | Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
--------------+--------------------------+------------+---------+---------+---------+-----------+----------
- testpub_foo | regress_publication_user | f | t | t | t | f | f
+ List of publications
+ Name | Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+-------------+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ testpub_foo | regress_publication_user | f | f | t | t | t | f | t | f
(1 row)
-- rename back to keep the rest simple
ALTER PUBLICATION testpub_foo RENAME TO testpub_default;
ALTER PUBLICATION testpub_default OWNER TO regress_publication_user2;
\dRp testpub_default
- List of publications
- Name | Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
------------------+---------------------------+------------+---------+---------+---------+-----------+----------
- testpub_default | regress_publication_user2 | f | t | t | t | f | f
+ List of publications
+ Name | Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+-----------------+---------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ testpub_default | regress_publication_user2 | f | f | t | t | t | f | t | f
(1 row)
-- adding schemas and tables
@@ -1185,19 +1709,19 @@ CREATE TABLE "CURRENT_SCHEMA"."CURRENT_SCHEMA"(id int);
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub1_forschema FOR ALL TABLES IN SCHEMA pub_test1;
\dRp+ testpub1_forschema
- Publication testpub1_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub1_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
CREATE PUBLICATION testpub2_forschema FOR ALL TABLES IN SCHEMA pub_test1, pub_test2, pub_test3;
\dRp+ testpub2_forschema
- Publication testpub2_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub2_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
"pub_test2"
@@ -1211,44 +1735,44 @@ CREATE PUBLICATION testpub6_forschema FOR ALL TABLES IN SCHEMA "CURRENT_SCHEMA",
CREATE PUBLICATION testpub_fortable FOR TABLE "CURRENT_SCHEMA"."CURRENT_SCHEMA";
RESET client_min_messages;
\dRp+ testpub3_forschema
- Publication testpub3_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub3_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"public"
\dRp+ testpub4_forschema
- Publication testpub4_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub4_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"CURRENT_SCHEMA"
\dRp+ testpub5_forschema
- Publication testpub5_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub5_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"CURRENT_SCHEMA"
"public"
\dRp+ testpub6_forschema
- Publication testpub6_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub6_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"CURRENT_SCHEMA"
"public"
\dRp+ testpub_fortable
- Publication testpub_fortable
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub_fortable
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables:
"CURRENT_SCHEMA.CURRENT_SCHEMA"
@@ -1282,10 +1806,10 @@ ERROR: schema "testpub_view" does not exist
-- dropping the schema should reflect the change in publication
DROP SCHEMA pub_test3;
\dRp+ testpub2_forschema
- Publication testpub2_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub2_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
"pub_test2"
@@ -1293,20 +1817,20 @@ Tables from schemas:
-- renaming the schema should reflect the change in publication
ALTER SCHEMA pub_test1 RENAME to pub_test1_renamed;
\dRp+ testpub2_forschema
- Publication testpub2_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub2_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1_renamed"
"pub_test2"
ALTER SCHEMA pub_test1_renamed RENAME to pub_test1;
\dRp+ testpub2_forschema
- Publication testpub2_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub2_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
"pub_test2"
@@ -1314,10 +1838,10 @@ Tables from schemas:
-- alter publication add schema
ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA pub_test2;
\dRp+ testpub1_forschema
- Publication testpub1_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub1_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
"pub_test2"
@@ -1326,10 +1850,10 @@ Tables from schemas:
ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA non_existent_schema;
ERROR: schema "non_existent_schema" does not exist
\dRp+ testpub1_forschema
- Publication testpub1_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub1_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
"pub_test2"
@@ -1338,10 +1862,10 @@ Tables from schemas:
ALTER PUBLICATION testpub1_forschema ADD ALL TABLES IN SCHEMA pub_test1;
ERROR: schema "pub_test1" is already member of publication "testpub1_forschema"
\dRp+ testpub1_forschema
- Publication testpub1_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub1_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
"pub_test2"
@@ -1349,10 +1873,10 @@ Tables from schemas:
-- alter publication drop schema
ALTER PUBLICATION testpub1_forschema DROP ALL TABLES IN SCHEMA pub_test2;
\dRp+ testpub1_forschema
- Publication testpub1_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub1_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
@@ -1360,10 +1884,10 @@ Tables from schemas:
ALTER PUBLICATION testpub1_forschema DROP ALL TABLES IN SCHEMA pub_test2;
ERROR: tables from schema "pub_test2" are not part of the publication
\dRp+ testpub1_forschema
- Publication testpub1_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub1_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
@@ -1371,29 +1895,29 @@ Tables from schemas:
ALTER PUBLICATION testpub1_forschema DROP ALL TABLES IN SCHEMA non_existent_schema;
ERROR: schema "non_existent_schema" does not exist
\dRp+ testpub1_forschema
- Publication testpub1_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub1_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
-- drop all schemas
ALTER PUBLICATION testpub1_forschema DROP ALL TABLES IN SCHEMA pub_test1;
\dRp+ testpub1_forschema
- Publication testpub1_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub1_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
(1 row)
-- alter publication set multiple schema
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test2;
\dRp+ testpub1_forschema
- Publication testpub1_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub1_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
"pub_test2"
@@ -1402,10 +1926,10 @@ Tables from schemas:
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA non_existent_schema;
ERROR: schema "non_existent_schema" does not exist
\dRp+ testpub1_forschema
- Publication testpub1_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub1_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
"pub_test2"
@@ -1414,10 +1938,10 @@ Tables from schemas:
-- removing the duplicate schemas
ALTER PUBLICATION testpub1_forschema SET ALL TABLES IN SCHEMA pub_test1, pub_test1;
\dRp+ testpub1_forschema
- Publication testpub1_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub1_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
@@ -1496,18 +2020,18 @@ SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub3_forschema;
RESET client_min_messages;
\dRp+ testpub3_forschema
- Publication testpub3_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub3_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
(1 row)
ALTER PUBLICATION testpub3_forschema SET ALL TABLES IN SCHEMA pub_test1;
\dRp+ testpub3_forschema
- Publication testpub3_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub3_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables from schemas:
"pub_test1"
@@ -1517,20 +2041,20 @@ CREATE PUBLICATION testpub_forschema_fortable FOR ALL TABLES IN SCHEMA pub_test1
CREATE PUBLICATION testpub_fortable_forschema FOR TABLE pub_test2.tbl1, ALL TABLES IN SCHEMA pub_test1;
RESET client_min_messages;
\dRp+ testpub_forschema_fortable
- Publication testpub_forschema_fortable
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub_forschema_fortable
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables:
"pub_test2.tbl1"
Tables from schemas:
"pub_test1"
\dRp+ testpub_fortable_forschema
- Publication testpub_fortable_forschema
- Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
---------------------------+------------+---------+---------+---------+-----------+----------
- regress_publication_user | f | t | t | t | t | f
+ Publication testpub_fortable_forschema
+ Owner | All tables | All sequences | Inserts | Updates | Deletes | Truncates | Sequences | Via root
+--------------------------+------------+---------------+---------+---------+---------+-----------+-----------+----------
+ regress_publication_user | f | f | t | t | t | t | t | f
Tables:
"pub_test2.tbl1"
Tables from schemas:
@@ -1574,40 +2098,85 @@ CREATE SCHEMA sch1;
CREATE SCHEMA sch2;
CREATE TABLE sch1.tbl1 (a int) PARTITION BY RANGE(a);
CREATE TABLE sch2.tbl1_part1 PARTITION OF sch1.tbl1 FOR VALUES FROM (1) to (10);
+CREATE SEQUENCE sch1.seq1;
+CREATE SEQUENCE sch2.seq2;
-- Schema publication that does not include the schema that has the parent table
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch2 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
+ALTER PUBLICATION pub ADD ALL SEQUENCES IN SCHEMA sch2;
SELECT * FROM pg_publication_tables;
pubname | schemaname | tablename
---------+------------+------------
pub | sch2 | tbl1_part1
(1 row)
+SELECT * FROM pg_publication_sequences;
+ pubname | schemaname | sequencename
+---------+------------+--------------
+ pub | sch2 | seq2
+(1 row)
+
DROP PUBLICATION pub;
-- Table publication that does not include the parent table
CREATE PUBLICATION pub FOR TABLE sch2.tbl1_part1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
+ALTER PUBLICATION pub ADD SEQUENCE sch2.seq2;
SELECT * FROM pg_publication_tables;
pubname | schemaname | tablename
---------+------------+------------
pub | sch2 | tbl1_part1
(1 row)
+SELECT * FROM pg_publication_sequences;
+ pubname | schemaname | sequencename
+---------+------------+--------------
+ pub | sch2 | seq2
+(1 row)
+
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
+ALTER PUBLICATION pub ADD SEQUENCE sch1.seq1;
SELECT * FROM pg_publication_tables;
pubname | schemaname | tablename
---------+------------+-----------
pub | sch1 | tbl1
(1 row)
+SELECT * FROM pg_publication_sequences;
+ pubname | schemaname | sequencename
+---------+------------+--------------
+ pub | sch1 | seq1
+ pub | sch2 | seq2
+(2 rows)
+
DROP PUBLICATION pub;
-- Schema publication that does not include the schema that has the parent table
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch2 WITH (PUBLISH_VIA_PARTITION_ROOT=0);
+ALTER PUBLICATION pub ADD SEQUENCE sch1.seq1;
SELECT * FROM pg_publication_tables;
pubname | schemaname | tablename
---------+------------+------------
pub | sch2 | tbl1_part1
(1 row)
+SELECT * FROM pg_publication_sequences;
+ pubname | schemaname | sequencename
+---------+------------+--------------
+ pub | sch1 | seq1
+(1 row)
+
+DROP PUBLICATION pub;
+-- Sequence publication
+CREATE PUBLICATION pub FOR SEQUENCE sch2.seq2;
+SELECT * FROM pg_publication_tables;
+ pubname | schemaname | tablename
+---------+------------+-----------
+(0 rows)
+
+SELECT * FROM pg_publication_sequences;
+ pubname | schemaname | sequencename
+---------+------------+--------------
+ pub | sch2 | seq2
+(1 row)
+
DROP PUBLICATION pub;
-- Table publication that does not include the parent table
CREATE PUBLICATION pub FOR TABLE sch2.tbl1_part1 WITH (PUBLISH_VIA_PARTITION_ROOT=0);
@@ -1617,14 +2186,26 @@ SELECT * FROM pg_publication_tables;
pub | sch2 | tbl1_part1
(1 row)
+SELECT * FROM pg_publication_sequences;
+ pubname | schemaname | sequencename
+---------+------------+--------------
+(0 rows)
+
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
+ALTER PUBLICATION pub ADD ALL SEQUENCES IN SCHEMA sch2;
SELECT * FROM pg_publication_tables;
pubname | schemaname | tablename
---------+------------+------------
pub | sch2 | tbl1_part1
(1 row)
+SELECT * FROM pg_publication_sequences;
+ pubname | schemaname | sequencename
+---------+------------+--------------
+ pub | sch2 | seq2
+(1 row)
+
DROP PUBLICATION pub;
DROP TABLE sch2.tbl1_part1;
DROP TABLE sch1.tbl1;
@@ -1640,9 +2221,81 @@ SELECT * FROM pg_publication_tables;
pub | sch1 | tbl1
(1 row)
+SELECT * FROM pg_publication_sequences;
+ pubname | schemaname | sequencename
+---------+------------+--------------
+(0 rows)
+
+DROP PUBLICATION pub;
+-- Schema publication
+CREATE PUBLICATION pub FOR SEQUENCE sch2.seq2;
+SELECT * FROM pg_publication_tables;
+ pubname | schemaname | tablename
+---------+------------+-----------
+(0 rows)
+
+SELECT * FROM pg_publication_sequences;
+ pubname | schemaname | sequencename
+---------+------------+--------------
+ pub | sch2 | seq2
+(1 row)
+
+DROP PUBLICATION pub;
+-- Sequence publication
+CREATE PUBLICATION pub FOR ALL SEQUENCES IN SCHEMA sch2;
+SELECT * FROM pg_publication_tables;
+ pubname | schemaname | tablename
+---------+------------+-----------
+(0 rows)
+
+SELECT * FROM pg_publication_sequences;
+ pubname | schemaname | sequencename
+---------+------------+--------------
+ pub | sch2 | seq2
+(1 row)
+
+ALTER PUBLICATION pub ADD SEQUENCE sch1.seq1;
+SELECT * FROM pg_publication_tables;
+ pubname | schemaname | tablename
+---------+------------+-----------
+(0 rows)
+
+SELECT * FROM pg_publication_sequences;
+ pubname | schemaname | sequencename
+---------+------------+--------------
+ pub | sch1 | seq1
+ pub | sch2 | seq2
+(2 rows)
+
+ALTER PUBLICATION pub DROP SEQUENCE sch1.seq1;
+SELECT * FROM pg_publication_tables;
+ pubname | schemaname | tablename
+---------+------------+-----------
+(0 rows)
+
+SELECT * FROM pg_publication_sequences;
+ pubname | schemaname | sequencename
+---------+------------+--------------
+ pub | sch2 | seq2
+(1 row)
+
+ALTER PUBLICATION pub ADD ALL SEQUENCES IN SCHEMA sch1;
+SELECT * FROM pg_publication_tables;
+ pubname | schemaname | tablename
+---------+------------+-----------
+(0 rows)
+
+SELECT * FROM pg_publication_sequences;
+ pubname | schemaname | sequencename
+---------+------------+--------------
+ pub | sch1 | seq1
+ pub | sch2 | seq2
+(2 rows)
+
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
+DROP SEQUENCE sch1.seq1, sch2.seq2;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
RESET SESSION AUTHORIZATION;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index f29375c2a90..db652ea8d89 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1435,6 +1435,14 @@ pg_prepared_xacts| SELECT p.transaction,
FROM ((pg_prepared_xact() p(transaction, gid, prepared, ownerid, dbid)
LEFT JOIN pg_authid u ON ((p.ownerid = u.oid)))
LEFT JOIN pg_database d ON ((p.dbid = d.oid)));
+pg_publication_sequences| SELECT p.pubname,
+ n.nspname AS schemaname,
+ c.relname AS sequencename
+ FROM pg_publication p,
+ LATERAL pg_get_publication_sequences((p.pubname)::text) gps(relid),
+ (pg_class c
+ JOIN pg_namespace n ON ((n.oid = c.relnamespace)))
+ WHERE (c.oid = gps.relid);
pg_publication_tables| SELECT p.pubname,
n.nspname AS schemaname,
c.relname AS tablename
diff --git a/src/test/regress/sql/object_address.sql b/src/test/regress/sql/object_address.sql
index acd0468a9d9..9d8323468d6 100644
--- a/src/test/regress/sql/object_address.sql
+++ b/src/test/regress/sql/object_address.sql
@@ -49,6 +49,7 @@ CREATE TRANSFORM FOR int LANGUAGE SQL (
SET client_min_messages = 'ERROR';
CREATE PUBLICATION addr_pub FOR TABLE addr_nsp.gentable;
CREATE PUBLICATION addr_pub_schema FOR ALL TABLES IN SCHEMA addr_nsp;
+CREATE PUBLICATION addr_pub_schema2 FOR ALL SEQUENCES IN SCHEMA addr_nsp;
RESET client_min_messages;
CREATE SUBSCRIPTION regress_addr_sub CONNECTION '' PUBLICATION bar WITH (connect = false, slot_name = NONE);
CREATE STATISTICS addr_nsp.gentable_stat ON a, b FROM addr_nsp.gentable;
@@ -198,7 +199,8 @@ WITH objects (type, name, args) AS (VALUES
('transform', '{int}', '{sql}'),
('access method', '{btree}', '{}'),
('publication', '{addr_pub}', '{}'),
- ('publication namespace', '{addr_nsp}', '{addr_pub_schema}'),
+ ('publication namespace', '{addr_nsp}', '{addr_pub_schema,t}'),
+ ('publication namespace', '{addr_nsp}', '{addr_pub_schema2,s}'),
('publication relation', '{addr_nsp, gentable}', '{addr_pub}'),
('subscription', '{regress_addr_sub}', '{}'),
('statistics object', '{addr_nsp, gentable_stat}', '{}')
@@ -218,6 +220,7 @@ SELECT (pg_identify_object(addr1.classid, addr1.objid, addr1.objsubid)).*,
DROP FOREIGN DATA WRAPPER addr_fdw CASCADE;
DROP PUBLICATION addr_pub;
DROP PUBLICATION addr_pub_schema;
+DROP PUBLICATION addr_pub_schema2;
DROP SUBSCRIPTION regress_addr_sub;
DROP SCHEMA addr_nsp CASCADE;
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 8539110025e..96b02947fa2 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -27,7 +27,7 @@ CREATE PUBLICATION testpub_xxx WITH (publish_via_partition_root = 'true', publis
\dRp
-ALTER PUBLICATION testpub_default SET (publish = 'insert, update, delete');
+ALTER PUBLICATION testpub_default SET (publish = 'insert, update, delete, sequence');
\dRp
@@ -46,6 +46,8 @@ ALTER PUBLICATION testpub_foralltables SET (publish = 'insert, update');
CREATE TABLE testpub_tbl2 (id serial primary key, data text);
-- fail - can't add to for all tables publication
ALTER PUBLICATION testpub_foralltables ADD TABLE testpub_tbl2;
+-- fail - can't add a table using ADD SEQUENCE command
+ALTER PUBLICATION testpub_foralltables ADD SEQUENCE testpub_tbl2;
-- fail - can't drop from all tables publication
ALTER PUBLICATION testpub_foralltables DROP TABLE testpub_tbl2;
-- fail - can't add to for all tables publication
@@ -104,6 +106,199 @@ RESET client_min_messages;
DROP TABLE testpub_tbl3, testpub_tbl3a;
DROP PUBLICATION testpub3, testpub4;
+--- adding sequences
+CREATE SEQUENCE testpub_seq0;
+CREATE SEQUENCE pub_test.testpub_seq1;
+
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_forallsequences FOR ALL SEQUENCES WITH (publish = 'sequence');
+RESET client_min_messages;
+ALTER PUBLICATION testpub_forallsequences SET (publish = 'insert, sequence');
+
+CREATE SEQUENCE testpub_seq2;
+-- fail - can't add to for all sequences publication
+ALTER PUBLICATION testpub_forallsequences ADD SEQUENCE testpub_seq2;
+-- fail - can't drop from all sequences publication
+ALTER PUBLICATION testpub_forallsequences DROP SEQUENCE testpub_seq2;
+-- fail - can't add to for all sequences publication
+ALTER PUBLICATION testpub_forallsequences SET SEQUENCE pub_test.testpub_seq1;
+
+-- fail - can't add schema to 'FOR ALL SEQUENCES' publication
+ALTER PUBLICATION testpub_forallsequences ADD ALL SEQUENCES IN SCHEMA pub_test;
+-- fail - can't drop schema from 'FOR ALL SEQUENCES' publication
+ALTER PUBLICATION testpub_forallsequences DROP ALL SEQUENCES IN SCHEMA pub_test;
+-- fail - can't set schema to 'FOR ALL SEQUENCES' publication
+ALTER PUBLICATION testpub_forallsequences SET ALL SEQUENCES IN SCHEMA pub_test;
+
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_forsequence FOR SEQUENCE testpub_seq0;
+RESET client_min_messages;
+-- should be able to add schema to 'FOR SEQUENCE' publication
+ALTER PUBLICATION testpub_forsequence ADD ALL SEQUENCES IN SCHEMA pub_test;
+\dRp+ testpub_forsequence
+-- fail - can't add sequence from the schema we already added
+ALTER PUBLICATION testpub_forsequence ADD SEQUENCE pub_test.testpub_seq1;
+-- fail - can't add sequence using ADD TABLE command
+ALTER PUBLICATION testpub_forsequence ADD TABLE pub_test.testpub_seq1;
+-- should be able to drop schema from 'FOR SEQUENCE' publication
+ALTER PUBLICATION testpub_forsequence DROP ALL SEQUENCES IN SCHEMA pub_test;
+\dRp+ testpub_forsequence
+-- should be able to set schema to 'FOR SEQUENCE' publication
+ALTER PUBLICATION testpub_forsequence SET ALL SEQUENCES IN SCHEMA pub_test;
+\dRp+ testpub_forsequence
+
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_forschema FOR ALL SEQUENCES IN SCHEMA pub_test;
+RESET client_min_messages;
+-- fail - can't create publication with schema and sequence of the same schema
+CREATE PUBLICATION testpub_for_seq_schema FOR ALL SEQUENCES IN SCHEMA pub_test, SEQUENCE pub_test.testpub_seq1;
+-- fail - can't add a sequence of the same schema to the schema publication
+ALTER PUBLICATION testpub_forschema ADD SEQUENCE pub_test.testpub_seq1;
+-- fail - can't drop a sequence from the schema publication which isn't in the
+-- publication
+ALTER PUBLICATION testpub_forschema DROP SEQUENCE pub_test.testpub_seq1;
+-- should be able to set sequence to schema publication
+ALTER PUBLICATION testpub_forschema SET SEQUENCE pub_test.testpub_seq1;
+\dRp+ testpub_forschema
+
+SELECT pubname, puballtables, puballsequences FROM pg_publication WHERE pubname = 'testpub_forallsequences';
+\d+ pub_test.testpub_seq1
+\dRp+ testpub_forallsequences
+DROP SEQUENCE testpub_seq0, pub_test.testpub_seq1, testpub_seq2;
+DROP PUBLICATION testpub_forallsequences, testpub_forsequence, testpub_forschema;
+
+
+-- publication testing multiple sequences at the same time
+CREATE SEQUENCE testpub_seq1;
+CREATE SEQUENCE testpub_seq2;
+
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_multi FOR SEQUENCE testpub_seq1, testpub_seq2;
+RESET client_min_messages;
+
+\dRp+ testpub_multi
+
+DROP PUBLICATION testpub_multi;
+DROP SEQUENCE testpub_seq1;
+DROP SEQUENCE testpub_seq2;
+
+
+-- Publication mixing tables and sequences
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_mix;
+RESET client_min_messages;
+
+CREATE SEQUENCE testpub_seq1;
+CREATE SEQUENCE pub_test.testpub_seq2;
+
+ALTER PUBLICATION testpub_mix ADD SEQUENCE testpub_seq1, TABLE testpub_tbl1;
+\dRp+ testpub_mix
+
+ALTER PUBLICATION testpub_mix ADD ALL SEQUENCES IN SCHEMA pub_test, ALL TABLES IN SCHEMA pub_test;
+\dRp+ testpub_mix
+
+ALTER PUBLICATION testpub_mix DROP ALL SEQUENCES IN SCHEMA pub_test;
+\dRp+ testpub_mix
+
+ALTER PUBLICATION testpub_mix DROP ALL TABLES IN SCHEMA pub_test;
+\dRp+ testpub_mix
+
+DROP PUBLICATION testpub_mix;
+DROP SEQUENCE testpub_seq1;
+DROP SEQUENCE pub_test.testpub_seq2;
+
+
+-- make sure we replicate only the correct relation type
+CREATE SCHEMA pub_test1;
+CREATE SEQUENCE pub_test1.test_seq1;
+CREATE TABLE pub_test1.test_tbl1 (a int primary key, b int);
+
+CREATE SCHEMA pub_test2;
+CREATE SEQUENCE pub_test2.test_seq2;
+CREATE TABLE pub_test2.test_tbl2 (a int primary key, b int);
+
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub_schemas;
+RESET client_min_messages;
+
+-- add tables from one schema, sequences from the other
+ALTER PUBLICATION testpub_schemas ADD ALL TABLES IN SCHEMA pub_test2;
+ALTER PUBLICATION testpub_schemas ADD ALL SEQUENCES IN SCHEMA pub_test1;
+
+\dRp+ testpub_schemas
+
+\dn+ pub_test1
+\dn+ pub_test2
+
+\d+ pub_test1.test_seq1;
+\d+ pub_test1.test_tbl1;
+
+\d+ pub_test2.test_seq2;
+\d+ pub_test2.test_tbl2;
+
+-- add the other object type from each schema
+ALTER PUBLICATION testpub_schemas ADD ALL TABLES IN SCHEMA pub_test1;
+ALTER PUBLICATION testpub_schemas ADD ALL SEQUENCES IN SCHEMA pub_test2;
+
+\dRp+ testpub_schemas
+
+\dn+ pub_test1
+\dn+ pub_test2
+
+\d+ pub_test1.test_seq1;
+\d+ pub_test1.test_tbl1;
+
+\d+ pub_test2.test_seq2;
+\d+ pub_test2.test_tbl2;
+
+-- now drop the object type added first
+ALTER PUBLICATION testpub_schemas DROP ALL TABLES IN SCHEMA pub_test2;
+ALTER PUBLICATION testpub_schemas DROP ALL SEQUENCES IN SCHEMA pub_test1;
+
+\dRp+ testpub_schemas
+
+\dn+ pub_test1
+\dn+ pub_test2
+
+\d+ pub_test1.test_seq1;
+\d+ pub_test1.test_tbl1;
+
+\d+ pub_test2.test_seq2;
+\d+ pub_test2.test_tbl2;
+
+-- should fail (publication contains the whole schema)
+ALTER PUBLICATION testpub_schemas ADD TABLE pub_test1.test_tbl1;
+ALTER PUBLICATION testpub_schemas ADD SEQUENCE pub_test2.test_seq2;
+
+-- should work (different schema)
+ALTER PUBLICATION testpub_schemas ADD TABLE pub_test2.test_tbl2;
+ALTER PUBLICATION testpub_schemas ADD SEQUENCE pub_test1.test_seq1;
+
+\dRp+ testpub_schemas
+
+\d+ pub_test1.test_seq1;
+\d+ pub_test1.test_tbl1;
+
+\d+ pub_test2.test_seq2;
+\d+ pub_test2.test_tbl2;
+
+-- now drop the explicitly added objects again
+ALTER PUBLICATION testpub_schemas DROP TABLE pub_test2.test_tbl2;
+ALTER PUBLICATION testpub_schemas DROP SEQUENCE pub_test1.test_seq1;
+
+\dRp+ testpub_schemas
+
+\d+ pub_test1.test_seq1;
+\d+ pub_test1.test_tbl1;
+
+\d+ pub_test2.test_seq2;
+\d+ pub_test2.test_tbl2;
+
+DROP PUBLICATION testpub_schemas;
+DROP TABLE pub_test1.test_tbl1, pub_test2.test_tbl2;
+DROP SEQUENCE pub_test1.test_seq1, pub_test2.test_seq2;
+DROP SCHEMA pub_test1, pub_test2;
+
-- Tests for partitioned tables
SET client_min_messages = 'ERROR';
CREATE PUBLICATION testpub_forparted;
@@ -1004,32 +1199,51 @@ CREATE SCHEMA sch1;
CREATE SCHEMA sch2;
CREATE TABLE sch1.tbl1 (a int) PARTITION BY RANGE(a);
CREATE TABLE sch2.tbl1_part1 PARTITION OF sch1.tbl1 FOR VALUES FROM (1) to (10);
+CREATE SEQUENCE sch1.seq1;
+CREATE SEQUENCE sch2.seq2;
-- Schema publication that does not include the schema that has the parent table
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch2 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
+ALTER PUBLICATION pub ADD ALL SEQUENCES IN SCHEMA sch2;
SELECT * FROM pg_publication_tables;
+SELECT * FROM pg_publication_sequences;
DROP PUBLICATION pub;
-- Table publication that does not include the parent table
CREATE PUBLICATION pub FOR TABLE sch2.tbl1_part1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
+ALTER PUBLICATION pub ADD SEQUENCE sch2.seq2;
SELECT * FROM pg_publication_tables;
+SELECT * FROM pg_publication_sequences;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
+ALTER PUBLICATION pub ADD SEQUENCE sch1.seq1;
SELECT * FROM pg_publication_tables;
+SELECT * FROM pg_publication_sequences;
DROP PUBLICATION pub;
-- Schema publication that does not include the schema that has the parent table
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch2 WITH (PUBLISH_VIA_PARTITION_ROOT=0);
+ALTER PUBLICATION pub ADD SEQUENCE sch1.seq1;
+SELECT * FROM pg_publication_tables;
+SELECT * FROM pg_publication_sequences;
+
+DROP PUBLICATION pub;
+-- Sequence publication
+CREATE PUBLICATION pub FOR SEQUENCE sch2.seq2;
SELECT * FROM pg_publication_tables;
+SELECT * FROM pg_publication_sequences;
DROP PUBLICATION pub;
-- Table publication that does not include the parent table
CREATE PUBLICATION pub FOR TABLE sch2.tbl1_part1 WITH (PUBLISH_VIA_PARTITION_ROOT=0);
SELECT * FROM pg_publication_tables;
+SELECT * FROM pg_publication_sequences;
-- Table publication that includes both the parent table and the child table
ALTER PUBLICATION pub ADD TABLE sch1.tbl1;
+ALTER PUBLICATION pub ADD ALL SEQUENCES IN SCHEMA sch2;
SELECT * FROM pg_publication_tables;
+SELECT * FROM pg_publication_sequences;
DROP PUBLICATION pub;
DROP TABLE sch2.tbl1_part1;
@@ -1042,10 +1256,36 @@ CREATE TABLE sch1.tbl1_part3 (a int) PARTITION BY RANGE(a);
ALTER TABLE sch1.tbl1 ATTACH PARTITION sch1.tbl1_part3 FOR VALUES FROM (20) to (30);
CREATE PUBLICATION pub FOR ALL TABLES IN SCHEMA sch1 WITH (PUBLISH_VIA_PARTITION_ROOT=1);
SELECT * FROM pg_publication_tables;
+SELECT * FROM pg_publication_sequences;
+
+DROP PUBLICATION pub;
+-- Schema publication
+CREATE PUBLICATION pub FOR SEQUENCE sch2.seq2;
+SELECT * FROM pg_publication_tables;
+SELECT * FROM pg_publication_sequences;
+
+DROP PUBLICATION pub;
+-- Sequence publication
+CREATE PUBLICATION pub FOR ALL SEQUENCES IN SCHEMA sch2;
+SELECT * FROM pg_publication_tables;
+SELECT * FROM pg_publication_sequences;
+
+ALTER PUBLICATION pub ADD SEQUENCE sch1.seq1;
+SELECT * FROM pg_publication_tables;
+SELECT * FROM pg_publication_sequences;
+
+ALTER PUBLICATION pub DROP SEQUENCE sch1.seq1;
+SELECT * FROM pg_publication_tables;
+SELECT * FROM pg_publication_sequences;
+
+ALTER PUBLICATION pub ADD ALL SEQUENCES IN SCHEMA sch1;
+SELECT * FROM pg_publication_tables;
+SELECT * FROM pg_publication_sequences;
RESET client_min_messages;
DROP PUBLICATION pub;
DROP TABLE sch1.tbl1;
+DROP SEQUENCE sch1.seq1, sch2.seq2;
DROP SCHEMA sch1 cascade;
DROP SCHEMA sch2 cascade;
diff --git a/src/test/subscription/t/030_sequences.pl b/src/test/subscription/t/030_sequences.pl
new file mode 100644
index 00000000000..9ae3c03d7d1
--- /dev/null
+++ b/src/test/subscription/t/030_sequences.pl
@@ -0,0 +1,202 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+# This tests that sequences are replicated correctly by logical replication
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+my $ddl = qq(
+ CREATE TABLE seq_test (v BIGINT);
+ CREATE SEQUENCE s;
+);
+
+# Setup structure on the publisher
+$node_publisher->safe_psql('postgres', $ddl);
+
+# Create some the same structure on subscriber, and an extra sequence that
+# we'll create on the publisher later
+$ddl = qq(
+ CREATE TABLE seq_test (v BIGINT);
+ CREATE SEQUENCE s;
+ CREATE SEQUENCE s2;
+);
+
+$node_subscriber->safe_psql('postgres', $ddl);
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION seq_pub");
+
+$node_publisher->safe_psql('postgres',
+ "ALTER PUBLICATION seq_pub ADD SEQUENCE s");
+
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION seq_sub CONNECTION '$publisher_connstr' PUBLICATION seq_pub"
+);
+
+$node_publisher->wait_for_catchup('seq_sub');
+
+# Wait for initial sync to finish as well
+my $synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('s', 'r');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Insert initial test data
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ -- generate a number of values using the sequence
+ INSERT INTO seq_test SELECT nextval('s') FROM generate_series(1,100);
+));
+
+$node_publisher->wait_for_catchup('seq_sub');
+
+# Check the data on subscriber
+my $result = $node_subscriber->safe_psql(
+ 'postgres', qq(
+ SELECT * FROM s;
+));
+
+is( $result, '132|0|t',
+ 'initial test data replicated');
+
+
+# advance the sequence in a rolled-back transaction - the rollback
+# does not wait for the replication, so we could see any intermediate state
+# so do something else after the test, to ensure we wait for everything
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ BEGIN;
+ INSERT INTO seq_test SELECT nextval('s') FROM generate_series(1,100);
+ ROLLBACK;
+ INSERT INTO seq_test VALUES (-1);
+));
+
+$node_publisher->wait_for_catchup('seq_sub');
+
+# Check the data on subscriber
+$result = $node_subscriber->safe_psql(
+ 'postgres', qq(
+ SELECT * FROM s;
+));
+
+is( $result, '231|0|t',
+ 'advance sequence in rolled-back transaction');
+
+
+# create a new sequence and roll it back - should not be replicated, due to
+# the transactional behavior
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ BEGIN;
+ CREATE SEQUENCE s2;
+ ALTER PUBLICATION seq_pub ADD SEQUENCE s2;
+ INSERT INTO seq_test SELECT nextval('s2') FROM generate_series(1,100);
+ ROLLBACK;
+));
+
+$node_publisher->wait_for_catchup('seq_sub');
+
+# Check the data on subscriber
+$result = $node_subscriber->safe_psql(
+ 'postgres', qq(
+ SELECT * FROM s2;
+));
+
+is( $result, '1|0|f',
+ 'create new sequence and roll it back');
+
+
+# create a new sequence, advance it in a rolled-back transaction, but commit
+# the create - the advance should be replicated nevertheless
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ BEGIN;
+ CREATE SEQUENCE s2;
+ ALTER PUBLICATION seq_pub ADD SEQUENCE s2;
+ SAVEPOINT sp1;
+ INSERT INTO seq_test SELECT nextval('s2') FROM generate_series(1,100);
+ ROLLBACK TO sp1;
+ COMMIT;
+));
+
+$node_publisher->wait_for_catchup('seq_sub');
+
+# Wait for sync of the second sequence we just added to finish
+$synced_query =
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('s', 'r');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check the data on subscriber
+$result = $node_subscriber->safe_psql(
+ 'postgres', qq(
+ SELECT * FROM s2;
+));
+
+is( $result, '132|0|t',
+ 'create sequence, advance it in rolled-back transaction, but commit the create');
+
+
+# advance the new sequence in a transaction, and roll it back - the rollback
+# does not wait for the replication, so we could see any intermediate state
+# so do something else after the test, to ensure we wait for everything
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ BEGIN;
+ INSERT INTO seq_test SELECT nextval('s2') FROM generate_series(1,100);
+ ROLLBACK;
+ INSERT INTO seq_test VALUES (-1);
+));
+
+$node_publisher->wait_for_catchup('seq_sub');
+
+# Check the data on subscriber
+$result = $node_subscriber->safe_psql(
+ 'postgres', qq(
+ SELECT * FROM s2;
+));
+
+is( $result, '231|0|t',
+ 'advance the new sequence in a transaction and roll it back');
+
+
+# advance the sequence in a subtransaction - the subtransaction gets rolled
+# back, but commit the main one - the changes should still be replicated
+$node_publisher->safe_psql(
+ 'postgres', qq(
+ BEGIN;
+ SAVEPOINT s1;
+ INSERT INTO seq_test SELECT nextval('s2') FROM generate_series(1,100);
+ ROLLBACK TO s1;
+ COMMIT;
+));
+
+$node_publisher->wait_for_catchup('seq_sub');
+
+# Check the data on subscriber
+$result = $node_subscriber->safe_psql(
+ 'postgres', qq(
+ SELECT * FROM s2;
+));
+
+is( $result, '330|0|t',
+ 'advance sequence in a subtransaction');
+
+
+done_testing();
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
@ 2022-08-07 00:36 ` Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
0 siblings, 1 reply; 33+ messages in thread
From: Noah Misch @ 2022-08-07 00:36 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Amit Kapila <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Apr 07, 2022 at 08:34:50PM +0200, Tomas Vondra wrote:
> I've pushed a revert af all the commits related to this - decoding of
> sequences and test_decoding / built-in replication changes.
Two July buildfarm runs failed with PANIC during standby promotion:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tern&dt=2022-07-19%2004%3A13%3A18
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tern&dt=2022-07-31%2011%3A33%3A13
The attached patch hacks things so an ordinary x86_64 GNU/Linux machine
reproduces this consistently. "git bisect" then traced the regression to the
above revert commit (2c7ea57e56ca5f668c32d4266e0a3e45b455bef5). The pg_ctl
test suite passes under this hack in all supported branches, and it passed on
v15 until that revert. Would you investigate?
The buildfarm animal uses keep_error_builds. From kept data directories, I
deduced these events:
- After the base backup, auto-analyze ran on the primary and wrote WAL.
- Standby streamed and wrote up to 0/301FFF.
- Standby received the promote signal. Terminated streaming. WAL page at 0/302000 remained all-zeros.
- Somehow, end-of-recovery became a PANIC.
Key portions from buildfarm logs:
=== good run standby2 log
2022-07-21 22:55:16.860 UTC [25034912:5] LOG: received promote request
2022-07-21 22:55:16.878 UTC [26804682:2] FATAL: terminating walreceiver process due to administrator command
2022-07-21 22:55:16.878 UTC [25034912:6] LOG: invalid record length at 0/3000060: wanted 24, got 0
2022-07-21 22:55:16.878 UTC [25034912:7] LOG: redo done at 0/3000028 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.42 s
2022-07-21 22:55:16.878 UTC [25034912:8] LOG: selected new timeline ID: 2
2022-07-21 22:55:17.004 UTC [25034912:9] LOG: archive recovery complete
2022-07-21 22:55:17.005 UTC [23724044:1] LOG: checkpoint starting: force
2022-07-21 22:55:17.008 UTC [14549364:4] LOG: database system is ready to accept connections
2022-07-21 22:55:17.093 UTC [23724044:2] LOG: checkpoint complete: wrote 3 buffers (2.3%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.019 s, sync=0.001 s, total=0.089 s; sync files=0, longest=0.000 s, average=0.000 s; distance=16384 kB, estimate=16384 kB
2022-07-21 22:55:17.143 UTC [27394418:1] [unknown] LOG: connection received: host=[local]
2022-07-21 22:55:17.144 UTC [27394418:2] [unknown] LOG: connection authorized: user=nm database=postgres application_name=003_promote.pl
2022-07-21 22:55:17.147 UTC [27394418:3] 003_promote.pl LOG: statement: SELECT pg_is_in_recovery()
2022-07-21 22:55:17.148 UTC [27394418:4] 003_promote.pl LOG: disconnection: session time: 0:00:00.005 user=nm database=postgres host=[local]
2022-07-21 22:55:58.301 UTC [14549364:5] LOG: received immediate shutdown request
2022-07-21 22:55:58.337 UTC [14549364:6] LOG: database system is shut down
=== failed run standby2 log, with my annotations
2022-07-19 05:28:22.136 UTC [7340406:5] LOG: received promote request
2022-07-19 05:28:22.163 UTC [8519860:2] FATAL: terminating walreceiver process due to administrator command
2022-07-19 05:28:22.166 UTC [7340406:6] LOG: invalid magic number 0000 in log segment 000000010000000000000003, offset 131072
New compared to the good run. XLOG_PAGE_MAGIC didn't match. This implies the WAL ended at a WAL page boundary.
2022-07-19 05:28:22.166 UTC [7340406:7] LOG: redo done at 0/301F168 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.18 s
2022-07-19 05:28:22.166 UTC [7340406:8] LOG: last completed transaction was at log time 2022-07-19 05:28:13.956716+00
New compared to the good run. The good run had no transactions to replay. The bad run replayed records from an auto-analyze.
2022-07-19 05:28:22.166 UTC [7340406:9] PANIC: invalid record length at 0/301F168: wanted 24, got 0
More WAL overall in bad run, due to auto-analyze. End of recovery wrongly considered a PANIC.
2022-07-19 05:28:22.583 UTC [8388800:4] LOG: startup process (PID 7340406) was terminated by signal 6: IOT/Abort trap
2022-07-19 05:28:22.584 UTC [8388800:5] LOG: terminating any other active server processes
2022-07-19 05:28:22.587 UTC [8388800:6] LOG: shutting down due to startup process failure
2022-07-19 05:28:22.627 UTC [8388800:7] LOG: database system is shut down
Let me know if I've left out details you want; I may be able to dig more out
of the buildfarm artifacts.
Author: Noah Misch <[email protected]>
Commit: Noah Misch <[email protected]>
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 8604fd4..6232237 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -885,6 +885,12 @@ XLogWalRcvWrite(char *buf, Size nbytes, XLogRecPtr recptr, TimeLineID tli)
{
int startoff;
int byteswritten;
+ bool spin_after = false;
+ /* force writing to end at a certain WAL page boundary */
+ if (recptr + nbytes > 0x03022000) {
+ nbytes = 0x03022000 - recptr;
+ spin_after = true;
+ }
Assert(tli != 0);
@@ -943,6 +949,8 @@ XLogWalRcvWrite(char *buf, Size nbytes, XLogRecPtr recptr, TimeLineID tli)
LogstreamResult.Write = recptr;
}
+ while (spin_after) /* we forced writing to end; wait for termination */
+ ProcessWalRcvInterrupts();
/* Update shared-memory status */
pg_atomic_write_u64(&WalRcv->writtenUpto, LogstreamResult.Write);
diff --git a/src/bin/pg_ctl/t/003_promote.pl b/src/bin/pg_ctl/t/003_promote.pl
index 84d28f4..57ae317 100644
--- a/src/bin/pg_ctl/t/003_promote.pl
+++ b/src/bin/pg_ctl/t/003_promote.pl
@@ -29,6 +29,7 @@ command_fails_like(
[ 'pg_ctl', '-D', $node_primary->data_dir, 'promote' ],
qr/not in standby mode/,
'pg_ctl promote of primary instance fails');
+if (1) { ok(1) for 1 .. 3; } else { # skip tests this way to avoid merge conflicts
my $node_standby = PostgreSQL::Test::Cluster->new('standby');
$node_primary->backup('my_backup');
@@ -46,12 +47,17 @@ ok( $node_standby->poll_query_until(
'postgres', 'SELECT NOT pg_is_in_recovery()'),
'promoted standby is not in recovery');
+}
+my $node_standby;
+$node_primary->backup('my_backup');
# same again with default wait option
$node_standby = PostgreSQL::Test::Cluster->new('standby2');
$node_standby->init_from_backup($node_primary, 'my_backup',
has_streaming => 1);
$node_standby->start;
+$node_primary->safe_psql('postgres', 'ANALYZE');
+
is($node_standby->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
't', 'standby is in recovery');
Attachments:
[text/plain] promote-invalid-magic-panic-v0.patch (2.1K, ../../[email protected]/2-promote-invalid-magic-panic-v0.patch)
download | inline diff:
Author: Noah Misch <[email protected]>
Commit: Noah Misch <[email protected]>
diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c
index 8604fd4..6232237 100644
--- a/src/backend/replication/walreceiver.c
+++ b/src/backend/replication/walreceiver.c
@@ -885,6 +885,12 @@ XLogWalRcvWrite(char *buf, Size nbytes, XLogRecPtr recptr, TimeLineID tli)
{
int startoff;
int byteswritten;
+ bool spin_after = false;
+ /* force writing to end at a certain WAL page boundary */
+ if (recptr + nbytes > 0x03022000) {
+ nbytes = 0x03022000 - recptr;
+ spin_after = true;
+ }
Assert(tli != 0);
@@ -943,6 +949,8 @@ XLogWalRcvWrite(char *buf, Size nbytes, XLogRecPtr recptr, TimeLineID tli)
LogstreamResult.Write = recptr;
}
+ while (spin_after) /* we forced writing to end; wait for termination */
+ ProcessWalRcvInterrupts();
/* Update shared-memory status */
pg_atomic_write_u64(&WalRcv->writtenUpto, LogstreamResult.Write);
diff --git a/src/bin/pg_ctl/t/003_promote.pl b/src/bin/pg_ctl/t/003_promote.pl
index 84d28f4..57ae317 100644
--- a/src/bin/pg_ctl/t/003_promote.pl
+++ b/src/bin/pg_ctl/t/003_promote.pl
@@ -29,6 +29,7 @@ command_fails_like(
[ 'pg_ctl', '-D', $node_primary->data_dir, 'promote' ],
qr/not in standby mode/,
'pg_ctl promote of primary instance fails');
+if (1) { ok(1) for 1 .. 3; } else { # skip tests this way to avoid merge conflicts
my $node_standby = PostgreSQL::Test::Cluster->new('standby');
$node_primary->backup('my_backup');
@@ -46,12 +47,17 @@ ok( $node_standby->poll_query_until(
'postgres', 'SELECT NOT pg_is_in_recovery()'),
'promoted standby is not in recovery');
+}
+my $node_standby;
+$node_primary->backup('my_backup');
# same again with default wait option
$node_standby = PostgreSQL::Test::Cluster->new('standby2');
$node_standby->init_from_backup($node_primary, 'my_backup',
has_streaming => 1);
$node_standby->start;
+$node_primary->safe_psql('postgres', 'ANALYZE');
+
is($node_standby->safe_psql('postgres', 'SELECT pg_is_in_recovery()'),
't', 'standby is in recovery');
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
@ 2022-08-07 13:18 ` Tomas Vondra <[email protected]>
2022-08-07 19:12 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
0 siblings, 1 reply; 33+ messages in thread
From: Tomas Vondra @ 2022-08-07 13:18 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Amit Kapila <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>; Thomas Munro <[email protected]>
On 8/7/22 02:36, Noah Misch wrote:
> On Thu, Apr 07, 2022 at 08:34:50PM +0200, Tomas Vondra wrote:
>> I've pushed a revert af all the commits related to this - decoding of
>> sequences and test_decoding / built-in replication changes.
>
> Two July buildfarm runs failed with PANIC during standby promotion:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tern&dt=2022-07-19%2004%3A13%3A18
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tern&dt=2022-07-31%2011%3A33%3A13
>
> The attached patch hacks things so an ordinary x86_64 GNU/Linux machine
> reproduces this consistently. "git bisect" then traced the regression to the
> above revert commit (2c7ea57e56ca5f668c32d4266e0a3e45b455bef5). The pg_ctl
> test suite passes under this hack in all supported branches, and it passed on
> v15 until that revert. Would you investigate?
>
> The buildfarm animal uses keep_error_builds. From kept data directories, I
> deduced these events:
>
> - After the base backup, auto-analyze ran on the primary and wrote WAL.
> - Standby streamed and wrote up to 0/301FFF.
> - Standby received the promote signal. Terminated streaming. WAL page at 0/302000 remained all-zeros.
> - Somehow, end-of-recovery became a PANIC.
>
I think it'd be really bizarre if this was due to the revert, as that
simply undoes minor WAL changes (and none of this should affect what
happens at WAL page boundary etc.). It just restores WAL as it was
before 0da92dc, nothing particularly complicated. I did go through all
of the changes again and I haven't spotted anything particularly
suspicious, but I'll give it another try tomorrow.
However, I did try bisecting this using the attached patch, and that
does not suggest the issue is in the revert commit. It actually fails
all the way back to 5dc0418fab2, and it starts working on 9553b4115f1.
...
6392f2a0968 Try to silence "-Wmissing-braces" complaints in ...
=> 5dc0418fab2 Prefetch data referenced by the WAL, take II.
9553b4115f1 Fix warning introduced in 5c279a6d350.
...
This is merely 10 commits before the revert, and it seems way more
related to WAL. Also, adding this to the two nodes in 003_standby.pl
makes the issue go away, it seems:
$node_standby->append_conf('postgresql.conf',
qq(recovery_prefetch = off));
I'd bet it's about WAL prefetching, not the revert, and the bisect was a
bit incorrect, because the commits are close and the failures happen to
be rare. (Presumably you first did the bisect and then wrote the patch
that reproduces this, right?)
Adding Thomas Munro to the thread, he's the WAL prefetching expert ;-)
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
@ 2022-08-07 19:12 ` Noah Misch <[email protected]>
2022-08-07 21:09 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
0 siblings, 1 reply; 33+ messages in thread
From: Noah Misch @ 2022-08-07 19:12 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Amit Kapila <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>; Thomas Munro <[email protected]>
On Sun, Aug 07, 2022 at 03:18:52PM +0200, Tomas Vondra wrote:
> On 8/7/22 02:36, Noah Misch wrote:
> > On Thu, Apr 07, 2022 at 08:34:50PM +0200, Tomas Vondra wrote:
> >> I've pushed a revert af all the commits related to this - decoding of
> >> sequences and test_decoding / built-in replication changes.
> >
> > Two July buildfarm runs failed with PANIC during standby promotion:
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tern&dt=2022-07-19%2004%3A13%3A18
> > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tern&dt=2022-07-31%2011%3A33%3A13
> >
> > The attached patch hacks things so an ordinary x86_64 GNU/Linux machine
> > reproduces this consistently. "git bisect" then traced the regression to the
> > above revert commit (2c7ea57e56ca5f668c32d4266e0a3e45b455bef5). The pg_ctl
> > test suite passes under this hack in all supported branches, and it passed on
> > v15 until that revert. Would you investigate?
> >
> > The buildfarm animal uses keep_error_builds. From kept data directories, I
> > deduced these events:
> >
> > - After the base backup, auto-analyze ran on the primary and wrote WAL.
> > - Standby streamed and wrote up to 0/301FFF.
> > - Standby received the promote signal. Terminated streaming. WAL page at 0/302000 remained all-zeros.
> > - Somehow, end-of-recovery became a PANIC.
>
> I think it'd be really bizarre if this was due to the revert, as that
> simply undoes minor WAL changes (and none of this should affect what
> happens at WAL page boundary etc.). It just restores WAL as it was
> before 0da92dc, nothing particularly complicated. I did go through all
> of the changes again and I haven't spotted anything particularly
> suspicious, but I'll give it another try tomorrow.
>
> However, I did try bisecting this using the attached patch, and that
> does not suggest the issue is in the revert commit. It actually fails
> all the way back to 5dc0418fab2, and it starts working on 9553b4115f1.
>
> ...
> 6392f2a0968 Try to silence "-Wmissing-braces" complaints in ...
> => 5dc0418fab2 Prefetch data referenced by the WAL, take II.
> 9553b4115f1 Fix warning introduced in 5c279a6d350.
> ...
>
> This is merely 10 commits before the revert, and it seems way more
> related to WAL. Also, adding this to the two nodes in 003_standby.pl
> makes the issue go away, it seems:
>
> $node_standby->append_conf('postgresql.conf',
> qq(recovery_prefetch = off));
>
> I'd bet it's about WAL prefetching, not the revert, and the bisect was a
> bit incorrect, because the commits are close and the failures happen to
> be rare. (Presumably you first did the bisect and then wrote the patch
> that reproduces this, right?)
No. I wrote the patch, then used the patch to drive the bisect. With ten
iterations, commit 2c7ea57 passes 0/10, while 2c7ea57^ passes 10/10. I've now
tried recovery_prefetch=off. With that, the test passes 10/10 at 2c7ea57.
Given your observation of a failure at 5dc0418fab2, I agree with your
conclusion. Whatever the role of 2c7ea57 in exposing the failure on my
machine, a root cause in WAL prefetching looks more likely.
> Adding Thomas Munro to the thread, he's the WAL prefetching expert ;-)
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 19:12 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
@ 2022-08-07 21:09 ` Thomas Munro <[email protected]>
2022-08-08 06:15 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
0 siblings, 1 reply; 33+ messages in thread
From: Thomas Munro @ 2022-08-07 21:09 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Aug 8, 2022 at 7:12 AM Noah Misch <[email protected]> wrote:
> On Sun, Aug 07, 2022 at 03:18:52PM +0200, Tomas Vondra wrote:
> > I'd bet it's about WAL prefetching, not the revert, and the bisect was a
> > bit incorrect, because the commits are close and the failures happen to
> > be rare. (Presumably you first did the bisect and then wrote the patch
> > that reproduces this, right?)
>
> No. I wrote the patch, then used the patch to drive the bisect. With ten
> iterations, commit 2c7ea57 passes 0/10, while 2c7ea57^ passes 10/10. I've now
> tried recovery_prefetch=off. With that, the test passes 10/10 at 2c7ea57.
> Given your observation of a failure at 5dc0418fab2, I agree with your
> conclusion. Whatever the role of 2c7ea57 in exposing the failure on my
> machine, a root cause in WAL prefetching looks more likely.
>
> > Adding Thomas Munro to the thread, he's the WAL prefetching expert ;-)
Thanks for the repro patch and bisection work. Looking...
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 19:12 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 21:09 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
@ 2022-08-08 06:15 ` Thomas Munro <[email protected]>
2022-08-08 08:33 ` Re: logical decoding and replication of sequences Kyotaro Horiguchi <[email protected]>
2022-08-23 04:21 ` Re: logical decoding and replication of sequences Michael Paquier <[email protected]>
0 siblings, 2 replies; 33+ messages in thread
From: Thomas Munro @ 2022-08-08 06:15 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Aug 8, 2022 at 9:09 AM Thomas Munro <[email protected]> wrote:
> Thanks for the repro patch and bisection work. Looking...
I don't have the complete explanation yet, but it's something like
this. We hit the following branch in xlogrecovery.c...
if (StandbyMode &&
!XLogReaderValidatePageHeader(xlogreader,
targetPagePtr, readBuf))
{
/*
* Emit this error right now then retry this page
immediately. Use
* errmsg_internal() because the message was already translated.
*/
if (xlogreader->errormsg_buf[0])
ereport(emode_for_corrupt_record(emode,
xlogreader->EndRecPtr),
(errmsg_internal("%s",
xlogreader->errormsg_buf)));
/* reset any error XLogReaderValidatePageHeader()
might have set */
xlogreader->errormsg_buf[0] = '\0';
goto next_record_is_invalid;
}
... but, even though there was a (suppressed) error, nothing
invalidates the reader's page buffer. Normally,
XLogReadValidatePageHeader() failure or any other kind of error
encountered by xlogreader.c'd decoding logic would do that, but here
the read_page callback is directly calling the header validation.
Without prefetching, that doesn't seem to matter, but reading ahead
can cause us to have the problem page in our buffer at the wrong time,
and then not re-read it when we should. Or something like that.
The attached patch that simply moves the cache invalidation into
report_invalid_record(), so that it's reached by the above code and
everything else that reports an error, seems to fix the problem in
src/bin/pg_ctl/t/003_promote.pl with Noah's spanner-in-the-works patch
applied, and passes check-world without it. I need to look at this
some more, though, and figure out if it's the right fix.
Attachments:
[text/x-patch] 0001-WIP-Fix-XLogPageRead-cache-invalidation.patch (1.6K, ../../CA+hUKGLptJc_k=cEqGtyRb-2pyF++uUMNi-+VEhrUV6pmezJmg@mail.gmail.com/2-0001-WIP-Fix-XLogPageRead-cache-invalidation.patch)
download | inline diff:
From 1f4fc659441a805a50b77f4e7810f9a25b5e62ae Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Mon, 8 Aug 2022 14:09:32 +1200
Subject: [PATCH] WIP Fix XLogPageRead() cache invalidation.
When ReadPageInternal() reports an error, including due to failure of
XLogReaderValidatePageHeader(), the reader calls
XLogReaderInvalReaderState() to invalidate the contents of its buffer,
forcing a re-read next time.
This did not work if a header validation error was detected by the
read_page callback itself. XLogPageRead() calls
XLogValidatePageHeader() directly, and if that failed, there was no
cache invalidation.
Fix, by invalidating the cache directly in report_invalid_record(), so
that XLogValidatePageHeader()'s failure invalidates the cache even with
called directly by xlogrecovery.c.
XXX More research/testing needed
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 06e91547dd..e6431722e7 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -81,6 +81,12 @@ report_invalid_record(XLogReaderState *state, const char *fmt,...)
va_end(args);
state->errormsg_deferred = true;
+
+ /*
+ * Invalidate the read state. We might read from a different source after
+ * failure.
+ */
+ XLogReaderInvalReadState(state);
}
/*
@@ -1066,11 +1072,6 @@ ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen)
return readLen;
err:
- if (state->errormsg_buf[0] != '\0')
- {
- state->errormsg_deferred = true;
- XLogReaderInvalReadState(state);
- }
return XLREAD_FAIL;
}
--
2.37.1
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 19:12 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 21:09 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 06:15 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
@ 2022-08-08 08:33 ` Kyotaro Horiguchi <[email protected]>
2022-08-08 08:56 ` Re: logical decoding and replication of sequences Kyotaro Horiguchi <[email protected]>
1 sibling, 1 reply; 33+ messages in thread
From: Kyotaro Horiguchi @ 2022-08-08 08:33 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
At Mon, 8 Aug 2022 18:15:46 +1200, Thomas Munro <[email protected]> wrote in
> On Mon, Aug 8, 2022 at 9:09 AM Thomas Munro <[email protected]> wrote:
> > Thanks for the repro patch and bisection work. Looking...
>
> I don't have the complete explanation yet, but it's something like
> this. We hit the following branch in xlogrecovery.c...
>
> if (StandbyMode &&
> !XLogReaderValidatePageHeader(xlogreader,
> targetPagePtr, readBuf))
> {
> /*
> * Emit this error right now then retry this page
> immediately. Use
> * errmsg_internal() because the message was already translated.
> */
> if (xlogreader->errormsg_buf[0])
> ereport(emode_for_corrupt_record(emode,
> xlogreader->EndRecPtr),
> (errmsg_internal("%s",
> xlogreader->errormsg_buf)));
>
> /* reset any error XLogReaderValidatePageHeader()
> might have set */
> xlogreader->errormsg_buf[0] = '\0';
> goto next_record_is_invalid;
> }
>
> ... but, even though there was a (suppressed) error, nothing
> invalidates the reader's page buffer. Normally,
> XLogReadValidatePageHeader() failure or any other kind of error
> encountered by xlogreader.c'd decoding logic would do that, but here
> the read_page callback is directly calling the header validation.
> Without prefetching, that doesn't seem to matter, but reading ahead
> can cause us to have the problem page in our buffer at the wrong time,
> and then not re-read it when we should. Or something like that.
>
> The attached patch that simply moves the cache invalidation into
> report_invalid_record(), so that it's reached by the above code and
> everything else that reports an error, seems to fix the problem in
> src/bin/pg_ctl/t/003_promote.pl with Noah's spanner-in-the-works patch
> applied, and passes check-world without it. I need to look at this
> some more, though, and figure out if it's the right fix.
If WaitForWALToBecomeAvailable returned by promotion, ReadPageInteral
misses the chance to inavlidate reader-state. That state is not an
error while in StandbyMode.
In the repro case, XLogPageRead returns XLREAD_WOULDBLOCK after the
first failure. This situation (of course) was not considered when
that code was introduced. If that function is going to return with
XLREAD_WOULDBLOCK while lastSourceFailed, it should be turned into
XLREAD_FAIL. So, the following also works.
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index 21088e78f6..9f242fe656 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -3220,7 +3220,9 @@ retry:
xlogreader->nonblocking))
{
case XLREAD_WOULDBLOCK:
- return XLREAD_WOULDBLOCK;
+ if (!lastSourceFailed)
+ return XLREAD_WOULDBLOCK;
+ /* Fall through. */
case XLREAD_FAIL:
if (readFile >= 0)
close(readFile);
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 19:12 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 21:09 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 06:15 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 08:33 ` Re: logical decoding and replication of sequences Kyotaro Horiguchi <[email protected]>
@ 2022-08-08 08:56 ` Kyotaro Horiguchi <[email protected]>
2022-08-29 10:21 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
0 siblings, 1 reply; 33+ messages in thread
From: Kyotaro Horiguchi @ 2022-08-08 08:56 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
At Mon, 08 Aug 2022 17:33:22 +0900 (JST), Kyotaro Horiguchi <[email protected]> wrote in
> If WaitForWALToBecomeAvailable returned by promotion, ReadPageInteral
> misses the chance to inavlidate reader-state. That state is not an
> error while in StandbyMode.
Mmm... Maybe I wanted to say: (Still I'm not sure the rewrite works..)
If WaitForWALToBecomeAvailable returned by promotion, ReadPageInteral
would miss the chance to invalidate reader-state. When XLogPageRead
is called in blocking mode while in StandbyMode (that is, the
traditional condition) , the function continues retrying until it
succeeds, or returns XLRAD_FAIL if promote is triggered. In other
words, it was not supposed to return non-failure while the header
validation is failing while in standby mode. But while in nonblocking
mode, the function can return non-failure with lastSourceFailed =
true, which seems wrong.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 19:12 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 21:09 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 06:15 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 08:33 ` Re: logical decoding and replication of sequences Kyotaro Horiguchi <[email protected]>
2022-08-08 08:56 ` Re: logical decoding and replication of sequences Kyotaro Horiguchi <[email protected]>
@ 2022-08-29 10:21 ` Thomas Munro <[email protected]>
2022-08-29 18:04 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
0 siblings, 1 reply; 33+ messages in thread
From: Thomas Munro @ 2022-08-29 10:21 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; +Cc: Noah Misch <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Aug 8, 2022 at 8:56 PM Kyotaro Horiguchi
<[email protected]> wrote:
> At Mon, 08 Aug 2022 17:33:22 +0900 (JST), Kyotaro Horiguchi <[email protected]> wrote in
> > If WaitForWALToBecomeAvailable returned by promotion, ReadPageInteral
> > misses the chance to inavlidate reader-state. That state is not an
> > error while in StandbyMode.
>
> Mmm... Maybe I wanted to say: (Still I'm not sure the rewrite works..)
>
> If WaitForWALToBecomeAvailable returned by promotion, ReadPageInteral
> would miss the chance to invalidate reader-state. When XLogPageRead
> is called in blocking mode while in StandbyMode (that is, the
> traditional condition) , the function continues retrying until it
> succeeds, or returns XLRAD_FAIL if promote is triggered. In other
> words, it was not supposed to return non-failure while the header
> validation is failing while in standby mode. But while in nonblocking
> mode, the function can return non-failure with lastSourceFailed =
> true, which seems wrong.
New ideas:
0001: Instead of figuring out when to invalidate the cache, let's
just invalidate it before every read attempt. It is only marked valid
after success (ie state->readLen > 0). No need to worry about error
cases.
0002: While here, I don't like xlogrecovery.c clobbering
xlogreader.c's internal error state, so I think we should have a
function for that with a documented purpose. It was also a little
inconsistent that it didn't clear a flag (but not buggy AFAICS; kinda
wondering if I should just get rid of that flag, but that's for
another day).
0003: Thinking about your comments above made me realise that I don't
really want XLogReadPage() to be internally retrying for obscure
failures while reading ahead. I think I prefer to give up on
prefetching as soon as anything tricky happens, and deal with
complexities once recovery catches up to that point. I am still
thinking about this point.
Here's the patch set I'm testing.
Attachments:
[text/x-patch] 0001-Fix-cache-invalidation-in-rare-recovery_prefetch-cas.patch (2.1K, ../../CA+hUKGJOnWuQqj42Q8xKfWN9NxGYUvEDEvEd_a+9Y9dF=78Oyw@mail.gmail.com/2-0001-Fix-cache-invalidation-in-rare-recovery_prefetch-cas.patch)
download | inline diff:
From 9f547088e54fcf279c74fbd03e4afaeffec69ea3 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Mon, 29 Aug 2022 19:56:27 +1200
Subject: [PATCH 1/3] Fix cache invalidation in rare recovery_prefetch case.
Since commit 0668719801 we've had an extra page validation and retry
loop inside XLogPageRead() to handle a rare special case. The new
recovery prefetch code from commit 5dc0418f could lead to a PANIC in
this case, because the WAL page cached internally by xlogreader.c was
not invalidated when it should have been.
Make tracking of the cached page's status more robust by invalidating it
just before we attempt any read. Because it is only marked valid when a
read succeeds, now we don't have to worry about invalidation in error
code paths.
The removed line that was setting errormsg_deferred was redundant
because it's already set by report_invalid_record().
Back-patch to 15.
Reported-by: Noah Misch <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20220807003627.GA4168930%40rfd.leadboat.com
---
src/backend/access/transam/xlogreader.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index f17e80948d..e883ade607 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -987,6 +987,13 @@ ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen)
targetPageOff == state->segoff && reqLen <= state->readLen)
return state->readLen;
+ /*
+ * Invalidate contents of internal buffer before read attempt. Just set
+ * the length to 0, rather than a full XLogReaderInvalReadState(), so we
+ * don't forget the segment we last read.
+ */
+ state->readLen = 0;
+
/*
* Data is not in our buffer.
*
@@ -1067,11 +1074,6 @@ ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen)
return readLen;
err:
- if (state->errormsg_buf[0] != '\0')
- {
- state->errormsg_deferred = true;
- XLogReaderInvalReadState(state);
- }
return XLREAD_FAIL;
}
--
2.30.2
[text/x-patch] 0002-Improve-xlogrecovery.c-xlogreader.c-boundary.patch (2.5K, ../../CA+hUKGJOnWuQqj42Q8xKfWN9NxGYUvEDEvEd_a+9Y9dF=78Oyw@mail.gmail.com/3-0002-Improve-xlogrecovery.c-xlogreader.c-boundary.patch)
download | inline diff:
From 72c55dfec6729e0f26a9fbf3ff184d3b4308a025 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Mon, 29 Aug 2022 21:08:25 +1200
Subject: [PATCH 2/3] Improve xlogrecovery.c/xlogreader.c boundary.
Create a function XLogReaderResetError() to clobber the error message
inside an XLogReaderState, instead of doing it directly from
xlogrecovery.c. This is older code from commit 0668719801, but commit
5dc0418f probably should have tidied this up because it leaves the
internal state a bit out of sync (not a live bug, but a little
confusing).
---
src/backend/access/transam/xlogreader.c | 10 ++++++++++
src/backend/access/transam/xlogrecovery.c | 2 +-
src/include/access/xlogreader.h | 3 +++
3 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index e883ade607..c100e18333 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1325,6 +1325,16 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr,
return true;
}
+/*
+ * Forget about an error produced by XLogReaderValidatePageHeader().
+ */
+void
+XLogReaderResetError(XLogReaderState *state)
+{
+ state->errormsg_buf[0] = '\0';
+ state->errormsg_deferred = false;
+}
+
/*
* Find the first record with an lsn >= RecPtr.
*
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index a59a0e826b..422322cf5a 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -3335,7 +3335,7 @@ retry:
(errmsg_internal("%s", xlogreader->errormsg_buf)));
/* reset any error XLogReaderValidatePageHeader() might have set */
- xlogreader->errormsg_buf[0] = '\0';
+ XLogReaderResetError(xlogreader);
goto next_record_is_invalid;
}
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 87ff00feb7..97b6f00114 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -373,6 +373,9 @@ extern DecodedXLogRecord *XLogReadAhead(XLogReaderState *state,
extern bool XLogReaderValidatePageHeader(XLogReaderState *state,
XLogRecPtr recptr, char *phdr);
+/* Forget error produced by XLogReaderValidatePageHeader(). */
+extern void XLogReaderResetError(XLogReaderState *state);
+
/*
* Error information from WALRead that both backend and frontend caller can
* process. Currently only errors from pread can be reported.
--
2.30.2
[text/x-patch] 0003-Don-t-retry-inside-XLogPageRead-if-reading-ahead.patch (1.2K, ../../CA+hUKGJOnWuQqj42Q8xKfWN9NxGYUvEDEvEd_a+9Y9dF=78Oyw@mail.gmail.com/4-0003-Don-t-retry-inside-XLogPageRead-if-reading-ahead.patch)
download | inline diff:
From bb8a6d52ca91d463993c3e01a48fc776297b0aeb Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Mon, 29 Aug 2022 21:14:25 +1200
Subject: [PATCH 3/3] Don't retry inside XLogPageRead() if reading ahead.
Give up immediately if we have a short read or a page header
invalidation failure inside XLogPageRead(). Once recovery catches up to
this point, it can deal with it. This doesn't fix any known live bug,
but it fits with the general philosophy of giving up fast when
prefetching runs into trouble.
---
src/backend/access/transam/xlogrecovery.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index 422322cf5a..76ce9704ca 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -3342,6 +3342,14 @@ retry:
return readLen;
next_record_is_invalid:
+ /*
+ * If we're reading ahead, give up fast. Standby retries and error
+ * reporting will be handled by a later read when recovery catches up to
+ * this point.
+ */
+ if (xlogreader->nonblocking)
+ return XLREAD_WOULDBLOCK;
+
lastSourceFailed = true;
if (readFile >= 0)
--
2.30.2
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 19:12 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 21:09 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 06:15 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 08:33 ` Re: logical decoding and replication of sequences Kyotaro Horiguchi <[email protected]>
2022-08-08 08:56 ` Re: logical decoding and replication of sequences Kyotaro Horiguchi <[email protected]>
2022-08-29 10:21 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
@ 2022-08-29 18:04 ` Tomas Vondra <[email protected]>
2022-08-29 21:42 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
0 siblings, 1 reply; 33+ messages in thread
From: Tomas Vondra @ 2022-08-29 18:04 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; Kyotaro Horiguchi <[email protected]>; +Cc: Noah Misch <[email protected]>; Amit Kapila <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 8/29/22 12:21, Thomas Munro wrote:
> On Mon, Aug 8, 2022 at 8:56 PM Kyotaro Horiguchi
> <[email protected]> wrote:
>> At Mon, 08 Aug 2022 17:33:22 +0900 (JST), Kyotaro Horiguchi <[email protected]> wrote in
>>> If WaitForWALToBecomeAvailable returned by promotion, ReadPageInteral
>>> misses the chance to inavlidate reader-state. That state is not an
>>> error while in StandbyMode.
>>
>> Mmm... Maybe I wanted to say: (Still I'm not sure the rewrite works..)
>>
>> If WaitForWALToBecomeAvailable returned by promotion, ReadPageInteral
>> would miss the chance to invalidate reader-state. When XLogPageRead
>> is called in blocking mode while in StandbyMode (that is, the
>> traditional condition) , the function continues retrying until it
>> succeeds, or returns XLRAD_FAIL if promote is triggered. In other
>> words, it was not supposed to return non-failure while the header
>> validation is failing while in standby mode. But while in nonblocking
>> mode, the function can return non-failure with lastSourceFailed =
>> true, which seems wrong.
>
> New ideas:
>
> 0001: Instead of figuring out when to invalidate the cache, let's
> just invalidate it before every read attempt. It is only marked valid
> after success (ie state->readLen > 0). No need to worry about error
> cases.
>
Maybe I misunderstand how all this works, but won't this have a really
bad performance impact. If not, why do we need the cache at all?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 19:12 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 21:09 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 06:15 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 08:33 ` Re: logical decoding and replication of sequences Kyotaro Horiguchi <[email protected]>
2022-08-08 08:56 ` Re: logical decoding and replication of sequences Kyotaro Horiguchi <[email protected]>
2022-08-29 10:21 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-29 18:04 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
@ 2022-08-29 21:42 ` Thomas Munro <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Thomas Munro @ 2022-08-29 21:42 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Kyotaro Horiguchi <[email protected]>; Noah Misch <[email protected]>; Amit Kapila <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Aug 30, 2022 at 6:04 AM Tomas Vondra
<[email protected]> wrote:
> On 8/29/22 12:21, Thomas Munro wrote:
> > 0001: Instead of figuring out when to invalidate the cache, let's
> > just invalidate it before every read attempt. It is only marked valid
> > after success (ie state->readLen > 0). No need to worry about error
> > cases.
>
> Maybe I misunderstand how all this works, but won't this have a really
> bad performance impact. If not, why do we need the cache at all?
It's a bit confusing because there are several levels of "read". The
cache remains valid as long as the caller of ReadPageInternal() keeps
asking for data that is in range (see early return after comment "/*
check whether we have all the requested data already */"). As soon as
the caller asks for something not in range, this patch marks the cache
invalid before calling the page_read() callback (= XLogPageRead()).
It is only marked valid again after that succeeds. Here's a new
version with no code change, just a better commit message to try to
explain that more clearly.
Attachments:
[text/x-patch] v2-0001-Fix-cache-invalidation-in-rare-recovery_prefetch-.patch (2.5K, ../../CA+hUKG+7z9L-xHwbZWvWkZcrn0KFdtfV5w3LsPdSJN1TrV9N6Q@mail.gmail.com/2-v2-0001-Fix-cache-invalidation-in-rare-recovery_prefetch-.patch)
download | inline diff:
From d02d851ab515d4a1f883f657bc43a9cae600a5d8 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Mon, 29 Aug 2022 19:56:27 +1200
Subject: [PATCH v2 1/3] Fix cache invalidation in rare recovery_prefetch
cases.
XLogPageRead() can retry internally in rare cases after a read has
succeeded, overwriting the page buffer. Namely, short reads, and page
validation failures while in standby mode (see commit 0668719801). Due
to an oversight in commit 3f1ce973, these cases could leave stale data
in the internal cache of xlogreader.c without marking it invalid. The
main defense against stale cached data was in the error handling path of
ReadPageInternal(), but that wasn't quite enough for errors handled
internally by XLogPageRead()'s retry loop.
Instead of trying to invalidate the cache in the relevant failure cases,
ReadPageInternal() now invalidates it before even calling the page_read
callback (ie XLogPageRead()). It is only marked valid by setting
state->readLen etc after page_read() succeeds. It remains valid until
the caller of ReadPageInternal() eventually asks for something out of
range and we have to go back to page_read() for more.
The removed line that was setting errormsg_deferred was redundant
because it's already set by report_invalid_record().
Back-patch to 15.
Reported-by: Noah Misch <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/20220807003627.GA4168930%40rfd.leadboat.com
---
src/backend/access/transam/xlogreader.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index f17e80948d..e883ade607 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -987,6 +987,13 @@ ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen)
targetPageOff == state->segoff && reqLen <= state->readLen)
return state->readLen;
+ /*
+ * Invalidate contents of internal buffer before read attempt. Just set
+ * the length to 0, rather than a full XLogReaderInvalReadState(), so we
+ * don't forget the segment we last read.
+ */
+ state->readLen = 0;
+
/*
* Data is not in our buffer.
*
@@ -1067,11 +1074,6 @@ ReadPageInternal(XLogReaderState *state, XLogRecPtr pageptr, int reqLen)
return readLen;
err:
- if (state->errormsg_buf[0] != '\0')
- {
- state->errormsg_deferred = true;
- XLogReaderInvalReadState(state);
- }
return XLREAD_FAIL;
}
--
2.30.2
[text/x-patch] v2-0002-Improve-xlogrecovery.c-xlogreader.c-boundary.patch (2.5K, ../../CA+hUKG+7z9L-xHwbZWvWkZcrn0KFdtfV5w3LsPdSJN1TrV9N6Q@mail.gmail.com/3-v2-0002-Improve-xlogrecovery.c-xlogreader.c-boundary.patch)
download | inline diff:
From 6439f5ed5b9cb6524c307f06a43c9c54f51d3b86 Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Mon, 29 Aug 2022 21:08:25 +1200
Subject: [PATCH v2 2/3] Improve xlogrecovery.c/xlogreader.c boundary.
Create a function XLogReaderResetError() to clobber the error message
inside an XLogReaderState, instead of doing it directly from
xlogrecovery.c. This is older code from commit 0668719801, but commit
5dc0418f probably should have tidied this up because it leaves the
internal state a bit out of sync (not a live bug, but a little
confusing).
---
src/backend/access/transam/xlogreader.c | 10 ++++++++++
src/backend/access/transam/xlogrecovery.c | 2 +-
src/include/access/xlogreader.h | 3 +++
3 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index e883ade607..c100e18333 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1325,6 +1325,16 @@ XLogReaderValidatePageHeader(XLogReaderState *state, XLogRecPtr recptr,
return true;
}
+/*
+ * Forget about an error produced by XLogReaderValidatePageHeader().
+ */
+void
+XLogReaderResetError(XLogReaderState *state)
+{
+ state->errormsg_buf[0] = '\0';
+ state->errormsg_deferred = false;
+}
+
/*
* Find the first record with an lsn >= RecPtr.
*
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index a59a0e826b..422322cf5a 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -3335,7 +3335,7 @@ retry:
(errmsg_internal("%s", xlogreader->errormsg_buf)));
/* reset any error XLogReaderValidatePageHeader() might have set */
- xlogreader->errormsg_buf[0] = '\0';
+ XLogReaderResetError(xlogreader);
goto next_record_is_invalid;
}
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 87ff00feb7..97b6f00114 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -373,6 +373,9 @@ extern DecodedXLogRecord *XLogReadAhead(XLogReaderState *state,
extern bool XLogReaderValidatePageHeader(XLogReaderState *state,
XLogRecPtr recptr, char *phdr);
+/* Forget error produced by XLogReaderValidatePageHeader(). */
+extern void XLogReaderResetError(XLogReaderState *state);
+
/*
* Error information from WALRead that both backend and frontend caller can
* process. Currently only errors from pread can be reported.
--
2.30.2
[text/x-patch] v2-0003-Don-t-retry-inside-XLogPageRead-if-reading-ahead.patch (1.2K, ../../CA+hUKG+7z9L-xHwbZWvWkZcrn0KFdtfV5w3LsPdSJN1TrV9N6Q@mail.gmail.com/4-v2-0003-Don-t-retry-inside-XLogPageRead-if-reading-ahead.patch)
download | inline diff:
From 394d2b08a896a7e3094ffb052adaa11486b7b14f Mon Sep 17 00:00:00 2001
From: Thomas Munro <[email protected]>
Date: Mon, 29 Aug 2022 21:14:25 +1200
Subject: [PATCH v2 3/3] Don't retry inside XLogPageRead() if reading ahead.
Give up immediately if we have a short read or a page header
invalidation failure inside XLogPageRead(). Once recovery catches up to
this point, it can deal with it. This doesn't fix any known live bug,
but it fits with the general philosophy of giving up fast when
prefetching runs into trouble.
---
src/backend/access/transam/xlogrecovery.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c
index 422322cf5a..76ce9704ca 100644
--- a/src/backend/access/transam/xlogrecovery.c
+++ b/src/backend/access/transam/xlogrecovery.c
@@ -3342,6 +3342,14 @@ retry:
return readLen;
next_record_is_invalid:
+ /*
+ * If we're reading ahead, give up fast. Standby retries and error
+ * reporting will be handled by a later read when recovery catches up to
+ * this point.
+ */
+ if (xlogreader->nonblocking)
+ return XLREAD_WOULDBLOCK;
+
lastSourceFailed = true;
if (readFile >= 0)
--
2.30.2
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 19:12 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 21:09 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 06:15 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
@ 2022-08-23 04:21 ` Michael Paquier <[email protected]>
2022-08-23 04:32 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
1 sibling, 1 reply; 33+ messages in thread
From: Michael Paquier @ 2022-08-23 04:21 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Noah Misch <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Aug 08, 2022 at 06:15:46PM +1200, Thomas Munro wrote:
> The attached patch that simply moves the cache invalidation into
> report_invalid_record(), so that it's reached by the above code and
> everything else that reports an error, seems to fix the problem in
> src/bin/pg_ctl/t/003_promote.pl with Noah's spanner-in-the-works patch
> applied, and passes check-world without it. I need to look at this
> some more, though, and figure out if it's the right fix.
Thomas, where are you on this open item? A potential PANIC at
promotion is bad. One possible exit path would be to switch the
default of recovery_prefetch, though that's a kind of last-resort
option seen from here.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 19:12 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 21:09 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 06:15 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-23 04:21 ` Re: logical decoding and replication of sequences Michael Paquier <[email protected]>
@ 2022-08-23 04:32 ` Thomas Munro <[email protected]>
2022-08-23 15:04 ` Re: logical decoding and replication of sequences Robert Haas <[email protected]>
0 siblings, 1 reply; 33+ messages in thread
From: Thomas Munro @ 2022-08-23 04:32 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Noah Misch <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Aug 23, 2022 at 4:21 PM Michael Paquier <[email protected]> wrote:
> On Mon, Aug 08, 2022 at 06:15:46PM +1200, Thomas Munro wrote:
> > The attached patch that simply moves the cache invalidation into
> > report_invalid_record(), so that it's reached by the above code and
> > everything else that reports an error, seems to fix the problem in
> > src/bin/pg_ctl/t/003_promote.pl with Noah's spanner-in-the-works patch
> > applied, and passes check-world without it. I need to look at this
> > some more, though, and figure out if it's the right fix.
>
> Thomas, where are you on this open item? A potential PANIC at
> promotion is bad. One possible exit path would be to switch the
> default of recovery_prefetch, though that's a kind of last-resort
> option seen from here.
I will get a fix committed this week -- I need to study
Horiguchi-san's analysis...
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 19:12 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 21:09 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 06:15 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-23 04:21 ` Re: logical decoding and replication of sequences Michael Paquier <[email protected]>
2022-08-23 04:32 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
@ 2022-08-23 15:04 ` Robert Haas <[email protected]>
2022-08-24 00:51 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
0 siblings, 1 reply; 33+ messages in thread
From: Robert Haas @ 2022-08-23 15:04 UTC (permalink / raw)
To: Thomas Munro <[email protected]>; +Cc: Michael Paquier <[email protected]>; Noah Misch <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Aug 23, 2022 at 12:33 AM Thomas Munro <[email protected]> wrote:
> On Tue, Aug 23, 2022 at 4:21 PM Michael Paquier <[email protected]> wrote:
> > On Mon, Aug 08, 2022 at 06:15:46PM +1200, Thomas Munro wrote:
> > > The attached patch that simply moves the cache invalidation into
> > > report_invalid_record(), so that it's reached by the above code and
> > > everything else that reports an error, seems to fix the problem in
> > > src/bin/pg_ctl/t/003_promote.pl with Noah's spanner-in-the-works patch
> > > applied, and passes check-world without it. I need to look at this
> > > some more, though, and figure out if it's the right fix.
> >
> > Thomas, where are you on this open item? A potential PANIC at
> > promotion is bad. One possible exit path would be to switch the
> > default of recovery_prefetch, though that's a kind of last-resort
> > option seen from here.
>
> I will get a fix committed this week -- I need to study
> Horiguchi-san's analysis...
Hi!
I haven't been paying attention to this thread, but my attention was
just drawn to it, and I'm wondering if the issue you're trying to
track down here is actually the same as what I reported yesterday
here:
https://www.postgresql.org/message-id/[email protected]...
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 19:12 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 21:09 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 06:15 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-23 04:21 ` Re: logical decoding and replication of sequences Michael Paquier <[email protected]>
2022-08-23 04:32 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-23 15:04 ` Re: logical decoding and replication of sequences Robert Haas <[email protected]>
@ 2022-08-24 00:51 ` Thomas Munro <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Thomas Munro @ 2022-08-24 00:51 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Michael Paquier <[email protected]>; Noah Misch <[email protected]>; Tomas Vondra <[email protected]>; Amit Kapila <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Aug 24, 2022 at 3:04 AM Robert Haas <[email protected]> wrote:
> I haven't been paying attention to this thread, but my attention was
> just drawn to it, and I'm wondering if the issue you're trying to
> track down here is actually the same as what I reported yesterday
> here:
>
> https://www.postgresql.org/message-id/[email protected]...
Summarising a chat we had about this: Different bug, similar
ingredients. Robert describes a screw-up in what is written, but here
we're talking about a cache invalidation bug while reading.
^ permalink raw reply [nested|flat] 33+ messages in thread
* Re: logical decoding and replication of sequences
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
@ 2022-05-25 20:42 ` Zheng Li <[email protected]>
1 sibling, 0 replies; 33+ messages in thread
From: Zheng Li @ 2022-05-25 20:42 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Amit Kapila <[email protected]>; Petr Jelinek <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
> But of course, if we expect/require to have a perfect snapshot for that
> exact position in the transaction, this won't work. IMO the whole idea
> that we can have non-transactional bits in naturally transactional
> decoding seems a bit suspicious (at least in hindsight).
>
> No matter what we do for sequences, though, this still affects logical
> messages too. Not sure what to do there :-(
Hi, I spent some time trying to understand this problem while I was
evaluating its impact on the DDL replication in [1]. I think for DDL
we could always remove the
non-transactional bits since DDL will probably always be processed
transactionally.
I attempted to solve the problem for messages. Here is a potential
solution by keeping track of
the last decoded/acked non-transactional message/operation lsn and use
it to check if a non-transactional message record should be skipped
during decoding,
to do that I added new fields
ReplicationSlotPersistentData.non_xact_op_at,
XLogReaderState.NonXactOpRecPtr and
SnapBuild.start_decoding_nonxactop_at.
This is the end LSN of the last non-transactional message/operation
decoded/acked. I verified this approach solves the issue of
missing decoding of non-transactional messages under
concurrency/before the builder state reaches SNAPBUILD_CONSISTENT.
Once
the builder state reach SNAPBUILD_CONSISTENT, the new field
ReplicationSlotPersistentData.non_xact_op_at can be set
to ReplicationSlotPersistentData.confirmed_flush.
Similar to the sequence issue, here is the test case for logical messages:
Test concurrent execution in 3 sessions that allows pg_logical_emit_message in
session-2 to happen before we reach a consistent point and commit
happens after a consistent point:
Session-2:
Begin;
SELECT pg_current_xact_id();
Session-1:
SELECT 'init' FROM pg_create_logical_replication_slot('test_slot',
'test_decoding', false, true);
Session-3:
Begin;
SELECT pg_current_xact_id();
Session-2:
Commit;
Begin;
SELECT pg_logical_emit_message(true, 'test_decoding', 'msg1');
SELECT pg_logical_emit_message(false, 'test_decoding', 'msg2');
Session-3:
Commit;
Session-1: (at this point, the session will crash without the fix)
SELECT data FROM pg_logical_slot_get_changes('test_slot', NULL, NULL,
'force-binary', '0', 'skip-empty-xacts', '1');
data
---------------------------------------------------------------------
message: transactional: 0 prefix: test_decoding, sz: 4 content:msg1
Session-2:
Commit;
Session-1:
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,
NULL, 'force-binary', '0', 'skip-empty-xacts', '1');
data
---------------------------------------------------------------------
message: transactional: 1 prefix: test_decoding, sz: 4 content:msg2
I also tried the same approach on sequences (on a commit before the
revert of sequence replication) and it seems to be working but
I think it needs further testing.
Patch 0001-Intorduce-new-field-ReplicationSlotPersistentData.no.patch
applies on master which contains the fix for logical messages.
[1] https://www.postgresql.org/message-id/flat/CAAD30U+pVmfKwUKy8cbZOnUXyguJ-uBNejwD75Kyo=OjdQGJ9g@mail....
Thoughts?
With Regards,
Zheng
Attachments:
[application/octet-stream] 0001-Intorduce-new-field-ReplicationSlotPersistentData.no.patch (12.9K, ../../CAAD30U+FD+QprMqY8wPo2n+3eS7B8=f2XYNjrUC9Sq-OFp6zDA@mail.gmail.com/2-0001-Intorduce-new-field-ReplicationSlotPersistentData.no.patch)
download | inline diff:
From 34621aef44b38af447c670f94da9e703bdd495fa Mon Sep 17 00:00:00 2001
From: "Zheng (Zane) Li" <[email protected]>
Date: Fri, 20 May 2022 19:55:50 +0000
Subject: [PATCH] Intorduce new field
ReplicationSlotPersistentData.non_xact_op_at, XLogReaderState.NonXactOpRecPtr
and SnapBuild.start_decoding_nonxactop_at. This is the end lsn of last
non-transactional message/operation decoded. We introduce this new field in
an attempt to solve the issue of missing decoding of non-transactional
message/operation under concurrency/before build state reach
SNAPBUILD_CONSISTENT. Once the build state reach SNAPBUILD_CONSISTENT,
non_xact_op_at can be set to ReplicationSlotPersistentData.confirmed_flush.
---
src/backend/replication/logical/decode.c | 18 ++++++++++---
src/backend/replication/logical/logical.c | 25 +++++++++++++++---
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/snapbuild.c | 26 +++++++++++++++++--
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 2 +-
src/include/access/xlogreader.h | 3 +++
src/include/replication/logical.h | 2 +-
src/include/replication/slot.h | 9 +++++++
src/include/replication/snapbuild.h | 5 +++-
10 files changed, 80 insertions(+), 14 deletions(-)
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index aa2427ba73..a0b700a662 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -589,12 +589,24 @@ logicalmsg_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
if (message->transactional &&
!SnapBuildProcessChange(builder, xid, buf->origptr))
return;
+
else if (!message->transactional &&
- (SnapBuildCurrentState(builder) != SNAPBUILD_CONSISTENT ||
- SnapBuildXactNeedsSkip(builder, buf->origptr)))
+ ((SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT) ||
+ /* Can't prcess non-transactional msg immediately if consumer functions are not provided */
+ ctx->prepare_write == NULL ||
+ SnapBuildNonXactOpNeedsSkip(builder, buf->origptr)))
return;
- snapshot = SnapBuildGetOrBuildSnapshot(builder, xid);
+ /*
+ * Build or get snapshot for non-transactional msg for immedieate processing.
+ * Transactional msg will be queued and processed upon decoding of the commit
+ * record.
+ */
+ if (!message->transactional)
+ {
+ snapshot = SnapBuildGetOrBuildSnapshot(builder, xid);
+ }
+
ReorderBufferQueueMessage(ctx->reorder, xid, snapshot, buf->endptr,
message->transactional,
message->message, /* first part of message is
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 625a7f4273..302a61efb8 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -208,7 +208,8 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder = ReorderBufferAllocate();
ctx->snapshot_builder =
AllocateSnapshotBuilder(ctx->reorder, xmin_horizon, start_lsn,
- need_full_snapshot, slot->data.two_phase_at);
+ need_full_snapshot, slot->data.two_phase_at,
+ slot->data.non_xact_op_at);
ctx->reorder->private_data = ctx;
@@ -1216,6 +1217,12 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
ctx->callbacks.message_cb(ctx, txn, message_lsn, transactional, prefix,
message_size, message);
+ if (!transactional)
+ {
+ SnapBuildSetNonXactOpAt(ctx->snapshot_builder, message_lsn);
+ ctx->reader->NonXactOpRecPtr = message_lsn;
+ }
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
}
@@ -1531,6 +1538,12 @@ stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
message_size, message);
+ if (!transactional)
+ {
+ SnapBuildSetNonXactOpAt(ctx->snapshot_builder, message_lsn);
+ ctx->reader->NonXactOpRecPtr = message_lsn;
+ }
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
}
@@ -1648,7 +1661,7 @@ LogicalIncreaseXminForSlot(XLogRecPtr current_lsn, TransactionId xmin)
/* candidate already valid with the current flush position, apply */
if (updated_xmin)
- LogicalConfirmReceivedLocation(slot->data.confirmed_flush);
+ LogicalConfirmReceivedLocation(slot->data.confirmed_flush, InvalidXLogRecPtr);
}
/*
@@ -1726,14 +1739,14 @@ LogicalIncreaseRestartDecodingForSlot(XLogRecPtr current_lsn, XLogRecPtr restart
/* candidates are already valid with the current flush position, apply */
if (updated_lsn)
- LogicalConfirmReceivedLocation(slot->data.confirmed_flush);
+ LogicalConfirmReceivedLocation(slot->data.confirmed_flush, InvalidXLogRecPtr);
}
/*
* Handle a consumer's confirmation having received all changes up to lsn.
*/
void
-LogicalConfirmReceivedLocation(XLogRecPtr lsn)
+LogicalConfirmReceivedLocation(XLogRecPtr lsn, XLogRecPtr lsn_nonxact)
{
Assert(lsn != InvalidXLogRecPtr);
@@ -1747,6 +1760,8 @@ LogicalConfirmReceivedLocation(XLogRecPtr lsn)
SpinLockAcquire(&MyReplicationSlot->mutex);
MyReplicationSlot->data.confirmed_flush = lsn;
+ if (lsn_nonxact != InvalidXLogRecPtr)
+ MyReplicationSlot->data.non_xact_op_at = lsn_nonxact;
/* if we're past the location required for bumping xmin, do so */
if (MyReplicationSlot->candidate_xmin_lsn != InvalidXLogRecPtr &&
@@ -1812,6 +1827,8 @@ LogicalConfirmReceivedLocation(XLogRecPtr lsn)
{
SpinLockAcquire(&MyReplicationSlot->mutex);
MyReplicationSlot->data.confirmed_flush = lsn;
+ if (lsn_nonxact != InvalidXLogRecPtr)
+ MyReplicationSlot->data.non_xact_op_at = lsn_nonxact;
SpinLockRelease(&MyReplicationSlot->mutex);
}
}
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index 6058d36e0d..5d579218d8 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -294,7 +294,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
*/
if (ctx->reader->EndRecPtr != InvalidXLogRecPtr && confirm)
{
- LogicalConfirmReceivedLocation(ctx->reader->EndRecPtr);
+ LogicalConfirmReceivedLocation(ctx->reader->EndRecPtr, ctx->reader->NonXactOpRecPtr);
/*
* If only the confirmed_flush_lsn has changed the slot won't get
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 1119a12db9..d5d378836a 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -175,6 +175,8 @@ struct SnapBuild
*/
XLogRecPtr two_phase_at;
+ XLogRecPtr start_decoding_nonxactop_at;
+
/*
* Don't start decoding WAL until the "xl_running_xacts" information
* indicates there are no running xids with an xid smaller than this.
@@ -281,7 +283,8 @@ AllocateSnapshotBuilder(ReorderBuffer *reorder,
TransactionId xmin_horizon,
XLogRecPtr start_lsn,
bool need_full_snapshot,
- XLogRecPtr two_phase_at)
+ XLogRecPtr two_phase_at,
+ XLogRecPtr non_xact_op_at)
{
MemoryContext context;
MemoryContext oldcontext;
@@ -310,6 +313,7 @@ AllocateSnapshotBuilder(ReorderBuffer *reorder,
builder->start_decoding_at = start_lsn;
builder->building_full_snapshot = need_full_snapshot;
builder->two_phase_at = two_phase_at;
+ builder->start_decoding_nonxactop_at = non_xact_op_at+1;
MemoryContextSwitchTo(oldcontext);
@@ -387,6 +391,15 @@ SnapBuildSetTwoPhaseAt(SnapBuild *builder, XLogRecPtr ptr)
builder->two_phase_at = ptr;
}
+/*
+ * Set the LSN at which non-transactional op decoding is at.
+ */
+void
+SnapBuildSetNonXactOpAt(SnapBuild *builder, XLogRecPtr ptr)
+{
+ builder->start_decoding_nonxactop_at = ptr;
+}
+
/*
* Should the contents of transaction ending at 'ptr' be decoded?
*/
@@ -396,6 +409,15 @@ SnapBuildXactNeedsSkip(SnapBuild *builder, XLogRecPtr ptr)
return ptr < builder->start_decoding_at;
}
+/*
+ * Should the contents of a non-transactional op ending at 'ptr' be decoded?
+ */
+bool
+SnapBuildNonXactOpNeedsSkip(SnapBuild *builder, XLogRecPtr ptr)
+{
+ return ptr < builder->start_decoding_nonxactop_at;
+}
+
/*
* Increase refcount of a snapshot.
*
@@ -661,7 +683,7 @@ SnapBuildExportSnapshot(SnapBuild *builder)
Snapshot
SnapBuildGetOrBuildSnapshot(SnapBuild *builder, TransactionId xid)
{
- Assert(builder->state == SNAPBUILD_CONSISTENT);
+ Assert(builder->state >= SNAPBUILD_FULL_SNAPSHOT);
/* only build a new snapshot if we don't have a prebuilt one */
if (builder->snapshot == NULL)
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index ca945994ef..3bbbe74058 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -530,7 +530,7 @@ pg_logical_replication_slot_advance(XLogRecPtr moveto)
if (ctx->reader->EndRecPtr != InvalidXLogRecPtr)
{
- LogicalConfirmReceivedLocation(moveto);
+ LogicalConfirmReceivedLocation(moveto, moveto);
/*
* If only the confirmed_flush LSN has changed the slot won't get
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index e42671722a..b1373e52fd 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -2170,7 +2170,7 @@ ProcessStandbyReplyMessage(void)
if (MyReplicationSlot && flushPtr != InvalidXLogRecPtr)
{
if (SlotIsLogical(MyReplicationSlot))
- LogicalConfirmReceivedLocation(flushPtr);
+ LogicalConfirmReceivedLocation(flushPtr, flushPtr);
else
PhysicalConfirmReceivedLocation(flushPtr);
}
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index e73ea4a840..b197a6a354 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -206,6 +206,9 @@ struct XLogReaderState
XLogRecPtr ReadRecPtr; /* start of last record read */
XLogRecPtr EndRecPtr; /* end+1 of last record read */
+ /* end of last non-transactional op record read */
+ XLogRecPtr NonXactOpRecPtr;
+
/*
* Set at the end of recovery: the start point of a partial record at the
* end of WAL (InvalidXLogRecPtr if there wasn't one), and the start
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index edadacd589..c9714b51f0 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -136,7 +136,7 @@ extern void FreeDecodingContext(LogicalDecodingContext *ctx);
extern void LogicalIncreaseXminForSlot(XLogRecPtr lsn, TransactionId xmin);
extern void LogicalIncreaseRestartDecodingForSlot(XLogRecPtr current_lsn,
XLogRecPtr restart_lsn);
-extern void LogicalConfirmReceivedLocation(XLogRecPtr lsn);
+extern void LogicalConfirmReceivedLocation(XLogRecPtr lsn, XLogRecPtr lsn_nonxact);
extern bool filter_prepare_cb_wrapper(LogicalDecodingContext *ctx,
TransactionId xid, const char *gid);
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 8c9f3321d5..941c6f6da3 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -83,6 +83,15 @@ typedef struct ReplicationSlotPersistentData
*/
XLogRecPtr confirmed_flush;
+ /*
+ * Oldest LSN that the client has acked receipt for a non-transactional
+ * operation such as a non-transactional message. This is introduced
+ * to slove missing non-transactional decoding when creating logical
+ * replication slot under concurrency. Under normal situation
+ * (SNAPBUILD_CONSISTENT) we can make non_xact_op_at = confirmed_flush
+ */
+ XLogRecPtr non_xact_op_at;
+
/*
* LSN at which we enabled two_phase commit for this slot or LSN at which
* we found a consistent point at the time of slot creation.
diff --git a/src/include/replication/snapbuild.h b/src/include/replication/snapbuild.h
index d179251aad..7b055dba7c 100644
--- a/src/include/replication/snapbuild.h
+++ b/src/include/replication/snapbuild.h
@@ -62,7 +62,8 @@ extern void CheckPointSnapBuild(void);
extern SnapBuild *AllocateSnapshotBuilder(struct ReorderBuffer *cache,
TransactionId xmin_horizon, XLogRecPtr start_lsn,
bool need_full_snapshot,
- XLogRecPtr two_phase_at);
+ XLogRecPtr two_phase_at,
+ XLogRecPtr non_xact_op_at);
extern void FreeSnapshotBuilder(SnapBuild *cache);
extern void SnapBuildSnapDecRefcount(Snapshot snap);
@@ -77,8 +78,10 @@ extern Snapshot SnapBuildGetOrBuildSnapshot(SnapBuild *builder,
TransactionId xid);
extern bool SnapBuildXactNeedsSkip(SnapBuild *snapstate, XLogRecPtr ptr);
+extern bool SnapBuildNonXactOpNeedsSkip(SnapBuild *builder, XLogRecPtr ptr);
extern XLogRecPtr SnapBuildGetTwoPhaseAt(SnapBuild *builder);
extern void SnapBuildSetTwoPhaseAt(SnapBuild *builder, XLogRecPtr ptr);
+extern void SnapBuildSetNonXactOpAt(SnapBuild *builder, XLogRecPtr ptr);
extern void SnapBuildCommitTxn(SnapBuild *builder, XLogRecPtr lsn,
TransactionId xid, int nsubxacts,
--
2.32.0
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v12 2/3] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-05-30 03:15 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-05-30 03:15 UTC (permalink / raw)
pg_hba.conf file now has support for "include", "include_dir" and
"include_if_exists" directives, which work similarly to the same directives in
the postgresql.conf file.
This fixes a possible crash if a secondary file tries to include itself as
there's now a nesting depth check in the inclusion code path, same as the
postgresql.conf.
Many regression tests added to cover both the new directives, but also error
detection for the whole pg_hba / pg_ident files.
Catversion is bumped.
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/client-auth.sgml | 86 ++-
doc/src/sgml/system-views.sgml | 22 +-
src/backend/libpq/hba.c | 483 ++++++++++---
src/backend/libpq/pg_hba.conf.sample | 25 +-
src/backend/libpq/pg_ident.conf.sample | 15 +-
src/backend/utils/adt/hbafuncs.c | 43 +-
src/backend/utils/misc/guc-file.l | 229 +++---
src/include/catalog/pg_proc.dat | 12 +-
src/include/libpq/hba.h | 5 +-
src/include/utils/guc.h | 2 +
.../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++
src/test/regress/expected/rules.out | 6 +-
12 files changed, 1311 insertions(+), 274 deletions(-)
create mode 100644 src/test/authentication/t/003_file_inclusion.pl
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index 32d5d45863..2ae723de66 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,23 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication
+ record. Inclusion directives specify files that can be included, which
+ contains additional records. The records will be inserted in lieu of the
+ inclusion records. Those records only contains two fields: the
+ <literal>include</literal>, <literal>include_if_exists</literal> or
+ <literal>include_dir</literal> directive and the file or directory to be
+ included. The file or directory can be a relative of absolute path, and can
+ be double quoted if needed. For the <literal>include_dir</literal> form,
+ all files not starting with a <literal>.</literal> and ending with
+ <literal>.conf</literal> will be included. Multiple files within an include
+ directory are processed in file name order (according to C locale rules,
+ i.e., numbers before letters, and uppercase letters before lowercase ones).
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,21 +118,57 @@
<para>
A record can have several formats:
<synopsis>
-local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+include <replaceable>file</replaceable>
+include_if_exists <replaceable>file</replaceable>
+include_dir <replaceable>directory</replaceable>
+local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
</synopsis>
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_if_exists</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file if the
+ file exists and can be read. Otherwise, a message will be logged to
+ indicate that the file is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_dir</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of all the files found in
+ the directory, if they don't start with a <literal>.</literal> and end
+ with <literal>.conf</literal>, processed in file name order (according
+ to C locale rules, i.e., numbers before letters, and uppercase letters
+ before lowercase ones).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -863,8 +914,10 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
+<replaceable>include_dir</replaceable> <replaceable>directory</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -875,6 +928,11 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ As for <filename>pg_hba.conf</filename>, the lines in this file can either
+ be inclusion directives or user name map records, and follow the same
+ rules.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 4723f712a7..7d1cec8b7f 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -1003,12 +1003,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule the given <literal>file_name</literal>
</para></entry>
</row>
@@ -1153,12 +1162,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this mapping
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_ident.conf</filename>
+ Line number of this mapping in the given <literal>file_name</literal>
</para></entry>
</row>
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index ea92f02a47..183312b10a 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -22,6 +22,7 @@
#include <sys/param.h>
#include <sys/socket.h>
#include <netdb.h>
+#include <sys/stat.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
@@ -70,6 +71,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -115,14 +122,26 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int depth,
+ int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
- const char *inc_filename, int elevel, char **err_msg);
+ const char *inc_filename, int depth, int elevel,
+ char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
static int regcomp_auth_token(AuthToken *token, char *filename, int line_num,
char **err_msg, int elevel);
static int regexec_auth_token(const char *match, AuthToken *token,
size_t nmatch, regmatch_t pmatch[]);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ bool strict, const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
+static char *process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth,
+ int elevel, MemoryContext linecxt,
+ List **tok_lines);
/*
@@ -413,7 +432,7 @@ regexec_auth_token(const char *match, AuthToken *token, size_t nmatch,
*/
static List *
next_field_expand(const char *filename, char **lineptr,
- int elevel, char **err_msg)
+ int depth, int elevel, char **err_msg)
{
char buf[MAX_TOKEN];
bool trailing_comma;
@@ -429,7 +448,7 @@ next_field_expand(const char *filename, char **lineptr,
/* Is this referencing a file? */
if (!initial_quote && buf[0] == '@' && buf[1] != '\0')
- tokens = tokenize_inc_file(tokens, filename, buf + 1,
+ tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1,
elevel, err_msg);
else
tokens = lappend(tokens, make_auth_token(buf, initial_quote));
@@ -457,6 +476,7 @@ static List *
tokenize_inc_file(List *tokens,
const char *outer_filename,
const char *inc_filename,
+ int depth,
int elevel,
char **err_msg)
{
@@ -466,39 +486,30 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
{
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
+ *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", *err_msg)));
+ return tokens;
}
- inc_file = AllocateFile(inc_fullname, "r");
- if (inc_file == NULL)
- {
- int save_errno = errno;
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true,
+ outer_filename, elevel, err_msg, &inc_fullname);
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
+ if (inc_file == NULL)
return tokens;
- }
/* There is possible recursion here if the file contains @ */
- linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
+ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1,
+ elevel);
FreeFile(inc_file);
pfree(inc_fullname);
@@ -536,11 +547,38 @@ tokenize_inc_file(List *tokens,
/*
* tokenize_auth_file
- * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a dedicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
+ int depth, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, depth,
+ elevel);
+
+ return linecxt;
+}
+
+/*
+ * Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see the struct definition
* in libpq/hba.h.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -549,30 +587,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int depth, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -625,7 +655,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
{
List *current_field;
- current_field = next_field_expand(filename, &lineptr,
+ current_field = next_field_expand(filename, &lineptr, depth,
elevel, &err_msg);
/* add field to line, unless we are at EOL or comment start */
if (current_field != NIL)
@@ -633,29 +663,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ AuthToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
+
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
+
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, true,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
+ else if (strcmp(first->string, "include_dir") == 0)
+ {
+ char **filenames;
+ char *dir_name = second->string;
+ int num_filenames;
+ StringInfoData err_buf;
+
+ filenames = GetDirConfFiles(dir_name, filename, elevel,
+ &num_filenames, &err_msg);
+
+ if (!filenames)
+ {
+ /* We have the error in err_msg, simply process it */
+ goto process_line;
+ }
+
+ initStringInfo(&err_buf);
+ for (int i = 0; i < num_filenames; i++)
+ {
+ /*
+ * err_msg is used here as a temp buffer, it will be
+ * overwritten at the end of the loop with the
+ * cumulated errors, if any.
+ */
+ err_msg = process_included_authfile(filenames[i], true,
+ filename, depth + 1, elevel,
+ linecxt, tok_lines);
+
+ /* Cumulate errors if any. */
+ if (err_msg)
+ {
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfoString(&err_buf, err_msg);
+ }
+ }
+
+ /*
+ * If there were no errors, the line is fully processed, bypass
+ * the general TokenizedAuthLine processing.
+ */
+ if (err_buf.len == 0)
+ goto next_line;
+
+ /* Otherwise, process the cumulated errors, if any. */
+ err_msg = err_buf.data;
+ }
+ else if (strcmp(first->string, "include_if_exists") == 0)
+ {
+ char *inc_filename;
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, false,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
}
+process_line:
+ /*
+ * General processing: report the error if any and emit line to the
+ * TokenizedAuthLine
+ */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -984,7 +1112,7 @@ do { \
errmsg("authentication option \"%s\" is only valid for authentication methods %s", \
optname, _(validmethods)), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \
optname, validmethods); \
return false; \
@@ -1004,7 +1132,7 @@ do { \
errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname); \
return NULL; \
@@ -1027,7 +1155,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("missing entry at end of line"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, file_name))); \
*err_msg = pstrdup("missing entry at end of line"); \
return NULL; \
} \
@@ -1040,7 +1168,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("multiple values in ident field"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, file_name))); \
*err_msg = pstrdup("multiple values in ident field"); \
return NULL; \
} \
@@ -1063,6 +1191,7 @@ HbaLine *
parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
{
int line_num = tok_line->line_num;
+ char *file_name = tok_line->file_name;
char **err_msg = &tok_line->err_msg;
char *str;
struct addrinfo *gai_result;
@@ -1077,6 +1206,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -1091,7 +1221,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for connection type"),
errhint("Specify exactly one connection type per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for connection type";
return NULL;
}
@@ -1119,7 +1249,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("hostssl record cannot match because SSL is disabled"),
errhint("Set ssl = on in postgresql.conf."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostssl record cannot match because SSL is disabled";
}
#else
@@ -1127,7 +1257,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("hostssl record cannot match because SSL is not supported by this build"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostssl record cannot match because SSL is not supported by this build";
#endif
}
@@ -1139,7 +1269,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("hostgssenc record cannot match because GSSAPI is not supported by this build"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostgssenc record cannot match because GSSAPI is not supported by this build";
#endif
}
@@ -1160,7 +1290,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid connection type \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid connection type \"%s\"", token->string);
return NULL;
}
@@ -1173,7 +1303,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before database specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before database specification";
return NULL;
}
@@ -1198,7 +1328,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before role specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before role specification";
return NULL;
}
@@ -1225,7 +1355,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before IP address specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before IP address specification";
return NULL;
}
@@ -1237,7 +1367,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for host address"),
errhint("Specify one address range per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for host address";
return NULL;
}
@@ -1296,7 +1426,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid IP address \"%s\": %s",
str, gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid IP address \"%s\": %s",
str, gai_strerror(ret));
if (gai_result)
@@ -1316,7 +1446,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("specifying both host name and CIDR mask is invalid: \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("specifying both host name and CIDR mask is invalid: \"%s\"",
token->string);
return NULL;
@@ -1330,7 +1460,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid CIDR mask in address \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid CIDR mask in address \"%s\"",
token->string);
return NULL;
@@ -1350,7 +1480,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("end-of-line before netmask specification"),
errhint("Specify an address range in CIDR notation, or provide a separate netmask."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before netmask specification";
return NULL;
}
@@ -1361,7 +1491,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("multiple values specified for netmask"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for netmask";
return NULL;
}
@@ -1376,7 +1506,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid IP mask \"%s\": %s",
token->string, gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid IP mask \"%s\": %s",
token->string, gai_strerror(ret));
if (gai_result)
@@ -1395,7 +1525,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("IP address and mask do not match"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "IP address and mask do not match";
return NULL;
}
@@ -1411,7 +1541,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before authentication method"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before authentication method";
return NULL;
}
@@ -1423,7 +1553,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for authentication type"),
errhint("Specify exactly one authentication type per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for authentication type";
return NULL;
}
@@ -1460,7 +1590,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("MD5 authentication is not supported when \"db_user_namespace\" is enabled"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "MD5 authentication is not supported when \"db_user_namespace\" is enabled";
return NULL;
}
@@ -1501,7 +1631,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid authentication method \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid authentication method \"%s\"",
token->string);
return NULL;
@@ -1514,7 +1644,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid authentication method \"%s\": not supported by this build",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid authentication method \"%s\": not supported by this build",
token->string);
return NULL;
@@ -1536,7 +1666,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("gssapi authentication is not supported on local sockets"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "gssapi authentication is not supported on local sockets";
return NULL;
}
@@ -1548,7 +1678,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("peer authentication is only supported on local sockets"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "peer authentication is only supported on local sockets";
return NULL;
}
@@ -1566,7 +1696,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cert authentication is only supported on hostssl connections"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cert authentication is only supported on hostssl connections";
return NULL;
}
@@ -1616,7 +1746,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("authentication option not in name=value format: %s", token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("authentication option not in name=value format: %s",
token->string);
return NULL;
@@ -1660,7 +1790,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix";
return NULL;
}
@@ -1671,7 +1801,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set";
return NULL;
}
@@ -1687,7 +1817,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cannot use ldapsearchattribute together with ldapsearchfilter"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cannot use ldapsearchattribute together with ldapsearchfilter";
return NULL;
}
@@ -1704,7 +1834,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("list of RADIUS servers cannot be empty"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "list of RADIUS servers cannot be empty";
return NULL;
}
@@ -1715,7 +1845,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("list of RADIUS secrets cannot be empty"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "list of RADIUS secrets cannot be empty";
return NULL;
}
@@ -1734,7 +1864,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiussecrets),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS secrets (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiussecrets),
list_length(parsedline->radiusservers));
@@ -1750,7 +1880,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiusports),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS ports (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiusports),
list_length(parsedline->radiusservers));
@@ -1766,7 +1896,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiusidentifiers),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS identifiers (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiusidentifiers),
list_length(parsedline->radiusservers));
@@ -1801,6 +1931,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg)
{
int line_num = hbaline->linenumber;
+ char *file_name = hbaline->sourcefile;
#ifdef USE_LDAP
hbaline->ldapscope = LDAP_SCOPE_SUBTREE;
@@ -1824,7 +1955,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientcert can only be configured for \"hostssl\" rows"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientcert can only be configured for \"hostssl\" rows";
return false;
}
@@ -1841,7 +1972,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientcert only accepts \"verify-full\" when using \"cert\" authentication"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientcert can only be set to \"verify-full\" when using \"cert\" authentication";
return false;
}
@@ -1854,7 +1985,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid value for clientcert: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
}
@@ -1866,7 +1997,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientname can only be configured for \"hostssl\" rows"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientname can only be configured for \"hostssl\" rows";
return false;
}
@@ -1885,7 +2016,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid value for clientname: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
}
@@ -1971,7 +2102,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid ldapscheme value: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
hbaline->ldapscheme = pstrdup(val);
}
else if (strcmp(name, "ldapserver") == 0)
@@ -1989,7 +2120,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid LDAP port number: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid LDAP port number: \"%s\"", val);
return false;
}
@@ -2083,7 +2214,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS server list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2102,7 +2233,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not translate RADIUS server name \"%s\" to address: %s",
(char *) lfirst(l), gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
if (gai_result)
pg_freeaddrinfo_all(hints.ai_family, gai_result);
@@ -2131,7 +2262,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS port list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid RADIUS port number: \"%s\"", val);
return false;
}
@@ -2144,7 +2275,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid RADIUS port number: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2167,7 +2298,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS secret list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2189,7 +2320,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS identifiers list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2203,7 +2334,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("unrecognized authentication option name: \"%s\"",
name),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("unrecognized authentication option name: \"%s\"",
name);
return false;
@@ -2351,7 +2482,7 @@ load_hba(void)
return false;
}
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
@@ -2441,6 +2572,135 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ if (strict)
+ {
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ }
+ else
+ {
+ Assert(kind == IncludedAuthFile);
+ ereport(LOG,
+ (errmsg("skipping missing authentication file \"%s\"",
+ *inc_fullname)));
+ }
+
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
+
+/*
+ * Try to open an included file, and tokenize it using the given context.
+ * Returns NULL if no error happens during tokenization, otherwise the error.
+ */
+static char *
+process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth, int elevel,
+ MemoryContext linecxt, List **tok_lines)
+{
+ char *inc_fullname;
+ FILE *inc_file;
+ char *err_msg = NULL;
+
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
+ {
+ err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", err_msg)));
+ return err_msg;
+ }
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict,
+ outer_filename, elevel, &err_msg, &inc_fullname);
+
+ if (inc_file == NULL)
+ {
+ if (strict)
+ {
+ /* open_inc_file should have reported an error. */
+ Assert(err_msg != NULL);
+ return err_msg;
+ }
+ else
+ return NULL;
+ }
+ else
+ {
+ /* No error message should have been reported. */
+ Assert(err_msg == NULL);
+ }
+
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, depth, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ return NULL;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
@@ -2459,6 +2719,7 @@ load_hba(void)
IdentLine *
parse_ident_line(TokenizedAuthLine *tok_line, int elevel)
{
+ char *file_name = tok_line->file_name;
int line_num = tok_line->line_num;
char **err_msg = &tok_line->err_msg;
ListCell *field;
@@ -2500,8 +2761,8 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel)
* Now that the field validation is done, compile a regex from the user
* token, if necessary.
*/
- if (regcomp_auth_token(parsedline->token, IdentFileName, line_num,
- err_msg, elevel))
+ if (regcomp_auth_token(parsedline->token, file_name, line_num, err_msg,
+ elevel))
{
/* err_msg includes the error to report */
return NULL;
@@ -2722,7 +2983,7 @@ load_ident(void)
return false;
}
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..7433050112 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,16 +9,27 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
-# local DATABASE USER METHOD [OPTIONS]
-# host DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# local DATABASE USER METHOD [OPTIONS]
+# host DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..8e3fa29135 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,23 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
-# MAPNAME SYSTEM-USERNAME PG-USERNAME
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index 21a451e391..ea8de9a057 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,12 +26,12 @@
static ArrayType *get_hba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int rule_number, int lineno, HbaLine *hba,
- const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int mapping_number, int lineno, IdentLine *ident,
- const char *err_msg);
+ int mapping_number, const char *filename,
+ int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
@@ -159,7 +159,7 @@ get_hba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 10
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line
@@ -168,7 +168,8 @@ get_hba_options(HbaLine *hba)
* tuple_store: where to store data
* tupdesc: tuple descriptor for the view
* rule_number: unique rule identifier among all valid rules
- * lineno: pg_hba.conf line number (must always be valid)
+ * filename: name of the file containing that line
+ * lineno: line number in that file (must always be valid)
* hba: parsed line data (can be NULL, in which case err_msg should be set)
* err_msg: error message (NULL if none)
*
@@ -177,7 +178,7 @@ get_hba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int rule_number, int lineno, HbaLine *hba,
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -202,6 +203,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -345,7 +348,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -386,7 +389,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open configuration file \"%s\": %m",
HbaFileName)));
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -407,8 +410,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
rule_number++;
- fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -445,7 +448,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
}
/* Number of columns in pg_hba_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -454,7 +457,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
* tuple_store: where to store data
* tupdesc: tuple descriptor for the view
* mapping_number: unique rule identifier among all valid rules
- * lineno: pg_ident.conf line number (must always be valid)
+ * filename: name of the file containing that line
+ * lineno: line number in that file (must always be valid)
* ident: parsed line data (can be NULL, in which case err_msg should be set)
* err_msg: error message (NULL if none)
*
@@ -463,8 +467,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int mapping_number, int lineno, IdentLine *ident,
- const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -482,6 +486,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -494,7 +500,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -534,7 +540,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open usermap file \"%s\": %m",
IdentFileName)));
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -556,7 +562,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
mapping_number++;
fill_ident_line(tuple_store, tupdesc, mapping_number,
- tok_line->line_num, identline, tok_line->err_msg);
+ tok_line->file_name, tok_line->line_num, identline,
+ tok_line->err_msg);
}
/* Free tokenizer memory */
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index 721628c0cf..86b6cc1c8a 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -345,6 +345,110 @@ GUC_flex_fatal(const char *msg)
return 0; /* keep compiler quiet */
}
+/*
+ * Returns the list of config files located in a directory, in alphabetical
+ * order.
+ *
+ * We don't check for recursion or too-deep nesting depth here, its up to the
+ * caller to take care of that.
+ */
+char **
+GetDirConfFiles(const char *includedir, const char *calling_file, int elevel,
+ int *num_filenames, char **err_msg)
+{
+ char *directory;
+ DIR *d;
+ struct dirent *de;
+ char **filenames;
+ int size_filenames;
+
+ /*
+ * Reject directory name that is all-blank (including empty), as that
+ * leads to confusion --- we'd read the containing directory, typically
+ * resulting in recursive inclusion of the same file(s).
+ */
+ if (strspn(includedir, " \t\r\n") == strlen(includedir))
+ {
+ ereport(elevel,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("empty configuration directory name: \"%s\"",
+ includedir)));
+ *err_msg = "empty configuration directory name";
+ return NULL;
+ }
+
+ directory = AbsoluteConfigLocation(includedir, calling_file);
+ d = AllocateDir(directory);
+ if (d == NULL)
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not open configuration directory \"%s\": %m",
+ directory)));
+ *err_msg = psprintf("could not open directory \"%s\"", directory);
+ filenames = NULL;
+ goto cleanup;
+ }
+
+ /*
+ * Read the directory and put the filenames in an array, so we can sort
+ * them prior to caller processing the contents.
+ */
+ size_filenames = 32;
+ filenames = (char **) palloc(size_filenames * sizeof(char *));
+ *num_filenames = 0;
+
+ while ((de = ReadDir(d, directory)) != NULL)
+ {
+ PGFileType de_type;
+ char filename[MAXPGPATH];
+
+ /*
+ * Only parse files with names ending in ".conf". Explicitly reject
+ * files starting with ".". This excludes things like "." and "..",
+ * as well as typical hidden files, backup files, and editor debris.
+ */
+ if (strlen(de->d_name) < 6)
+ continue;
+ if (de->d_name[0] == '.')
+ continue;
+ if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
+ continue;
+
+ join_path_components(filename, directory, de->d_name);
+ canonicalize_path(filename);
+ de_type = get_dirent_type(filename, de, true, elevel);
+ if (de_type == PGFILETYPE_ERROR)
+ {
+ *err_msg = psprintf("could not stat file \"%s\"", filename);
+ pfree(filenames);
+ filenames = NULL;
+ goto cleanup;
+ }
+ else if (de_type != PGFILETYPE_DIR)
+ {
+ /* Add file to array, increasing its size in blocks of 32 */
+ if (*num_filenames >= size_filenames)
+ {
+ size_filenames += 32;
+ filenames = (char **) repalloc(filenames,
+ size_filenames * sizeof(char *));
+ }
+ filenames[*num_filenames] = pstrdup(filename);
+ (*num_filenames)++;
+ }
+ }
+
+ if (*num_filenames > 0)
+ qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp);
+
+cleanup:
+ if (d)
+ FreeDir(d);
+ pfree(directory);
+ return filenames;
+}
+
/*
* Read and parse a single configuration file. This function recurses
* to handle "include" directives.
@@ -606,127 +710,30 @@ ParseConfigDirectory(const char *includedir,
ConfigVariable **head_p,
ConfigVariable **tail_p)
{
- char *directory;
- DIR *d;
- struct dirent *de;
+ char *err_msg;
char **filenames;
int num_filenames;
- int size_filenames;
- bool status;
-
- /*
- * Reject directory name that is all-blank (including empty), as that
- * leads to confusion --- we'd read the containing directory, typically
- * resulting in recursive inclusion of the same file(s).
- */
- if (strspn(includedir, " \t\r\n") == strlen(includedir))
- {
- ereport(elevel,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("empty configuration directory name: \"%s\"",
- includedir)));
- record_config_file_error("empty configuration directory name",
- calling_file, calling_lineno,
- head_p, tail_p);
- return false;
- }
-
- /*
- * We don't check for recursion or too-deep nesting depth here; the
- * subsequent calls to ParseConfigFile will take care of that.
- */
-
- directory = AbsoluteConfigLocation(includedir, calling_file);
- d = AllocateDir(directory);
- if (d == NULL)
- {
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open configuration directory \"%s\": %m",
- directory)));
- record_config_file_error(psprintf("could not open directory \"%s\"",
- directory),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
- /*
- * Read the directory and put the filenames in an array, so we can sort
- * them prior to processing the contents.
- */
- size_filenames = 32;
- filenames = (char **) palloc(size_filenames * sizeof(char *));
- num_filenames = 0;
+ filenames = GetDirConfFiles(includedir, calling_file, elevel,
+ &num_filenames, &err_msg);
- while ((de = ReadDir(d, directory)) != NULL)
+ if (!filenames)
{
- PGFileType de_type;
- char filename[MAXPGPATH];
-
- /*
- * Only parse files with names ending in ".conf". Explicitly reject
- * files starting with ".". This excludes things like "." and "..",
- * as well as typical hidden files, backup files, and editor debris.
- */
- if (strlen(de->d_name) < 6)
- continue;
- if (de->d_name[0] == '.')
- continue;
- if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
- continue;
-
- join_path_components(filename, directory, de->d_name);
- canonicalize_path(filename);
- de_type = get_dirent_type(filename, de, true, elevel);
- if (de_type == PGFILETYPE_ERROR)
- {
- record_config_file_error(psprintf("could not stat file \"%s\"",
- filename),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
- else if (de_type != PGFILETYPE_DIR)
- {
- /* Add file to array, increasing its size in blocks of 32 */
- if (num_filenames >= size_filenames)
- {
- size_filenames += 32;
- filenames = (char **) repalloc(filenames,
- size_filenames * sizeof(char *));
- }
- filenames[num_filenames] = pstrdup(filename);
- num_filenames++;
- }
+ record_config_file_error(err_msg, calling_file, calling_lineno, head_p,
+ tail_p);
+ return false;
}
- if (num_filenames > 0)
+ for (int i = 0; i < num_filenames; i++)
{
- int i;
-
- qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp);
- for (i = 0; i < num_filenames; i++)
- {
- if (!ParseConfigFile(filenames[i], true,
- calling_file, calling_lineno,
- depth, elevel,
- head_p, tail_p))
- {
- status = false;
- goto cleanup;
- }
- }
+ if (!ParseConfigFile(filenames[i], true,
+ calling_file, calling_lineno,
+ depth, elevel,
+ head_p, tail_p))
+ return false;
}
- status = true;
-cleanup:
- if (d)
- FreeDir(d);
- pfree(directory);
- return status;
+ return true;
}
/*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4f5d05d0ce..2ad06c4d3e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6135,16 +6135,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '6250', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,text,text,text}',
- proargmodes => '{o,o,o,o,o,o}',
- proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index cec2e2665f..6b3bf814b5 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -93,6 +93,7 @@ typedef struct AuthToken
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -157,6 +158,7 @@ typedef struct IdentLine
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of AuthTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
@@ -176,6 +178,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel);
extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel);
extern bool pg_isblank(const char c);
extern MemoryContext tokenize_auth_file(const char *filename, FILE *file,
- List **tok_lines, int elevel);
+ List **tok_lines, int depth,
+ int elevel);
#endif /* HBA_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index b3aaff9665..59ca39d908 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -144,6 +144,8 @@ typedef struct ConfigVariable
struct ConfigVariable *next;
} ConfigVariable;
+extern char **GetDirConfFiles(const char *includedir, const char *calling_file,
+ int elevel, int *num_filenames, char **err_msg);
extern bool ParseConfigFile(const char *config_file, bool strict,
const char *calling_file, int calling_lineno,
int depth, int elevel,
diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl
new file mode 100644
index 0000000000..8eae72b8d4
--- /dev/null
+++ b/src/test/authentication/t/003_file_inclusion.pl
@@ -0,0 +1,657 @@
+
+# Copyright (c) 2021-2022, PostgreSQL Global Development Group
+
+# Set of tests for authentication and pg_hba.conf inclusion.
+# This test can only run with Unix-domain sockets.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Time::HiRes qw(usleep);
+use IPC::Run qw(pump finish timer);
+use Data::Dumper;
+
+if (!$use_unix_sockets)
+{
+ plan skip_all =>
+ "authentication tests cannot run without Unix-domain sockets";
+}
+
+# stores the current line counter for each file. hba_rule and ident_rule are
+# fake file names used for the global rule number for each auth view.
+my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+my $hba_file = 'subdir1/pg_hba_custom.conf';
+my $ident_file = 'subdir2/pg_ident_custom.conf';
+
+# Initialize primary node
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+$node->start;
+
+my $data_dir = $node->data_dir;
+
+# Normalize the data directory for Windows
+$data_dir =~ s/\/\.\//\//g; # reduce /./ to /
+$data_dir =~ s/\/\//\//g; # reduce // to /
+$data_dir =~ s/\/$//; # remove trailing /
+
+
+# Add the given payload to the given relative HBA file of the given node.
+# This function maintains the %cur_line metadata, so it has to be called in the
+# expected inclusion evaluation order in order to keep it in sync.
+#
+# If the payload starts with "include" or "ignore", the function doesn't
+# increase the general hba rule number.
+#
+# If an err_str is provided, it returns an arrayref containing the provided
+# filename, the current line number in that file and the provided err_str. The
+# err_str has to be a valid regex string.
+# Otherwise it only returns the line number of the payload in the wanted file.
+# This function has to be called in the expected inclusion evaluation order to
+# keep the %cur_line information in sync.
+sub add_hba_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'hba_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_hba_file_rules line
+ @tokens = split(/ /, $payload);
+ $tokens[1] = '{' . $tokens[1] . '}'; # database
+ $tokens[2] = '{' . $tokens[2] . '}'; # user_name
+
+ # add empty address and netmask betweed user_name and auth_method
+ splice @tokens, 3, 0, '';
+ splice @tokens, 3, 0, '';
+
+ # append empty options and error
+ push @tokens, '';
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Add the given payload to the given relative ident file of the given node.
+# Same as add_hba_line but for pg_ident files
+sub add_ident_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'ident_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_ident_file_mappings line
+ @tokens = split(/ /, $payload);
+
+ # append empty error
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Delete pg_hba.conf from the given node, add various entries to test the
+# include infrastructure and then execute a reload to refresh it.
+sub generate_valid_auth_files
+{
+ my $node = shift;
+ my $hba_expected = '';
+ my $ident_expected = '';
+
+ # customise main auth file names
+ $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'");
+ $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'");
+
+ # and make original ones invalid to be sure they're not used anywhere
+ $node->append_conf('pg_hba.conf', "some invalid line");
+ $node->append_conf('pg_ident.conf', "some invalid line");
+
+ # pg_hba stuff
+ mkdir("$data_dir/subdir1");
+ mkdir("$data_dir/hba_inc");
+ mkdir("$data_dir/hba_inc_if");
+ mkdir("$data_dir/hba_pos");
+
+ # Make sure we will still be able to connect
+ $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust');
+
+ # Add include data
+ add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf");
+ $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject");
+
+ $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject");
+
+ add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject");
+ # include is relative to current path
+ add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject");
+
+ # include_if_exists data
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none");
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some");
+ $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject");
+
+ # include_dir data
+ add_hba_line($node, "$hba_file", "include_dir ../hba_inc");
+ add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included");
+ $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject");
+ $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject");
+
+ # secondary auth file
+ add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject');
+ $node->append_conf('dbnames.conf', "db1");
+ $node->append_conf('dbnames.conf', "db3");
+ $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1)
+ . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1)
+ . '|local|{db1,db3}|{all}|||reject||';
+
+ # pg_ident stuff
+ mkdir("$data_dir/subdir2");
+ mkdir("$data_dir/ident_inc");
+ mkdir("$data_dir/ident_inc_if");
+ mkdir("$data_dir/ident_pos");
+
+ # Add include data
+ add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf");
+ $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar");
+
+ $ident_expected .= add_ident_line($node, "$ident_file", "test a b");
+
+ add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar");
+ # include is relative to current path
+ add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar");
+
+ # include_if_exists data
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none");
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some");
+ $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar");
+
+ # include_dir data
+ add_ident_line($node, "$ident_file", "include_dir ../ident_inc");
+ add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included");
+ $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar");
+ $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar");
+
+ $node->restart;
+ $node->connect_ok('dbname=postgres',
+ 'Connection ok after generating valid auth files');
+
+ return ($hba_expected, $ident_expected);
+}
+
+# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal
+# entries to allow authentication.
+sub reset_auth_files
+{
+ my $node = shift;
+
+ unlink("$data_dir/$hba_file");
+ unlink("$data_dir/$ident_file");
+
+ %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+ return add_hba_line($node, "$hba_file", 'local all all trust');
+}
+
+# Generate a list of expected error regex for the given array of error
+# conditions, as generated by add_hba_line/add_ident_line with an err_str.
+#
+# 2 regex are generated per array entry: one for the given err_str, and one for
+# the expected line in the specific file. Since all lines are independant,
+# there's no guarantee that a specific failure regex and the per-line regex
+# will match the same error. Calling code should add at least one test with a
+# single error to make sure that the line number / file name is correct.
+#
+# On top of that, an extra line is generated for the general failure to process
+# the main auth file.
+sub generate_log_err_patterns
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $is_hba_err = shift;
+ my @errors;
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ push @errors, qr/$err_str/;
+
+ # Context messages with the file / line location aren't always emitted
+ if ($err_str !~ /maximum nesting depth exceeded/ and
+ $err_str !~ /could not open secondary authentication file/)
+ {
+ push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/
+ }
+ }
+
+ push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err);
+
+ return \@errors;
+}
+
+# Generate the expected output for the auth file view error reporting (file
+# name, file line, error), for the given array of error conditions, as
+# generated generated by add_hba_line/add_ident_line with an err_str.
+sub generate_log_err_rows
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $exp_rows = '';
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ $exp_rows .= "\n" if ($exp_rows ne "");
+
+ # Unescape regex patterns if any
+ $err_str =~ s/\\([\(\)])/$1/g;
+ $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str"
+ }
+
+ return $exp_rows;
+}
+
+# Reset the main auth files, append the given payload to the given config file,
+# and check that the instance cannot start, raising the expected error line(s).
+sub start_errors_like
+{
+ my $node = shift;
+ my $file = shift;
+ my $payload = shift;
+ my $pattern = shift;
+ my $should_fail = shift;
+
+ reset_auth_files($node);
+ $node->append_conf($file, $payload);
+
+ unlink($node->logfile);
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir,
+ '-l', $node->logfile, 'start');
+
+ if ($should_fail)
+ {
+ ok($ret != 0, "Cannot start postgres with faulty $file");
+ }
+ else
+ {
+ ok($ret == 0, "postgres can start with faulty $file");
+ }
+
+ my $log_contents = slurp_file($node->logfile);
+
+ foreach (@{$pattern})
+ {
+ like($log_contents,
+ $_,
+ "Expected failure found in the logs");
+ }
+
+ if (not $should_fail)
+ {
+ # We can't simply call $node->stop here as the call is optimized out
+ # when the server isn't started with $node->start.
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D',
+ $data_dir, 'stop', '-m', 'fast');
+ ok($ret == 0, "Could stop postgres");
+ }
+}
+
+# We should be able to connect, and see an empty pg_ident.conf
+is($node->psql(
+ 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'),
+ qq(0),
+ 'pg_ident.conf is empty');
+
+############################################
+# part 1, test view reporting for valid data
+############################################
+my ($exp_hba, $exp_ident) = generate_valid_auth_files($node);
+
+$node->connect_ok('dbname=postgres', 'Connection still ok');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_hba_file_rules'),
+ qq($exp_hba),
+ 'pg_hba_file_rules content is expected');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_ident_file_mappings'),
+ qq($exp_ident),
+ 'pg_ident_file_mappings content is expected');
+
+#############################################
+# part 2, test log reporting for invalid data
+#############################################
+reset_auth_files($node);
+$node->restart('fast');
+$node->connect_ok('dbname=postgres',
+ 'Connection ok after resetting auth files');
+
+$node->stop('fast');
+
+start_errors_like($node, $hba_file, "include ../not_a_file",
+ [
+ qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file
+mkdir("$data_dir/hba_inc_fail");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token");
+start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail",
+ [
+ qr/invalid connection type "not_a_token"/,
+ qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file with nested inclusion
+unlink("$data_dir/hba_inc_fail/inc_dir.conf");
+my @hba_raw_errors_step1;
+
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1");
+
+add_hba_line($node, "hba_inc_fail/file1", "include file2");
+add_hba_line($node, "hba_inc_fail/file2", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file2", "include file3");
+
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3",
+ "local all all zuul",
+ 'invalid authentication method "zuul"');
+
+start_errors_like(
+ $node, $hba_file, "include_dir ../hba_inc_fail",
+ generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @hba_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local",
+ "end-of-line before database specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local,host",
+ "multiple values specified for connection type");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all",
+ "end-of-line before role specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all all",
+ "end-of-line before authentication method");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "host all all test/42",
+ 'specifying both host name and CIDR mask is invalid: "test/42"');
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ 'local @dbnames_fails.conf all reject',
+ "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory");
+
+add_hba_line($node, "hba_if_exists.conf", "include recurse.conf");
+push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf",
+ "include recurse.conf",
+ 'could not open configuration file "recurse.conf": maximum nesting depth exceeded');
+
+# Generate the regex for the expected errors in the logs. There's no guarantee
+# that the generated "line X of file..." will be emitted for the expected line,
+# but previous tests already ensured that the correct line number / file name
+# was emitted, so ensuring that there's an error in all expected lines is
+# enough here.
+my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2,
+ 1);
+
+# Not an error, but it should raise a message in the logs. Manually add an
+# extra log message to detect
+add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none");
+push @{$expected_errors},
+ qr/skipping missing authentication file "$data_dir\/if_exists_none"/;
+
+start_errors_like(
+ $node, $hba_file, "include_if_exists ../hba_if_exists.conf",
+ $expected_errors, 1);
+
+# Mostly the same, but for ident files
+reset_auth_files($node);
+
+my @ident_raw_errors_step1;
+
+# include_dir, single included file with nested inclusion
+mkdir("$data_dir/ident_inc_fail");
+add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1");
+
+add_ident_line($node, "ident_inc_fail/file1", "include file2");
+add_ident_line($node, "ident_inc_fail/file2", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file2", "include file3");
+
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3",
+ "failmap /(fail postgres",
+ 'invalid regular expression "\(fail": parentheses \(\) not balanced');
+
+start_errors_like(
+ $node, $ident_file, "include_dir ../ident_inc_fail",
+ generate_log_err_patterns($node, \@ident_raw_errors_step1, 0),
+ 0);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @ident_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map",
+ "missing entry at end of line");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2",
+ "multiple values in ident field");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf",
+ 'map @osnames_fails.conf postgres',
+ "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory");
+
+add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf",
+ 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded');
+
+start_errors_like(
+ $node, $ident_file, "include_if_exists ../ident_if_exists.conf",
+ # There's no guarantee that the generated "line X of file..." will be
+ # emitted for the expected line, but previous tests already ensured that
+ # the correct line number / file name was emitted, so ensuring that there's
+ # an error in all expected lines is enough here.
+ generate_log_err_patterns($node, \@ident_raw_errors_step2, 0),
+ 0);
+
+#####################################################
+# part 3, test reporting of various error scenario
+# NOTE: this will be bypassed -DEXEC_BACKEND or win32
+#####################################################
+reset_auth_files($node);
+
+$node->start;
+$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset');
+
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_hba_file_rules');
+
+add_ident_line($node, $ident_file, '');
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_ident_file_mappings');
+
+# The instance could be restarted and no error is detected. Now check if the
+# build is compatible with the view error reporting (EXEC_BACKEND / win32 will
+# fail when trying to connect as they always rely on the current auth files
+# content)
+my @hba_raw_errors;
+
+push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+my ($stdout, $stderr);
+my $cmdret = $node->psql('postgres', 'SELECT 1',
+ stdout => \$stdout, stderr => \$stderr);
+
+if ($cmdret != 0)
+{
+ # Connection failed. Bail out, but make sure to raise a failure if it
+ # didn't fail for the expected hba file modification.
+ like($stderr,
+ qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/,
+ "Connection failed due to loading an invalid hba file");
+
+ done_testing();
+ diag("Build not compatible with auth file view error reporting, bail out.\n");
+ exit;
+}
+
+# Combine errors generated at step 2, in the same order.
+$node->append_conf($hba_file, "include_dir ../hba_inc_fail");
+push @hba_raw_errors, @hba_raw_errors_step1;
+
+$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf");
+push @hba_raw_errors, @hba_raw_errors_step2;
+
+my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules'
+ . ' WHERE error IS NOT NULL ORDER BY rule_number'),
+ qq($hba_expected),
+ 'Detected all error in hba file');
+
+# and do the same for pg_ident
+my @ident_raw_errors;
+
+push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+$node->append_conf($ident_file, "include_dir ../ident_inc_fail");
+push @ident_raw_errors, @ident_raw_errors_step1;
+
+$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf");
+push @ident_raw_errors, @ident_raw_errors_step2;
+
+my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings'
+ . ' WHERE error IS NOT NULL ORDER BY mapping_number'),
+ qq($ident_expected),
+ 'Detected all error in ident file');
+
+done_testing();
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 178e536e21..3d8f182674 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname,
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
a.line_number,
a.type,
a.database,
@@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
a.line_number,
a.map_name,
a.sys_name,
a.pg_username,
a.error
- FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.37.0
--mkp7bqke54pr5bne
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v12-0003-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v9 2/3] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-05-30 03:15 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-05-30 03:15 UTC (permalink / raw)
pg_hba.conf file now has support for "include", "include_dir" and
"include_if_exists" directives, which work similarly to the same directives in
the postgresql.conf file.
This fixes a possible crash if a secondary file tries to include itself as
there's now a nesting depth check in the inclusion code path, same as the
postgresql.conf.
Many regression tests added to cover both the new directives, but also error
detection for the whole pg_hba / pg_ident files.
Catversion is bumped.
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/client-auth.sgml | 86 ++-
doc/src/sgml/system-views.sgml | 22 +-
src/backend/libpq/hba.c | 483 ++++++++++---
src/backend/libpq/pg_hba.conf.sample | 25 +-
src/backend/libpq/pg_ident.conf.sample | 15 +-
src/backend/utils/adt/hbafuncs.c | 43 +-
src/backend/utils/misc/guc-file.l | 250 +++----
src/include/catalog/pg_proc.dat | 12 +-
src/include/libpq/hba.h | 5 +-
src/include/utils/guc.h | 2 +
.../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++
src/test/regress/expected/rules.out | 6 +-
12 files changed, 1324 insertions(+), 282 deletions(-)
create mode 100644 src/test/authentication/t/003_file_inclusion.pl
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index 433759928b..e4eacab4a5 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,23 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication
+ record. Inclusion directives specify files that can be included, which
+ contains additional records. The records will be inserted in lieu of the
+ inclusion records. Those records only contains two fields: the
+ <literal>include</literal>, <literal>include_if_exists</literal> or
+ <literal>include_dir</literal> directive and the file or directory to be
+ included. The file or directory can be a relative of absolute path, and can
+ be double quoted if needed. For the <literal>include_dir</literal> form,
+ all files not starting with a <literal>.</literal> and ending with
+ <literal>.conf</literal> will be included. Multiple files within an include
+ directory are processed in file name order (according to C locale rules,
+ i.e., numbers before letters, and uppercase letters before lowercase ones).
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,21 +118,57 @@
<para>
A record can have several formats:
<synopsis>
-local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+include <replaceable>file</replaceable>
+include_if_exists <replaceable>file</replaceable>
+include_dir <replaceable>directory</replaceable>
+local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
</synopsis>
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_if_exists</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file if the
+ file exists and can be read. Otherwise, a message will be logged to
+ indicate that the file is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_dir</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of all the files found in
+ the directory, if they don't start with a <literal>.</literal> and end
+ with <literal>.conf</literal>, processed in file name order (according
+ to C locale rules, i.e., numbers before letters, and uppercase letters
+ before lowercase ones).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -837,8 +888,10 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
+<replaceable>include_dir</replaceable> <replaceable>directory</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -849,6 +902,11 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ As for <filename>pg_hba.conf</filename>, the lines in this file can either
+ be inclusion directives or user name map records, and follow the same
+ rules.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 1d619427c1..4e63dec74c 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -1003,12 +1003,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule the given <literal>file_name</literal>
</para></entry>
</row>
@@ -1153,12 +1162,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this mapping
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_ident.conf</filename>
+ Line number of this mapping in the given <literal>file_name</literal>
</para></entry>
</row>
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index 857b9e5eb2..49a8c56f41 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -22,6 +22,7 @@
#include <sys/param.h>
#include <sys/socket.h>
#include <netdb.h>
+#include <sys/stat.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
@@ -69,6 +70,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -113,10 +120,22 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int depth,
+ int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
- const char *inc_filename, int elevel, char **err_msg);
+ const char *inc_filename, int depth, int elevel,
+ char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ bool strict, const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
+static char *process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth,
+ int elevel, MemoryContext linecxt,
+ List **tok_lines);
/*
@@ -303,7 +322,7 @@ copy_auth_token(AuthToken *in)
*/
static List *
next_field_expand(const char *filename, char **lineptr,
- int elevel, char **err_msg)
+ int depth, int elevel, char **err_msg)
{
char buf[MAX_TOKEN];
bool trailing_comma;
@@ -319,7 +338,7 @@ next_field_expand(const char *filename, char **lineptr,
/* Is this referencing a file? */
if (!initial_quote && buf[0] == '@' && buf[1] != '\0')
- tokens = tokenize_inc_file(tokens, filename, buf + 1,
+ tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1,
elevel, err_msg);
else
tokens = lappend(tokens, make_auth_token(buf, initial_quote));
@@ -347,6 +366,7 @@ static List *
tokenize_inc_file(List *tokens,
const char *outer_filename,
const char *inc_filename,
+ int depth,
int elevel,
char **err_msg)
{
@@ -356,39 +376,30 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
{
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
+ *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", *err_msg)));
+ return tokens;
}
- inc_file = AllocateFile(inc_fullname, "r");
- if (inc_file == NULL)
- {
- int save_errno = errno;
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true,
+ outer_filename, elevel, err_msg, &inc_fullname);
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
+ if (inc_file == NULL)
return tokens;
- }
/* There is possible recursion here if the file contains @ */
- linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
+ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1,
+ elevel);
FreeFile(inc_file);
pfree(inc_fullname);
@@ -426,11 +437,38 @@ tokenize_inc_file(List *tokens,
/*
* tokenize_auth_file
- * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a dedicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
+ int depth, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, depth,
+ elevel);
+
+ return linecxt;
+}
+
+/*
+ * Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see the struct definition
* in libpq/hba.h.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -439,30 +477,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int depth, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -515,7 +545,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
{
List *current_field;
- current_field = next_field_expand(filename, &lineptr,
+ current_field = next_field_expand(filename, &lineptr, depth,
elevel, &err_msg);
/* add field to line, unless we are at EOL or comment start */
if (current_field != NIL)
@@ -523,29 +553,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ AuthToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
+
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
+
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, true,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
+ else if (strcmp(first->string, "include_dir") == 0)
+ {
+ char **filenames;
+ char *dir_name = second->string;
+ int num_filenames;
+ StringInfoData err_buf;
+
+ filenames = GetDirConfFiles(dir_name, filename, elevel,
+ &num_filenames, &err_msg);
+
+ if (!filenames)
+ {
+ /* We have the error in err_msg, simply process it */
+ goto process_line;
+ }
+
+ initStringInfo(&err_buf);
+ for (int i = 0; i < num_filenames; i++)
+ {
+ /*
+ * err_msg is used here as a temp buffer, it will be
+ * overwritten at the end of the loop with the
+ * cumulated errors, if any.
+ */
+ err_msg = process_included_authfile(filenames[i], true,
+ filename, depth + 1, elevel,
+ linecxt, tok_lines);
+
+ /* Cumulate errors if any. */
+ if (err_msg)
+ {
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfoString(&err_buf, err_msg);
+ }
+ }
+
+ /*
+ * If there were no errors, the line is fully processed, bypass
+ * the general TokenizedAuthLine processing.
+ */
+ if (err_buf.len == 0)
+ goto next_line;
+
+ /* Otherwise, process the cumulated errors, if any. */
+ err_msg = err_buf.data;
+ }
+ else if (strcmp(first->string, "include_if_exists") == 0)
+ {
+ char *inc_filename;
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, false,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
}
+process_line:
+ /*
+ * General processing: report the error if any and emit line to the
+ * TokenizedAuthLine
+ */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -860,7 +988,7 @@ do { \
errmsg("authentication option \"%s\" is only valid for authentication methods %s", \
optname, _(validmethods)), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \
optname, validmethods); \
return false; \
@@ -880,7 +1008,7 @@ do { \
errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname); \
return NULL; \
@@ -903,7 +1031,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("missing entry at end of line"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("missing entry at end of line"); \
return NULL; \
} \
@@ -916,7 +1044,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("multiple values in ident field"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("multiple values in ident field"); \
return NULL; \
} \
@@ -939,6 +1067,7 @@ HbaLine *
parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
{
int line_num = tok_line->line_num;
+ char *file_name = tok_line->file_name;
char **err_msg = &tok_line->err_msg;
char *str;
struct addrinfo *gai_result;
@@ -953,6 +1082,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -967,7 +1097,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for connection type"),
errhint("Specify exactly one connection type per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for connection type";
return NULL;
}
@@ -995,7 +1125,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("hostssl record cannot match because SSL is disabled"),
errhint("Set ssl = on in postgresql.conf."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostssl record cannot match because SSL is disabled";
}
#else
@@ -1003,7 +1133,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("hostssl record cannot match because SSL is not supported by this build"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostssl record cannot match because SSL is not supported by this build";
#endif
}
@@ -1015,7 +1145,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("hostgssenc record cannot match because GSSAPI is not supported by this build"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostgssenc record cannot match because GSSAPI is not supported by this build";
#endif
}
@@ -1036,7 +1166,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid connection type \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid connection type \"%s\"", token->string);
return NULL;
}
@@ -1049,7 +1179,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before database specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before database specification";
return NULL;
}
@@ -1069,7 +1199,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before role specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before role specification";
return NULL;
}
@@ -1091,7 +1221,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before IP address specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before IP address specification";
return NULL;
}
@@ -1103,7 +1233,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for host address"),
errhint("Specify one address range per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for host address";
return NULL;
}
@@ -1162,7 +1292,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid IP address \"%s\": %s",
str, gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid IP address \"%s\": %s",
str, gai_strerror(ret));
if (gai_result)
@@ -1182,7 +1312,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("specifying both host name and CIDR mask is invalid: \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("specifying both host name and CIDR mask is invalid: \"%s\"",
token->string);
return NULL;
@@ -1196,7 +1326,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid CIDR mask in address \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid CIDR mask in address \"%s\"",
token->string);
return NULL;
@@ -1216,7 +1346,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("end-of-line before netmask specification"),
errhint("Specify an address range in CIDR notation, or provide a separate netmask."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before netmask specification";
return NULL;
}
@@ -1227,7 +1357,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("multiple values specified for netmask"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for netmask";
return NULL;
}
@@ -1242,7 +1372,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid IP mask \"%s\": %s",
token->string, gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid IP mask \"%s\": %s",
token->string, gai_strerror(ret));
if (gai_result)
@@ -1261,7 +1391,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("IP address and mask do not match"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "IP address and mask do not match";
return NULL;
}
@@ -1277,7 +1407,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before authentication method"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before authentication method";
return NULL;
}
@@ -1289,7 +1419,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for authentication type"),
errhint("Specify exactly one authentication type per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for authentication type";
return NULL;
}
@@ -1326,7 +1456,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("MD5 authentication is not supported when \"db_user_namespace\" is enabled"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "MD5 authentication is not supported when \"db_user_namespace\" is enabled";
return NULL;
}
@@ -1367,7 +1497,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid authentication method \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid authentication method \"%s\"",
token->string);
return NULL;
@@ -1380,7 +1510,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid authentication method \"%s\": not supported by this build",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid authentication method \"%s\": not supported by this build",
token->string);
return NULL;
@@ -1402,7 +1532,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("gssapi authentication is not supported on local sockets"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "gssapi authentication is not supported on local sockets";
return NULL;
}
@@ -1414,7 +1544,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("peer authentication is only supported on local sockets"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "peer authentication is only supported on local sockets";
return NULL;
}
@@ -1432,7 +1562,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cert authentication is only supported on hostssl connections"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cert authentication is only supported on hostssl connections";
return NULL;
}
@@ -1482,7 +1612,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("authentication option not in name=value format: %s", token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("authentication option not in name=value format: %s",
token->string);
return NULL;
@@ -1526,7 +1656,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix";
return NULL;
}
@@ -1537,7 +1667,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set";
return NULL;
}
@@ -1553,7 +1683,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cannot use ldapsearchattribute together with ldapsearchfilter"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cannot use ldapsearchattribute together with ldapsearchfilter";
return NULL;
}
@@ -1570,7 +1700,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("list of RADIUS servers cannot be empty"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "list of RADIUS servers cannot be empty";
return NULL;
}
@@ -1581,7 +1711,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("list of RADIUS secrets cannot be empty"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "list of RADIUS secrets cannot be empty";
return NULL;
}
@@ -1600,7 +1730,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiussecrets),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS secrets (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiussecrets),
list_length(parsedline->radiusservers));
@@ -1616,7 +1746,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiusports),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS ports (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiusports),
list_length(parsedline->radiusservers));
@@ -1632,7 +1762,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiusidentifiers),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS identifiers (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiusidentifiers),
list_length(parsedline->radiusservers));
@@ -1667,6 +1797,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg)
{
int line_num = hbaline->linenumber;
+ char *file_name = hbaline->sourcefile;
#ifdef USE_LDAP
hbaline->ldapscope = LDAP_SCOPE_SUBTREE;
@@ -1690,7 +1821,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientcert can only be configured for \"hostssl\" rows"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientcert can only be configured for \"hostssl\" rows";
return false;
}
@@ -1707,7 +1838,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientcert only accepts \"verify-full\" when using \"cert\" authentication"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientcert can only be set to \"verify-full\" when using \"cert\" authentication";
return false;
}
@@ -1720,7 +1851,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid value for clientcert: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
}
@@ -1732,7 +1863,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientname can only be configured for \"hostssl\" rows"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientname can only be configured for \"hostssl\" rows";
return false;
}
@@ -1751,7 +1882,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid value for clientname: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
}
@@ -1837,7 +1968,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid ldapscheme value: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
hbaline->ldapscheme = pstrdup(val);
}
else if (strcmp(name, "ldapserver") == 0)
@@ -1855,7 +1986,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid LDAP port number: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid LDAP port number: \"%s\"", val);
return false;
}
@@ -1949,7 +2080,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS server list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -1968,7 +2099,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not translate RADIUS server name \"%s\" to address: %s",
(char *) lfirst(l), gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
if (gai_result)
pg_freeaddrinfo_all(hints.ai_family, gai_result);
@@ -1997,7 +2128,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS port list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid RADIUS port number: \"%s\"", val);
return false;
}
@@ -2010,7 +2141,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid RADIUS port number: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2033,7 +2164,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS secret list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2055,7 +2186,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS identifiers list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2069,7 +2200,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("unrecognized authentication option name: \"%s\"",
name),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("unrecognized authentication option name: \"%s\"",
name);
return false;
@@ -2217,7 +2348,7 @@ load_hba(void)
return false;
}
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
@@ -2288,6 +2419,137 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ if (strict)
+ {
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ }
+ else
+ {
+ Assert(kind == IncludedAuthFile);
+ ereport(LOG,
+ (errmsg("skipping missing authentication file \"%s\"",
+ *inc_fullname)));
+ }
+
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
+
+/*
+ * Try to open an included file, and tokenize it using the given context.
+ * Returns NULL if no error happens during tokenization, otherwise the error.
+ */
+static char *
+process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth, int elevel,
+ MemoryContext linecxt, List **tok_lines)
+{
+ char *inc_fullname;
+ FILE *inc_file;
+ char *err_msg = NULL;
+
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
+ {
+ char *err_msg;
+
+ err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", err_msg)));
+ return err_msg;
+ }
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict,
+ outer_filename, elevel, &err_msg, &inc_fullname);
+
+ if (inc_file == NULL)
+ {
+ if (strict)
+ {
+ /* open_inc_file should have reported an error. */
+ Assert(err_msg != NULL);
+ return err_msg;
+ }
+ else
+ return NULL;
+ }
+ else
+ {
+ /* No error message should have been reported. */
+ Assert(err_msg == NULL);
+ }
+
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, depth, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ return NULL;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
@@ -2306,6 +2568,7 @@ load_hba(void)
IdentLine *
parse_ident_line(TokenizedAuthLine *tok_line, int elevel)
{
+ char *file_name = tok_line->file_name;
int line_num = tok_line->line_num;
char **err_msg = &tok_line->err_msg;
ListCell *field;
@@ -2366,7 +2629,7 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid regular expression \"%s\": %s",
parsedline->ident_user + 1, errstr),
errcontext("line %d of configuration file \"%s\"",
- line_num, IdentFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid regular expression \"%s\": %s",
parsedline->ident_user + 1, errstr);
@@ -2601,7 +2864,7 @@ load_ident(void)
return false;
}
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..7433050112 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,16 +9,27 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
-# local DATABASE USER METHOD [OPTIONS]
-# host DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# local DATABASE USER METHOD [OPTIONS]
+# host DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..8e3fa29135 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,23 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
-# MAPNAME SYSTEM-USERNAME PG-USERNAME
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index c9be4bff1f..15326a01e2 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,12 +26,12 @@
static ArrayType *get_hba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int rule_number, int lineno, HbaLine *hba,
- const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int mapping_number, int lineno, IdentLine *ident,
- const char *err_msg);
+ int mapping_number, const char *filename,
+ int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
@@ -159,7 +159,7 @@ get_hba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 10
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line
@@ -168,7 +168,8 @@ get_hba_options(HbaLine *hba)
* tuple_store: where to store data
* tupdesc: tuple descriptor for the view
* rule_number: unique rule identifier among all valid rules
- * lineno: pg_hba.conf line number (must always be valid)
+ * filename: name of the file containing that line
+ * lineno: line number in that file (must always be valid)
* hba: parsed line data (can be NULL, in which case err_msg should be set)
* err_msg: error message (NULL if none)
*
@@ -177,7 +178,7 @@ get_hba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int rule_number, int lineno, HbaLine *hba,
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -202,6 +203,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -345,7 +348,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -386,7 +389,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open configuration file \"%s\": %m",
HbaFileName)));
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -407,8 +410,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
rule_number++;
- fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -445,7 +448,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
}
/* Number of columns in pg_hba_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -454,7 +457,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
* tuple_store: where to store data
* tupdesc: tuple descriptor for the view
* mapping_number: unique rule identifier among all valid rules
- * lineno: pg_ident.conf line number (must always be valid)
+ * filename: name of the file containing that line
+ * lineno: line number in that file (must always be valid)
* ident: parsed line data (can be NULL, in which case err_msg should be set)
* err_msg: error message (NULL if none)
*
@@ -463,8 +467,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int mapping_number, int lineno, IdentLine *ident,
- const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -482,6 +486,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -494,7 +500,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -534,7 +540,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open usermap file \"%s\": %m",
IdentFileName)));
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -556,7 +562,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
mapping_number++;
fill_ident_line(tuple_store, tupdesc, mapping_number,
- tok_line->line_num, identline, tok_line->err_msg);
+ tok_line->file_name, tok_line->line_num, identline,
+ tok_line->err_msg);
}
/* Free tokenizer memory */
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index ce5633844c..b3e18e48cf 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -700,6 +700,122 @@ GUC_flex_fatal(const char *msg)
return 0; /* keep compiler quiet */
}
+/*
+ * Returns the list of config files located in a directory, in alphabetical
+ * order.
+ *
+ * We don't check for recursion or too-deep nesting depth here, its up to the
+ * caller to take care of that.
+ */
+char **
+GetDirConfFiles(const char *includedir, const char *calling_file, int elevel,
+ int *num_filenames, char **err_msg)
+{
+ char *directory;
+ DIR *d;
+ struct dirent *de;
+ char **filenames;
+ int size_filenames;
+
+ /*
+ * Reject directory name that is all-blank (including empty), as that
+ * leads to confusion --- we'd read the containing directory, typically
+ * resulting in recursive inclusion of the same file(s).
+ */
+ if (strspn(includedir, " \t\r\n") == strlen(includedir))
+ {
+ ereport(elevel,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("empty configuration directory name: \"%s\"",
+ includedir)));
+ *err_msg = "empty configuration directory name";
+ return NULL;
+ }
+
+ directory = AbsoluteConfigLocation(includedir, calling_file);
+ d = AllocateDir(directory);
+ if (d == NULL)
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not open configuration directory \"%s\": %m",
+ directory)));
+ *err_msg = psprintf("could not open directory \"%s\"", directory);
+ filenames = NULL;
+ goto cleanup;
+ }
+
+ /*
+ * Read the directory and put the filenames in an array, so we can sort
+ * them prior to caller processing the contents.
+ */
+ size_filenames = 32;
+ filenames = (char **) palloc(size_filenames * sizeof(char *));
+ *num_filenames = 0;
+
+ while ((de = ReadDir(d, directory)) != NULL)
+ {
+ struct stat st;
+ char filename[MAXPGPATH];
+
+ /*
+ * Only parse files with names ending in ".conf". Explicitly reject
+ * files starting with ".". This excludes things like "." and "..",
+ * as well as typical hidden files, backup files, and editor debris.
+ */
+ if (strlen(de->d_name) < 6)
+ continue;
+ if (de->d_name[0] == '.')
+ continue;
+ if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
+ continue;
+
+ join_path_components(filename, directory, de->d_name);
+ canonicalize_path(filename);
+ if (stat(filename, &st) == 0)
+ {
+ /* Ignore directories. */
+ if (S_ISDIR(st.st_mode))
+ continue;
+
+ /* Add file to array, increasing its size in blocks of 32 */
+ if (*num_filenames >= size_filenames)
+ {
+ size_filenames += 32;
+ filenames = (char **) repalloc(filenames,
+ size_filenames * sizeof(char *));
+ }
+ filenames[*num_filenames] = pstrdup(filename);
+ (*num_filenames)++;
+ }
+ else
+ {
+ /*
+ * stat does not care about permissions, so the most likely reason
+ * a file can't be accessed now is if it was removed between the
+ * directory listing and now.
+ */
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not stat file \"%s\": %m",
+ filename)));
+ *err_msg = psprintf("could not stat file \"%s\"", filename);
+ pfree(filenames);
+ filenames = NULL;
+ goto cleanup;
+ }
+ }
+
+ if (*num_filenames > 0)
+ qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp);
+
+cleanup:
+ if (d)
+ FreeDir(d);
+ pfree(directory);
+ return filenames;
+}
+
/*
* Read and parse a single configuration file. This function recurses
* to handle "include" directives.
@@ -961,138 +1077,32 @@ ParseConfigDirectory(const char *includedir,
ConfigVariable **head_p,
ConfigVariable **tail_p)
{
- char *directory;
- DIR *d;
- struct dirent *de;
+ char *err_msg;
char **filenames;
int num_filenames;
- int size_filenames;
- bool status;
- /*
- * Reject directory name that is all-blank (including empty), as that
- * leads to confusion --- we'd read the containing directory, typically
- * resulting in recursive inclusion of the same file(s).
- */
- if (strspn(includedir, " \t\r\n") == strlen(includedir))
- {
- ereport(elevel,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("empty configuration directory name: \"%s\"",
- includedir)));
- record_config_file_error("empty configuration directory name",
- calling_file, calling_lineno,
- head_p, tail_p);
- return false;
- }
+ filenames = GetDirConfFiles(includedir, calling_file, elevel,
+ &num_filenames, &err_msg);
- /*
- * We don't check for recursion or too-deep nesting depth here; the
- * subsequent calls to ParseConfigFile will take care of that.
- */
-
- directory = AbsoluteConfigLocation(includedir, calling_file);
- d = AllocateDir(directory);
- if (d == NULL)
+ if (!filenames)
{
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open configuration directory \"%s\": %m",
- directory)));
- record_config_file_error(psprintf("could not open directory \"%s\"",
- directory),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
-
- /*
- * Read the directory and put the filenames in an array, so we can sort
- * them prior to processing the contents.
- */
- size_filenames = 32;
- filenames = (char **) palloc(size_filenames * sizeof(char *));
- num_filenames = 0;
-
- while ((de = ReadDir(d, directory)) != NULL)
- {
- struct stat st;
- char filename[MAXPGPATH];
-
- /*
- * Only parse files with names ending in ".conf". Explicitly reject
- * files starting with ".". This excludes things like "." and "..",
- * as well as typical hidden files, backup files, and editor debris.
- */
- if (strlen(de->d_name) < 6)
- continue;
- if (de->d_name[0] == '.')
- continue;
- if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
- continue;
-
- join_path_components(filename, directory, de->d_name);
- canonicalize_path(filename);
- if (stat(filename, &st) == 0)
- {
- if (!S_ISDIR(st.st_mode))
- {
- /* Add file to array, increasing its size in blocks of 32 */
- if (num_filenames >= size_filenames)
- {
- size_filenames += 32;
- filenames = (char **) repalloc(filenames,
- size_filenames * sizeof(char *));
- }
- filenames[num_filenames] = pstrdup(filename);
- num_filenames++;
- }
- }
- else
- {
- /*
- * stat does not care about permissions, so the most likely reason
- * a file can't be accessed now is if it was removed between the
- * directory listing and now.
- */
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not stat file \"%s\": %m",
- filename)));
- record_config_file_error(psprintf("could not stat file \"%s\"",
- filename),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
+ record_config_file_error(err_msg, calling_file, calling_lineno, head_p,
+ tail_p);
+ return false;
}
- if (num_filenames > 0)
+ for (int i = 0; i < num_filenames; i++)
{
- int i;
-
- qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp);
- for (i = 0; i < num_filenames; i++)
+ if (!ParseConfigFile(filenames[i], true,
+ calling_file, calling_lineno,
+ depth, elevel,
+ head_p, tail_p))
{
- if (!ParseConfigFile(filenames[i], true,
- calling_file, calling_lineno,
- depth, elevel,
- head_p, tail_p))
- {
- status = false;
- goto cleanup;
- }
+ return false;
}
}
- status = true;
-cleanup:
- if (d)
- FreeDir(d);
- pfree(directory);
- return status;
+ return true;
}
/*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index b284212bd3..4470011977 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6128,16 +6128,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '6250', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,text,text,text}',
- proargmodes => '{o,o,o,o,o,o}',
- proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index 90036f7bcd..0ea100d1b8 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -155,6 +156,7 @@ typedef struct AuthToken
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of AuthTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
@@ -174,6 +176,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel);
extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel);
extern bool pg_isblank(const char c);
extern MemoryContext tokenize_auth_file(const char *filename, FILE *file,
- List **tok_lines, int elevel);
+ List **tok_lines, int depth,
+ int elevel);
#endif /* HBA_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index e734493a48..1a3ab6306d 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -145,6 +145,8 @@ typedef struct ConfigVariable
struct ConfigVariable *next;
} ConfigVariable;
+extern char **GetDirConfFiles(const char *includedir, const char *calling_file,
+ int elevel, int *num_filenames, char **err_msg);
extern bool ParseConfigFile(const char *config_file, bool strict,
const char *calling_file, int calling_lineno,
int depth, int elevel,
diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl
new file mode 100644
index 0000000000..8eae72b8d4
--- /dev/null
+++ b/src/test/authentication/t/003_file_inclusion.pl
@@ -0,0 +1,657 @@
+
+# Copyright (c) 2021-2022, PostgreSQL Global Development Group
+
+# Set of tests for authentication and pg_hba.conf inclusion.
+# This test can only run with Unix-domain sockets.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Time::HiRes qw(usleep);
+use IPC::Run qw(pump finish timer);
+use Data::Dumper;
+
+if (!$use_unix_sockets)
+{
+ plan skip_all =>
+ "authentication tests cannot run without Unix-domain sockets";
+}
+
+# stores the current line counter for each file. hba_rule and ident_rule are
+# fake file names used for the global rule number for each auth view.
+my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+my $hba_file = 'subdir1/pg_hba_custom.conf';
+my $ident_file = 'subdir2/pg_ident_custom.conf';
+
+# Initialize primary node
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+$node->start;
+
+my $data_dir = $node->data_dir;
+
+# Normalize the data directory for Windows
+$data_dir =~ s/\/\.\//\//g; # reduce /./ to /
+$data_dir =~ s/\/\//\//g; # reduce // to /
+$data_dir =~ s/\/$//; # remove trailing /
+
+
+# Add the given payload to the given relative HBA file of the given node.
+# This function maintains the %cur_line metadata, so it has to be called in the
+# expected inclusion evaluation order in order to keep it in sync.
+#
+# If the payload starts with "include" or "ignore", the function doesn't
+# increase the general hba rule number.
+#
+# If an err_str is provided, it returns an arrayref containing the provided
+# filename, the current line number in that file and the provided err_str. The
+# err_str has to be a valid regex string.
+# Otherwise it only returns the line number of the payload in the wanted file.
+# This function has to be called in the expected inclusion evaluation order to
+# keep the %cur_line information in sync.
+sub add_hba_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'hba_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_hba_file_rules line
+ @tokens = split(/ /, $payload);
+ $tokens[1] = '{' . $tokens[1] . '}'; # database
+ $tokens[2] = '{' . $tokens[2] . '}'; # user_name
+
+ # add empty address and netmask betweed user_name and auth_method
+ splice @tokens, 3, 0, '';
+ splice @tokens, 3, 0, '';
+
+ # append empty options and error
+ push @tokens, '';
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Add the given payload to the given relative ident file of the given node.
+# Same as add_hba_line but for pg_ident files
+sub add_ident_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'ident_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_ident_file_mappings line
+ @tokens = split(/ /, $payload);
+
+ # append empty error
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Delete pg_hba.conf from the given node, add various entries to test the
+# include infrastructure and then execute a reload to refresh it.
+sub generate_valid_auth_files
+{
+ my $node = shift;
+ my $hba_expected = '';
+ my $ident_expected = '';
+
+ # customise main auth file names
+ $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'");
+ $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'");
+
+ # and make original ones invalid to be sure they're not used anywhere
+ $node->append_conf('pg_hba.conf', "some invalid line");
+ $node->append_conf('pg_ident.conf', "some invalid line");
+
+ # pg_hba stuff
+ mkdir("$data_dir/subdir1");
+ mkdir("$data_dir/hba_inc");
+ mkdir("$data_dir/hba_inc_if");
+ mkdir("$data_dir/hba_pos");
+
+ # Make sure we will still be able to connect
+ $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust');
+
+ # Add include data
+ add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf");
+ $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject");
+
+ $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject");
+
+ add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject");
+ # include is relative to current path
+ add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject");
+
+ # include_if_exists data
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none");
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some");
+ $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject");
+
+ # include_dir data
+ add_hba_line($node, "$hba_file", "include_dir ../hba_inc");
+ add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included");
+ $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject");
+ $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject");
+
+ # secondary auth file
+ add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject');
+ $node->append_conf('dbnames.conf', "db1");
+ $node->append_conf('dbnames.conf', "db3");
+ $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1)
+ . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1)
+ . '|local|{db1,db3}|{all}|||reject||';
+
+ # pg_ident stuff
+ mkdir("$data_dir/subdir2");
+ mkdir("$data_dir/ident_inc");
+ mkdir("$data_dir/ident_inc_if");
+ mkdir("$data_dir/ident_pos");
+
+ # Add include data
+ add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf");
+ $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar");
+
+ $ident_expected .= add_ident_line($node, "$ident_file", "test a b");
+
+ add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar");
+ # include is relative to current path
+ add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar");
+
+ # include_if_exists data
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none");
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some");
+ $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar");
+
+ # include_dir data
+ add_ident_line($node, "$ident_file", "include_dir ../ident_inc");
+ add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included");
+ $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar");
+ $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar");
+
+ $node->restart;
+ $node->connect_ok('dbname=postgres',
+ 'Connection ok after generating valid auth files');
+
+ return ($hba_expected, $ident_expected);
+}
+
+# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal
+# entries to allow authentication.
+sub reset_auth_files
+{
+ my $node = shift;
+
+ unlink("$data_dir/$hba_file");
+ unlink("$data_dir/$ident_file");
+
+ %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+ return add_hba_line($node, "$hba_file", 'local all all trust');
+}
+
+# Generate a list of expected error regex for the given array of error
+# conditions, as generated by add_hba_line/add_ident_line with an err_str.
+#
+# 2 regex are generated per array entry: one for the given err_str, and one for
+# the expected line in the specific file. Since all lines are independant,
+# there's no guarantee that a specific failure regex and the per-line regex
+# will match the same error. Calling code should add at least one test with a
+# single error to make sure that the line number / file name is correct.
+#
+# On top of that, an extra line is generated for the general failure to process
+# the main auth file.
+sub generate_log_err_patterns
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $is_hba_err = shift;
+ my @errors;
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ push @errors, qr/$err_str/;
+
+ # Context messages with the file / line location aren't always emitted
+ if ($err_str !~ /maximum nesting depth exceeded/ and
+ $err_str !~ /could not open secondary authentication file/)
+ {
+ push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/
+ }
+ }
+
+ push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err);
+
+ return \@errors;
+}
+
+# Generate the expected output for the auth file view error reporting (file
+# name, file line, error), for the given array of error conditions, as
+# generated generated by add_hba_line/add_ident_line with an err_str.
+sub generate_log_err_rows
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $exp_rows = '';
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ $exp_rows .= "\n" if ($exp_rows ne "");
+
+ # Unescape regex patterns if any
+ $err_str =~ s/\\([\(\)])/$1/g;
+ $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str"
+ }
+
+ return $exp_rows;
+}
+
+# Reset the main auth files, append the given payload to the given config file,
+# and check that the instance cannot start, raising the expected error line(s).
+sub start_errors_like
+{
+ my $node = shift;
+ my $file = shift;
+ my $payload = shift;
+ my $pattern = shift;
+ my $should_fail = shift;
+
+ reset_auth_files($node);
+ $node->append_conf($file, $payload);
+
+ unlink($node->logfile);
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir,
+ '-l', $node->logfile, 'start');
+
+ if ($should_fail)
+ {
+ ok($ret != 0, "Cannot start postgres with faulty $file");
+ }
+ else
+ {
+ ok($ret == 0, "postgres can start with faulty $file");
+ }
+
+ my $log_contents = slurp_file($node->logfile);
+
+ foreach (@{$pattern})
+ {
+ like($log_contents,
+ $_,
+ "Expected failure found in the logs");
+ }
+
+ if (not $should_fail)
+ {
+ # We can't simply call $node->stop here as the call is optimized out
+ # when the server isn't started with $node->start.
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D',
+ $data_dir, 'stop', '-m', 'fast');
+ ok($ret == 0, "Could stop postgres");
+ }
+}
+
+# We should be able to connect, and see an empty pg_ident.conf
+is($node->psql(
+ 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'),
+ qq(0),
+ 'pg_ident.conf is empty');
+
+############################################
+# part 1, test view reporting for valid data
+############################################
+my ($exp_hba, $exp_ident) = generate_valid_auth_files($node);
+
+$node->connect_ok('dbname=postgres', 'Connection still ok');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_hba_file_rules'),
+ qq($exp_hba),
+ 'pg_hba_file_rules content is expected');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_ident_file_mappings'),
+ qq($exp_ident),
+ 'pg_ident_file_mappings content is expected');
+
+#############################################
+# part 2, test log reporting for invalid data
+#############################################
+reset_auth_files($node);
+$node->restart('fast');
+$node->connect_ok('dbname=postgres',
+ 'Connection ok after resetting auth files');
+
+$node->stop('fast');
+
+start_errors_like($node, $hba_file, "include ../not_a_file",
+ [
+ qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file
+mkdir("$data_dir/hba_inc_fail");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token");
+start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail",
+ [
+ qr/invalid connection type "not_a_token"/,
+ qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file with nested inclusion
+unlink("$data_dir/hba_inc_fail/inc_dir.conf");
+my @hba_raw_errors_step1;
+
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1");
+
+add_hba_line($node, "hba_inc_fail/file1", "include file2");
+add_hba_line($node, "hba_inc_fail/file2", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file2", "include file3");
+
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3",
+ "local all all zuul",
+ 'invalid authentication method "zuul"');
+
+start_errors_like(
+ $node, $hba_file, "include_dir ../hba_inc_fail",
+ generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @hba_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local",
+ "end-of-line before database specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local,host",
+ "multiple values specified for connection type");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all",
+ "end-of-line before role specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all all",
+ "end-of-line before authentication method");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "host all all test/42",
+ 'specifying both host name and CIDR mask is invalid: "test/42"');
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ 'local @dbnames_fails.conf all reject',
+ "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory");
+
+add_hba_line($node, "hba_if_exists.conf", "include recurse.conf");
+push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf",
+ "include recurse.conf",
+ 'could not open configuration file "recurse.conf": maximum nesting depth exceeded');
+
+# Generate the regex for the expected errors in the logs. There's no guarantee
+# that the generated "line X of file..." will be emitted for the expected line,
+# but previous tests already ensured that the correct line number / file name
+# was emitted, so ensuring that there's an error in all expected lines is
+# enough here.
+my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2,
+ 1);
+
+# Not an error, but it should raise a message in the logs. Manually add an
+# extra log message to detect
+add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none");
+push @{$expected_errors},
+ qr/skipping missing authentication file "$data_dir\/if_exists_none"/;
+
+start_errors_like(
+ $node, $hba_file, "include_if_exists ../hba_if_exists.conf",
+ $expected_errors, 1);
+
+# Mostly the same, but for ident files
+reset_auth_files($node);
+
+my @ident_raw_errors_step1;
+
+# include_dir, single included file with nested inclusion
+mkdir("$data_dir/ident_inc_fail");
+add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1");
+
+add_ident_line($node, "ident_inc_fail/file1", "include file2");
+add_ident_line($node, "ident_inc_fail/file2", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file2", "include file3");
+
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3",
+ "failmap /(fail postgres",
+ 'invalid regular expression "\(fail": parentheses \(\) not balanced');
+
+start_errors_like(
+ $node, $ident_file, "include_dir ../ident_inc_fail",
+ generate_log_err_patterns($node, \@ident_raw_errors_step1, 0),
+ 0);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @ident_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map",
+ "missing entry at end of line");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2",
+ "multiple values in ident field");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf",
+ 'map @osnames_fails.conf postgres',
+ "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory");
+
+add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf",
+ 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded');
+
+start_errors_like(
+ $node, $ident_file, "include_if_exists ../ident_if_exists.conf",
+ # There's no guarantee that the generated "line X of file..." will be
+ # emitted for the expected line, but previous tests already ensured that
+ # the correct line number / file name was emitted, so ensuring that there's
+ # an error in all expected lines is enough here.
+ generate_log_err_patterns($node, \@ident_raw_errors_step2, 0),
+ 0);
+
+#####################################################
+# part 3, test reporting of various error scenario
+# NOTE: this will be bypassed -DEXEC_BACKEND or win32
+#####################################################
+reset_auth_files($node);
+
+$node->start;
+$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset');
+
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_hba_file_rules');
+
+add_ident_line($node, $ident_file, '');
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_ident_file_mappings');
+
+# The instance could be restarted and no error is detected. Now check if the
+# build is compatible with the view error reporting (EXEC_BACKEND / win32 will
+# fail when trying to connect as they always rely on the current auth files
+# content)
+my @hba_raw_errors;
+
+push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+my ($stdout, $stderr);
+my $cmdret = $node->psql('postgres', 'SELECT 1',
+ stdout => \$stdout, stderr => \$stderr);
+
+if ($cmdret != 0)
+{
+ # Connection failed. Bail out, but make sure to raise a failure if it
+ # didn't fail for the expected hba file modification.
+ like($stderr,
+ qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/,
+ "Connection failed due to loading an invalid hba file");
+
+ done_testing();
+ diag("Build not compatible with auth file view error reporting, bail out.\n");
+ exit;
+}
+
+# Combine errors generated at step 2, in the same order.
+$node->append_conf($hba_file, "include_dir ../hba_inc_fail");
+push @hba_raw_errors, @hba_raw_errors_step1;
+
+$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf");
+push @hba_raw_errors, @hba_raw_errors_step2;
+
+my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules'
+ . ' WHERE error IS NOT NULL ORDER BY rule_number'),
+ qq($hba_expected),
+ 'Detected all error in hba file');
+
+# and do the same for pg_ident
+my @ident_raw_errors;
+
+push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+$node->append_conf($ident_file, "include_dir ../ident_inc_fail");
+push @ident_raw_errors, @ident_raw_errors_step1;
+
+$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf");
+push @ident_raw_errors, @ident_raw_errors_step2;
+
+my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings'
+ . ' WHERE error IS NOT NULL ORDER BY mapping_number'),
+ qq($ident_expected),
+ 'Detected all error in ident file');
+
+done_testing();
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 79408710e0..5ed2fe3704 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname,
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
a.line_number,
a.type,
a.database,
@@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
a.line_number,
a.map_name,
a.sys_name,
a.pg_username,
a.error
- FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.37.0
--u2elhxoozhwco244
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0003-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v10 2/3] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-05-30 03:15 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-05-30 03:15 UTC (permalink / raw)
pg_hba.conf file now has support for "include", "include_dir" and
"include_if_exists" directives, which work similarly to the same directives in
the postgresql.conf file.
This fixes a possible crash if a secondary file tries to include itself as
there's now a nesting depth check in the inclusion code path, same as the
postgresql.conf.
Many regression tests added to cover both the new directives, but also error
detection for the whole pg_hba / pg_ident files.
Catversion is bumped.
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/client-auth.sgml | 86 ++-
doc/src/sgml/system-views.sgml | 22 +-
src/backend/libpq/hba.c | 483 ++++++++++---
src/backend/libpq/pg_hba.conf.sample | 25 +-
src/backend/libpq/pg_ident.conf.sample | 15 +-
src/backend/utils/adt/hbafuncs.c | 43 +-
src/backend/utils/misc/guc-file.l | 229 +++---
src/include/catalog/pg_proc.dat | 12 +-
src/include/libpq/hba.h | 5 +-
src/include/utils/guc.h | 2 +
.../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++
src/test/regress/expected/rules.out | 6 +-
12 files changed, 1312 insertions(+), 273 deletions(-)
create mode 100644 src/test/authentication/t/003_file_inclusion.pl
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index c6f1b70fd3..42ceb6f3e6 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,23 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication
+ record. Inclusion directives specify files that can be included, which
+ contains additional records. The records will be inserted in lieu of the
+ inclusion records. Those records only contains two fields: the
+ <literal>include</literal>, <literal>include_if_exists</literal> or
+ <literal>include_dir</literal> directive and the file or directory to be
+ included. The file or directory can be a relative of absolute path, and can
+ be double quoted if needed. For the <literal>include_dir</literal> form,
+ all files not starting with a <literal>.</literal> and ending with
+ <literal>.conf</literal> will be included. Multiple files within an include
+ directory are processed in file name order (according to C locale rules,
+ i.e., numbers before letters, and uppercase letters before lowercase ones).
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,21 +118,57 @@
<para>
A record can have several formats:
<synopsis>
-local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+include <replaceable>file</replaceable>
+include_if_exists <replaceable>file</replaceable>
+include_dir <replaceable>directory</replaceable>
+local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
</synopsis>
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_if_exists</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file if the
+ file exists and can be read. Otherwise, a message will be logged to
+ indicate that the file is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_dir</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of all the files found in
+ the directory, if they don't start with a <literal>.</literal> and end
+ with <literal>.conf</literal>, processed in file name order (according
+ to C locale rules, i.e., numbers before letters, and uppercase letters
+ before lowercase ones).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -835,8 +886,10 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
+<replaceable>include_dir</replaceable> <replaceable>directory</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -847,6 +900,11 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ As for <filename>pg_hba.conf</filename>, the lines in this file can either
+ be inclusion directives or user name map records, and follow the same
+ rules.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 1d619427c1..4e63dec74c 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -1003,12 +1003,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule the given <literal>file_name</literal>
</para></entry>
</row>
@@ -1153,12 +1162,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this mapping
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_ident.conf</filename>
+ Line number of this mapping in the given <literal>file_name</literal>
</para></entry>
</row>
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index 4637426d62..071bf1ff95 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -22,6 +22,7 @@
#include <sys/param.h>
#include <sys/socket.h>
#include <netdb.h>
+#include <sys/stat.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
@@ -69,6 +70,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -113,10 +120,22 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int depth,
+ int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
- const char *inc_filename, int elevel, char **err_msg);
+ const char *inc_filename, int depth, int elevel,
+ char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ bool strict, const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
+static char *process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth,
+ int elevel, MemoryContext linecxt,
+ List **tok_lines);
/*
@@ -303,7 +322,7 @@ copy_auth_token(AuthToken *in)
*/
static List *
next_field_expand(const char *filename, char **lineptr,
- int elevel, char **err_msg)
+ int depth, int elevel, char **err_msg)
{
char buf[MAX_TOKEN];
bool trailing_comma;
@@ -319,7 +338,7 @@ next_field_expand(const char *filename, char **lineptr,
/* Is this referencing a file? */
if (!initial_quote && buf[0] == '@' && buf[1] != '\0')
- tokens = tokenize_inc_file(tokens, filename, buf + 1,
+ tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1,
elevel, err_msg);
else
tokens = lappend(tokens, make_auth_token(buf, initial_quote));
@@ -347,6 +366,7 @@ static List *
tokenize_inc_file(List *tokens,
const char *outer_filename,
const char *inc_filename,
+ int depth,
int elevel,
char **err_msg)
{
@@ -356,39 +376,30 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
{
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
+ *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", *err_msg)));
+ return tokens;
}
- inc_file = AllocateFile(inc_fullname, "r");
- if (inc_file == NULL)
- {
- int save_errno = errno;
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true,
+ outer_filename, elevel, err_msg, &inc_fullname);
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
+ if (inc_file == NULL)
return tokens;
- }
/* There is possible recursion here if the file contains @ */
- linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
+ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1,
+ elevel);
FreeFile(inc_file);
pfree(inc_fullname);
@@ -426,11 +437,38 @@ tokenize_inc_file(List *tokens,
/*
* tokenize_auth_file
- * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a dedicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
+ int depth, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, depth,
+ elevel);
+
+ return linecxt;
+}
+
+/*
+ * Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see the struct definition
* in libpq/hba.h.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -439,30 +477,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int depth, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -515,7 +545,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
{
List *current_field;
- current_field = next_field_expand(filename, &lineptr,
+ current_field = next_field_expand(filename, &lineptr, depth,
elevel, &err_msg);
/* add field to line, unless we are at EOL or comment start */
if (current_field != NIL)
@@ -523,29 +553,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ AuthToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
+
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
+
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, true,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
+ else if (strcmp(first->string, "include_dir") == 0)
+ {
+ char **filenames;
+ char *dir_name = second->string;
+ int num_filenames;
+ StringInfoData err_buf;
+
+ filenames = GetDirConfFiles(dir_name, filename, elevel,
+ &num_filenames, &err_msg);
+
+ if (!filenames)
+ {
+ /* We have the error in err_msg, simply process it */
+ goto process_line;
+ }
+
+ initStringInfo(&err_buf);
+ for (int i = 0; i < num_filenames; i++)
+ {
+ /*
+ * err_msg is used here as a temp buffer, it will be
+ * overwritten at the end of the loop with the
+ * cumulated errors, if any.
+ */
+ err_msg = process_included_authfile(filenames[i], true,
+ filename, depth + 1, elevel,
+ linecxt, tok_lines);
+
+ /* Cumulate errors if any. */
+ if (err_msg)
+ {
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfoString(&err_buf, err_msg);
+ }
+ }
+
+ /*
+ * If there were no errors, the line is fully processed, bypass
+ * the general TokenizedAuthLine processing.
+ */
+ if (err_buf.len == 0)
+ goto next_line;
+
+ /* Otherwise, process the cumulated errors, if any. */
+ err_msg = err_buf.data;
+ }
+ else if (strcmp(first->string, "include_if_exists") == 0)
+ {
+ char *inc_filename;
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, false,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
}
+process_line:
+ /*
+ * General processing: report the error if any and emit line to the
+ * TokenizedAuthLine
+ */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -855,7 +983,7 @@ do { \
errmsg("authentication option \"%s\" is only valid for authentication methods %s", \
optname, _(validmethods)), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \
optname, validmethods); \
return false; \
@@ -875,7 +1003,7 @@ do { \
errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname); \
return NULL; \
@@ -898,7 +1026,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("missing entry at end of line"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, file_name))); \
*err_msg = pstrdup("missing entry at end of line"); \
return NULL; \
} \
@@ -911,7 +1039,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("multiple values in ident field"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, file_name))); \
*err_msg = pstrdup("multiple values in ident field"); \
return NULL; \
} \
@@ -934,6 +1062,7 @@ HbaLine *
parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
{
int line_num = tok_line->line_num;
+ char *file_name = tok_line->file_name;
char **err_msg = &tok_line->err_msg;
char *str;
struct addrinfo *gai_result;
@@ -948,6 +1077,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -962,7 +1092,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for connection type"),
errhint("Specify exactly one connection type per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for connection type";
return NULL;
}
@@ -990,7 +1120,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("hostssl record cannot match because SSL is disabled"),
errhint("Set ssl = on in postgresql.conf."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostssl record cannot match because SSL is disabled";
}
#else
@@ -998,7 +1128,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("hostssl record cannot match because SSL is not supported by this build"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostssl record cannot match because SSL is not supported by this build";
#endif
}
@@ -1010,7 +1140,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("hostgssenc record cannot match because GSSAPI is not supported by this build"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostgssenc record cannot match because GSSAPI is not supported by this build";
#endif
}
@@ -1031,7 +1161,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid connection type \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid connection type \"%s\"", token->string);
return NULL;
}
@@ -1044,7 +1174,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before database specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before database specification";
return NULL;
}
@@ -1064,7 +1194,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before role specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before role specification";
return NULL;
}
@@ -1086,7 +1216,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before IP address specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before IP address specification";
return NULL;
}
@@ -1098,7 +1228,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for host address"),
errhint("Specify one address range per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for host address";
return NULL;
}
@@ -1157,7 +1287,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid IP address \"%s\": %s",
str, gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid IP address \"%s\": %s",
str, gai_strerror(ret));
if (gai_result)
@@ -1177,7 +1307,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("specifying both host name and CIDR mask is invalid: \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("specifying both host name and CIDR mask is invalid: \"%s\"",
token->string);
return NULL;
@@ -1191,7 +1321,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid CIDR mask in address \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid CIDR mask in address \"%s\"",
token->string);
return NULL;
@@ -1211,7 +1341,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("end-of-line before netmask specification"),
errhint("Specify an address range in CIDR notation, or provide a separate netmask."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before netmask specification";
return NULL;
}
@@ -1222,7 +1352,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("multiple values specified for netmask"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for netmask";
return NULL;
}
@@ -1237,7 +1367,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid IP mask \"%s\": %s",
token->string, gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid IP mask \"%s\": %s",
token->string, gai_strerror(ret));
if (gai_result)
@@ -1256,7 +1386,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("IP address and mask do not match"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "IP address and mask do not match";
return NULL;
}
@@ -1272,7 +1402,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before authentication method"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before authentication method";
return NULL;
}
@@ -1284,7 +1414,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for authentication type"),
errhint("Specify exactly one authentication type per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for authentication type";
return NULL;
}
@@ -1321,7 +1451,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("MD5 authentication is not supported when \"db_user_namespace\" is enabled"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "MD5 authentication is not supported when \"db_user_namespace\" is enabled";
return NULL;
}
@@ -1362,7 +1492,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid authentication method \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid authentication method \"%s\"",
token->string);
return NULL;
@@ -1375,7 +1505,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid authentication method \"%s\": not supported by this build",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid authentication method \"%s\": not supported by this build",
token->string);
return NULL;
@@ -1397,7 +1527,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("gssapi authentication is not supported on local sockets"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "gssapi authentication is not supported on local sockets";
return NULL;
}
@@ -1409,7 +1539,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("peer authentication is only supported on local sockets"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "peer authentication is only supported on local sockets";
return NULL;
}
@@ -1427,7 +1557,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cert authentication is only supported on hostssl connections"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cert authentication is only supported on hostssl connections";
return NULL;
}
@@ -1477,7 +1607,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("authentication option not in name=value format: %s", token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("authentication option not in name=value format: %s",
token->string);
return NULL;
@@ -1521,7 +1651,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix";
return NULL;
}
@@ -1532,7 +1662,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set";
return NULL;
}
@@ -1548,7 +1678,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cannot use ldapsearchattribute together with ldapsearchfilter"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cannot use ldapsearchattribute together with ldapsearchfilter";
return NULL;
}
@@ -1565,7 +1695,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("list of RADIUS servers cannot be empty"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "list of RADIUS servers cannot be empty";
return NULL;
}
@@ -1576,7 +1706,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("list of RADIUS secrets cannot be empty"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "list of RADIUS secrets cannot be empty";
return NULL;
}
@@ -1595,7 +1725,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiussecrets),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS secrets (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiussecrets),
list_length(parsedline->radiusservers));
@@ -1611,7 +1741,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiusports),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS ports (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiusports),
list_length(parsedline->radiusservers));
@@ -1627,7 +1757,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiusidentifiers),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS identifiers (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiusidentifiers),
list_length(parsedline->radiusservers));
@@ -1662,6 +1792,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg)
{
int line_num = hbaline->linenumber;
+ char *file_name = hbaline->sourcefile;
#ifdef USE_LDAP
hbaline->ldapscope = LDAP_SCOPE_SUBTREE;
@@ -1685,7 +1816,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientcert can only be configured for \"hostssl\" rows"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientcert can only be configured for \"hostssl\" rows";
return false;
}
@@ -1702,7 +1833,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientcert only accepts \"verify-full\" when using \"cert\" authentication"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientcert can only be set to \"verify-full\" when using \"cert\" authentication";
return false;
}
@@ -1715,7 +1846,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid value for clientcert: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
}
@@ -1727,7 +1858,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientname can only be configured for \"hostssl\" rows"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientname can only be configured for \"hostssl\" rows";
return false;
}
@@ -1746,7 +1877,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid value for clientname: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
}
@@ -1832,7 +1963,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid ldapscheme value: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
hbaline->ldapscheme = pstrdup(val);
}
else if (strcmp(name, "ldapserver") == 0)
@@ -1850,7 +1981,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid LDAP port number: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid LDAP port number: \"%s\"", val);
return false;
}
@@ -1944,7 +2075,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS server list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -1963,7 +2094,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not translate RADIUS server name \"%s\" to address: %s",
(char *) lfirst(l), gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
if (gai_result)
pg_freeaddrinfo_all(hints.ai_family, gai_result);
@@ -1992,7 +2123,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS port list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid RADIUS port number: \"%s\"", val);
return false;
}
@@ -2005,7 +2136,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid RADIUS port number: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2028,7 +2159,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS secret list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2050,7 +2181,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS identifiers list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2064,7 +2195,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("unrecognized authentication option name: \"%s\"",
name),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("unrecognized authentication option name: \"%s\"",
name);
return false;
@@ -2212,7 +2343,7 @@ load_hba(void)
return false;
}
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
@@ -2283,6 +2414,137 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ if (strict)
+ {
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ }
+ else
+ {
+ Assert(kind == IncludedAuthFile);
+ ereport(LOG,
+ (errmsg("skipping missing authentication file \"%s\"",
+ *inc_fullname)));
+ }
+
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
+
+/*
+ * Try to open an included file, and tokenize it using the given context.
+ * Returns NULL if no error happens during tokenization, otherwise the error.
+ */
+static char *
+process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth, int elevel,
+ MemoryContext linecxt, List **tok_lines)
+{
+ char *inc_fullname;
+ FILE *inc_file;
+ char *err_msg = NULL;
+
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
+ {
+ char *err_msg;
+
+ err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", err_msg)));
+ return err_msg;
+ }
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict,
+ outer_filename, elevel, &err_msg, &inc_fullname);
+
+ if (inc_file == NULL)
+ {
+ if (strict)
+ {
+ /* open_inc_file should have reported an error. */
+ Assert(err_msg != NULL);
+ return err_msg;
+ }
+ else
+ return NULL;
+ }
+ else
+ {
+ /* No error message should have been reported. */
+ Assert(err_msg == NULL);
+ }
+
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, depth, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ return NULL;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
@@ -2301,6 +2563,7 @@ load_hba(void)
IdentLine *
parse_ident_line(TokenizedAuthLine *tok_line, int elevel)
{
+ char *file_name = tok_line->file_name;
int line_num = tok_line->line_num;
char **err_msg = &tok_line->err_msg;
ListCell *field;
@@ -2361,7 +2624,7 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid regular expression \"%s\": %s",
parsedline->ident_user + 1, errstr),
errcontext("line %d of configuration file \"%s\"",
- line_num, IdentFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid regular expression \"%s\": %s",
parsedline->ident_user + 1, errstr);
@@ -2596,7 +2859,7 @@ load_ident(void)
return false;
}
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..7433050112 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,16 +9,27 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
-# local DATABASE USER METHOD [OPTIONS]
-# host DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# local DATABASE USER METHOD [OPTIONS]
+# host DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..8e3fa29135 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,23 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
-# MAPNAME SYSTEM-USERNAME PG-USERNAME
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index c9be4bff1f..15326a01e2 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,12 +26,12 @@
static ArrayType *get_hba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int rule_number, int lineno, HbaLine *hba,
- const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int mapping_number, int lineno, IdentLine *ident,
- const char *err_msg);
+ int mapping_number, const char *filename,
+ int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
@@ -159,7 +159,7 @@ get_hba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 10
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line
@@ -168,7 +168,8 @@ get_hba_options(HbaLine *hba)
* tuple_store: where to store data
* tupdesc: tuple descriptor for the view
* rule_number: unique rule identifier among all valid rules
- * lineno: pg_hba.conf line number (must always be valid)
+ * filename: name of the file containing that line
+ * lineno: line number in that file (must always be valid)
* hba: parsed line data (can be NULL, in which case err_msg should be set)
* err_msg: error message (NULL if none)
*
@@ -177,7 +178,7 @@ get_hba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int rule_number, int lineno, HbaLine *hba,
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -202,6 +203,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -345,7 +348,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -386,7 +389,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open configuration file \"%s\": %m",
HbaFileName)));
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -407,8 +410,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
rule_number++;
- fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -445,7 +448,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
}
/* Number of columns in pg_hba_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -454,7 +457,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
* tuple_store: where to store data
* tupdesc: tuple descriptor for the view
* mapping_number: unique rule identifier among all valid rules
- * lineno: pg_ident.conf line number (must always be valid)
+ * filename: name of the file containing that line
+ * lineno: line number in that file (must always be valid)
* ident: parsed line data (can be NULL, in which case err_msg should be set)
* err_msg: error message (NULL if none)
*
@@ -463,8 +467,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int mapping_number, int lineno, IdentLine *ident,
- const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -482,6 +486,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -494,7 +500,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -534,7 +540,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open usermap file \"%s\": %m",
IdentFileName)));
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -556,7 +562,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
mapping_number++;
fill_ident_line(tuple_store, tupdesc, mapping_number,
- tok_line->line_num, identline, tok_line->err_msg);
+ tok_line->file_name, tok_line->line_num, identline,
+ tok_line->err_msg);
}
/* Free tokenizer memory */
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index 4f43430db4..ecfb4a2a28 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -345,6 +345,110 @@ GUC_flex_fatal(const char *msg)
return 0; /* keep compiler quiet */
}
+/*
+ * Returns the list of config files located in a directory, in alphabetical
+ * order.
+ *
+ * We don't check for recursion or too-deep nesting depth here, its up to the
+ * caller to take care of that.
+ */
+char **
+GetDirConfFiles(const char *includedir, const char *calling_file, int elevel,
+ int *num_filenames, char **err_msg)
+{
+ char *directory;
+ DIR *d;
+ struct dirent *de;
+ char **filenames;
+ int size_filenames;
+
+ /*
+ * Reject directory name that is all-blank (including empty), as that
+ * leads to confusion --- we'd read the containing directory, typically
+ * resulting in recursive inclusion of the same file(s).
+ */
+ if (strspn(includedir, " \t\r\n") == strlen(includedir))
+ {
+ ereport(elevel,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("empty configuration directory name: \"%s\"",
+ includedir)));
+ *err_msg = "empty configuration directory name";
+ return NULL;
+ }
+
+ directory = AbsoluteConfigLocation(includedir, calling_file);
+ d = AllocateDir(directory);
+ if (d == NULL)
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not open configuration directory \"%s\": %m",
+ directory)));
+ *err_msg = psprintf("could not open directory \"%s\"", directory);
+ filenames = NULL;
+ goto cleanup;
+ }
+
+ /*
+ * Read the directory and put the filenames in an array, so we can sort
+ * them prior to caller processing the contents.
+ */
+ size_filenames = 32;
+ filenames = (char **) palloc(size_filenames * sizeof(char *));
+ *num_filenames = 0;
+
+ while ((de = ReadDir(d, directory)) != NULL)
+ {
+ PGFileType de_type;
+ char filename[MAXPGPATH];
+
+ /*
+ * Only parse files with names ending in ".conf". Explicitly reject
+ * files starting with ".". This excludes things like "." and "..",
+ * as well as typical hidden files, backup files, and editor debris.
+ */
+ if (strlen(de->d_name) < 6)
+ continue;
+ if (de->d_name[0] == '.')
+ continue;
+ if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
+ continue;
+
+ join_path_components(filename, directory, de->d_name);
+ canonicalize_path(filename);
+ de_type = get_dirent_type(filename, de, true, elevel);
+ if (de_type == PGFILETYPE_ERROR)
+ {
+ *err_msg = psprintf("could not stat file \"%s\"", filename);
+ pfree(filenames);
+ filenames = NULL;
+ goto cleanup;
+ }
+ else if (de_type != PGFILETYPE_DIR)
+ {
+ /* Add file to array, increasing its size in blocks of 32 */
+ if (*num_filenames >= size_filenames)
+ {
+ size_filenames += 32;
+ filenames = (char **) repalloc(filenames,
+ size_filenames * sizeof(char *));
+ }
+ filenames[*num_filenames] = pstrdup(filename);
+ (*num_filenames)++;
+ }
+ }
+
+ if (*num_filenames > 0)
+ qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp);
+
+cleanup:
+ if (d)
+ FreeDir(d);
+ pfree(directory);
+ return filenames;
+}
+
/*
* Read and parse a single configuration file. This function recurses
* to handle "include" directives.
@@ -606,127 +710,30 @@ ParseConfigDirectory(const char *includedir,
ConfigVariable **head_p,
ConfigVariable **tail_p)
{
- char *directory;
- DIR *d;
- struct dirent *de;
+ char *err_msg;
char **filenames;
int num_filenames;
- int size_filenames;
- bool status;
-
- /*
- * Reject directory name that is all-blank (including empty), as that
- * leads to confusion --- we'd read the containing directory, typically
- * resulting in recursive inclusion of the same file(s).
- */
- if (strspn(includedir, " \t\r\n") == strlen(includedir))
- {
- ereport(elevel,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("empty configuration directory name: \"%s\"",
- includedir)));
- record_config_file_error("empty configuration directory name",
- calling_file, calling_lineno,
- head_p, tail_p);
- return false;
- }
-
- /*
- * We don't check for recursion or too-deep nesting depth here; the
- * subsequent calls to ParseConfigFile will take care of that.
- */
-
- directory = AbsoluteConfigLocation(includedir, calling_file);
- d = AllocateDir(directory);
- if (d == NULL)
- {
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open configuration directory \"%s\": %m",
- directory)));
- record_config_file_error(psprintf("could not open directory \"%s\"",
- directory),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
- /*
- * Read the directory and put the filenames in an array, so we can sort
- * them prior to processing the contents.
- */
- size_filenames = 32;
- filenames = (char **) palloc(size_filenames * sizeof(char *));
- num_filenames = 0;
+ filenames = GetDirConfFiles(includedir, calling_file, elevel,
+ &num_filenames, &err_msg);
- while ((de = ReadDir(d, directory)) != NULL)
+ if (!filenames)
{
- PGFileType de_type;
- char filename[MAXPGPATH];
-
- /*
- * Only parse files with names ending in ".conf". Explicitly reject
- * files starting with ".". This excludes things like "." and "..",
- * as well as typical hidden files, backup files, and editor debris.
- */
- if (strlen(de->d_name) < 6)
- continue;
- if (de->d_name[0] == '.')
- continue;
- if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
- continue;
-
- join_path_components(filename, directory, de->d_name);
- canonicalize_path(filename);
- de_type = get_dirent_type(filename, de, true, elevel);
- if (de_type == PGFILETYPE_ERROR)
- {
- record_config_file_error(psprintf("could not stat file \"%s\"",
- filename),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
- else if (de_type != PGFILETYPE_DIR)
- {
- /* Add file to array, increasing its size in blocks of 32 */
- if (num_filenames >= size_filenames)
- {
- size_filenames += 32;
- filenames = (char **) repalloc(filenames,
- size_filenames * sizeof(char *));
- }
- filenames[num_filenames] = pstrdup(filename);
- num_filenames++;
- }
+ record_config_file_error(err_msg, calling_file, calling_lineno, head_p,
+ tail_p);
+ return false;
}
- if (num_filenames > 0)
+ for (int i = 0; i < num_filenames; i++)
{
- int i;
-
- qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp);
- for (i = 0; i < num_filenames; i++)
- {
- if (!ParseConfigFile(filenames[i], true,
- calling_file, calling_lineno,
- depth, elevel,
- head_p, tail_p))
- {
- status = false;
- goto cleanup;
- }
- }
+ if (!ParseConfigFile(filenames[i], true,
+ calling_file, calling_lineno,
+ depth, elevel,
+ head_p, tail_p))
+ return false;
}
- status = true;
-cleanup:
- if (d)
- FreeDir(d);
- pfree(directory);
- return status;
+ return true;
}
/*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 060bafaced..e6a54cc3d6 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6128,16 +6128,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '6250', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,text,text,text}',
- proargmodes => '{o,o,o,o,o,o}',
- proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index 90036f7bcd..0ea100d1b8 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -155,6 +156,7 @@ typedef struct AuthToken
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of AuthTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
@@ -174,6 +176,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel);
extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel);
extern bool pg_isblank(const char c);
extern MemoryContext tokenize_auth_file(const char *filename, FILE *file,
- List **tok_lines, int elevel);
+ List **tok_lines, int depth,
+ int elevel);
#endif /* HBA_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 45ae1b537f..cb3bbcf949 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -145,6 +145,8 @@ typedef struct ConfigVariable
struct ConfigVariable *next;
} ConfigVariable;
+extern char **GetDirConfFiles(const char *includedir, const char *calling_file,
+ int elevel, int *num_filenames, char **err_msg);
extern bool ParseConfigFile(const char *config_file, bool strict,
const char *calling_file, int calling_lineno,
int depth, int elevel,
diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl
new file mode 100644
index 0000000000..8eae72b8d4
--- /dev/null
+++ b/src/test/authentication/t/003_file_inclusion.pl
@@ -0,0 +1,657 @@
+
+# Copyright (c) 2021-2022, PostgreSQL Global Development Group
+
+# Set of tests for authentication and pg_hba.conf inclusion.
+# This test can only run with Unix-domain sockets.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Time::HiRes qw(usleep);
+use IPC::Run qw(pump finish timer);
+use Data::Dumper;
+
+if (!$use_unix_sockets)
+{
+ plan skip_all =>
+ "authentication tests cannot run without Unix-domain sockets";
+}
+
+# stores the current line counter for each file. hba_rule and ident_rule are
+# fake file names used for the global rule number for each auth view.
+my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+my $hba_file = 'subdir1/pg_hba_custom.conf';
+my $ident_file = 'subdir2/pg_ident_custom.conf';
+
+# Initialize primary node
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+$node->start;
+
+my $data_dir = $node->data_dir;
+
+# Normalize the data directory for Windows
+$data_dir =~ s/\/\.\//\//g; # reduce /./ to /
+$data_dir =~ s/\/\//\//g; # reduce // to /
+$data_dir =~ s/\/$//; # remove trailing /
+
+
+# Add the given payload to the given relative HBA file of the given node.
+# This function maintains the %cur_line metadata, so it has to be called in the
+# expected inclusion evaluation order in order to keep it in sync.
+#
+# If the payload starts with "include" or "ignore", the function doesn't
+# increase the general hba rule number.
+#
+# If an err_str is provided, it returns an arrayref containing the provided
+# filename, the current line number in that file and the provided err_str. The
+# err_str has to be a valid regex string.
+# Otherwise it only returns the line number of the payload in the wanted file.
+# This function has to be called in the expected inclusion evaluation order to
+# keep the %cur_line information in sync.
+sub add_hba_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'hba_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_hba_file_rules line
+ @tokens = split(/ /, $payload);
+ $tokens[1] = '{' . $tokens[1] . '}'; # database
+ $tokens[2] = '{' . $tokens[2] . '}'; # user_name
+
+ # add empty address and netmask betweed user_name and auth_method
+ splice @tokens, 3, 0, '';
+ splice @tokens, 3, 0, '';
+
+ # append empty options and error
+ push @tokens, '';
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Add the given payload to the given relative ident file of the given node.
+# Same as add_hba_line but for pg_ident files
+sub add_ident_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'ident_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_ident_file_mappings line
+ @tokens = split(/ /, $payload);
+
+ # append empty error
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Delete pg_hba.conf from the given node, add various entries to test the
+# include infrastructure and then execute a reload to refresh it.
+sub generate_valid_auth_files
+{
+ my $node = shift;
+ my $hba_expected = '';
+ my $ident_expected = '';
+
+ # customise main auth file names
+ $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'");
+ $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'");
+
+ # and make original ones invalid to be sure they're not used anywhere
+ $node->append_conf('pg_hba.conf', "some invalid line");
+ $node->append_conf('pg_ident.conf', "some invalid line");
+
+ # pg_hba stuff
+ mkdir("$data_dir/subdir1");
+ mkdir("$data_dir/hba_inc");
+ mkdir("$data_dir/hba_inc_if");
+ mkdir("$data_dir/hba_pos");
+
+ # Make sure we will still be able to connect
+ $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust');
+
+ # Add include data
+ add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf");
+ $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject");
+
+ $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject");
+
+ add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject");
+ # include is relative to current path
+ add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject");
+
+ # include_if_exists data
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none");
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some");
+ $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject");
+
+ # include_dir data
+ add_hba_line($node, "$hba_file", "include_dir ../hba_inc");
+ add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included");
+ $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject");
+ $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject");
+
+ # secondary auth file
+ add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject');
+ $node->append_conf('dbnames.conf', "db1");
+ $node->append_conf('dbnames.conf', "db3");
+ $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1)
+ . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1)
+ . '|local|{db1,db3}|{all}|||reject||';
+
+ # pg_ident stuff
+ mkdir("$data_dir/subdir2");
+ mkdir("$data_dir/ident_inc");
+ mkdir("$data_dir/ident_inc_if");
+ mkdir("$data_dir/ident_pos");
+
+ # Add include data
+ add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf");
+ $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar");
+
+ $ident_expected .= add_ident_line($node, "$ident_file", "test a b");
+
+ add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar");
+ # include is relative to current path
+ add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar");
+
+ # include_if_exists data
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none");
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some");
+ $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar");
+
+ # include_dir data
+ add_ident_line($node, "$ident_file", "include_dir ../ident_inc");
+ add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included");
+ $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar");
+ $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar");
+
+ $node->restart;
+ $node->connect_ok('dbname=postgres',
+ 'Connection ok after generating valid auth files');
+
+ return ($hba_expected, $ident_expected);
+}
+
+# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal
+# entries to allow authentication.
+sub reset_auth_files
+{
+ my $node = shift;
+
+ unlink("$data_dir/$hba_file");
+ unlink("$data_dir/$ident_file");
+
+ %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+ return add_hba_line($node, "$hba_file", 'local all all trust');
+}
+
+# Generate a list of expected error regex for the given array of error
+# conditions, as generated by add_hba_line/add_ident_line with an err_str.
+#
+# 2 regex are generated per array entry: one for the given err_str, and one for
+# the expected line in the specific file. Since all lines are independant,
+# there's no guarantee that a specific failure regex and the per-line regex
+# will match the same error. Calling code should add at least one test with a
+# single error to make sure that the line number / file name is correct.
+#
+# On top of that, an extra line is generated for the general failure to process
+# the main auth file.
+sub generate_log_err_patterns
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $is_hba_err = shift;
+ my @errors;
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ push @errors, qr/$err_str/;
+
+ # Context messages with the file / line location aren't always emitted
+ if ($err_str !~ /maximum nesting depth exceeded/ and
+ $err_str !~ /could not open secondary authentication file/)
+ {
+ push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/
+ }
+ }
+
+ push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err);
+
+ return \@errors;
+}
+
+# Generate the expected output for the auth file view error reporting (file
+# name, file line, error), for the given array of error conditions, as
+# generated generated by add_hba_line/add_ident_line with an err_str.
+sub generate_log_err_rows
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $exp_rows = '';
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ $exp_rows .= "\n" if ($exp_rows ne "");
+
+ # Unescape regex patterns if any
+ $err_str =~ s/\\([\(\)])/$1/g;
+ $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str"
+ }
+
+ return $exp_rows;
+}
+
+# Reset the main auth files, append the given payload to the given config file,
+# and check that the instance cannot start, raising the expected error line(s).
+sub start_errors_like
+{
+ my $node = shift;
+ my $file = shift;
+ my $payload = shift;
+ my $pattern = shift;
+ my $should_fail = shift;
+
+ reset_auth_files($node);
+ $node->append_conf($file, $payload);
+
+ unlink($node->logfile);
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir,
+ '-l', $node->logfile, 'start');
+
+ if ($should_fail)
+ {
+ ok($ret != 0, "Cannot start postgres with faulty $file");
+ }
+ else
+ {
+ ok($ret == 0, "postgres can start with faulty $file");
+ }
+
+ my $log_contents = slurp_file($node->logfile);
+
+ foreach (@{$pattern})
+ {
+ like($log_contents,
+ $_,
+ "Expected failure found in the logs");
+ }
+
+ if (not $should_fail)
+ {
+ # We can't simply call $node->stop here as the call is optimized out
+ # when the server isn't started with $node->start.
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D',
+ $data_dir, 'stop', '-m', 'fast');
+ ok($ret == 0, "Could stop postgres");
+ }
+}
+
+# We should be able to connect, and see an empty pg_ident.conf
+is($node->psql(
+ 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'),
+ qq(0),
+ 'pg_ident.conf is empty');
+
+############################################
+# part 1, test view reporting for valid data
+############################################
+my ($exp_hba, $exp_ident) = generate_valid_auth_files($node);
+
+$node->connect_ok('dbname=postgres', 'Connection still ok');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_hba_file_rules'),
+ qq($exp_hba),
+ 'pg_hba_file_rules content is expected');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_ident_file_mappings'),
+ qq($exp_ident),
+ 'pg_ident_file_mappings content is expected');
+
+#############################################
+# part 2, test log reporting for invalid data
+#############################################
+reset_auth_files($node);
+$node->restart('fast');
+$node->connect_ok('dbname=postgres',
+ 'Connection ok after resetting auth files');
+
+$node->stop('fast');
+
+start_errors_like($node, $hba_file, "include ../not_a_file",
+ [
+ qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file
+mkdir("$data_dir/hba_inc_fail");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token");
+start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail",
+ [
+ qr/invalid connection type "not_a_token"/,
+ qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file with nested inclusion
+unlink("$data_dir/hba_inc_fail/inc_dir.conf");
+my @hba_raw_errors_step1;
+
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1");
+
+add_hba_line($node, "hba_inc_fail/file1", "include file2");
+add_hba_line($node, "hba_inc_fail/file2", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file2", "include file3");
+
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3",
+ "local all all zuul",
+ 'invalid authentication method "zuul"');
+
+start_errors_like(
+ $node, $hba_file, "include_dir ../hba_inc_fail",
+ generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @hba_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local",
+ "end-of-line before database specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local,host",
+ "multiple values specified for connection type");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all",
+ "end-of-line before role specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all all",
+ "end-of-line before authentication method");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "host all all test/42",
+ 'specifying both host name and CIDR mask is invalid: "test/42"');
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ 'local @dbnames_fails.conf all reject',
+ "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory");
+
+add_hba_line($node, "hba_if_exists.conf", "include recurse.conf");
+push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf",
+ "include recurse.conf",
+ 'could not open configuration file "recurse.conf": maximum nesting depth exceeded');
+
+# Generate the regex for the expected errors in the logs. There's no guarantee
+# that the generated "line X of file..." will be emitted for the expected line,
+# but previous tests already ensured that the correct line number / file name
+# was emitted, so ensuring that there's an error in all expected lines is
+# enough here.
+my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2,
+ 1);
+
+# Not an error, but it should raise a message in the logs. Manually add an
+# extra log message to detect
+add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none");
+push @{$expected_errors},
+ qr/skipping missing authentication file "$data_dir\/if_exists_none"/;
+
+start_errors_like(
+ $node, $hba_file, "include_if_exists ../hba_if_exists.conf",
+ $expected_errors, 1);
+
+# Mostly the same, but for ident files
+reset_auth_files($node);
+
+my @ident_raw_errors_step1;
+
+# include_dir, single included file with nested inclusion
+mkdir("$data_dir/ident_inc_fail");
+add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1");
+
+add_ident_line($node, "ident_inc_fail/file1", "include file2");
+add_ident_line($node, "ident_inc_fail/file2", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file2", "include file3");
+
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3",
+ "failmap /(fail postgres",
+ 'invalid regular expression "\(fail": parentheses \(\) not balanced');
+
+start_errors_like(
+ $node, $ident_file, "include_dir ../ident_inc_fail",
+ generate_log_err_patterns($node, \@ident_raw_errors_step1, 0),
+ 0);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @ident_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map",
+ "missing entry at end of line");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2",
+ "multiple values in ident field");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf",
+ 'map @osnames_fails.conf postgres',
+ "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory");
+
+add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf",
+ 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded');
+
+start_errors_like(
+ $node, $ident_file, "include_if_exists ../ident_if_exists.conf",
+ # There's no guarantee that the generated "line X of file..." will be
+ # emitted for the expected line, but previous tests already ensured that
+ # the correct line number / file name was emitted, so ensuring that there's
+ # an error in all expected lines is enough here.
+ generate_log_err_patterns($node, \@ident_raw_errors_step2, 0),
+ 0);
+
+#####################################################
+# part 3, test reporting of various error scenario
+# NOTE: this will be bypassed -DEXEC_BACKEND or win32
+#####################################################
+reset_auth_files($node);
+
+$node->start;
+$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset');
+
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_hba_file_rules');
+
+add_ident_line($node, $ident_file, '');
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_ident_file_mappings');
+
+# The instance could be restarted and no error is detected. Now check if the
+# build is compatible with the view error reporting (EXEC_BACKEND / win32 will
+# fail when trying to connect as they always rely on the current auth files
+# content)
+my @hba_raw_errors;
+
+push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+my ($stdout, $stderr);
+my $cmdret = $node->psql('postgres', 'SELECT 1',
+ stdout => \$stdout, stderr => \$stderr);
+
+if ($cmdret != 0)
+{
+ # Connection failed. Bail out, but make sure to raise a failure if it
+ # didn't fail for the expected hba file modification.
+ like($stderr,
+ qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/,
+ "Connection failed due to loading an invalid hba file");
+
+ done_testing();
+ diag("Build not compatible with auth file view error reporting, bail out.\n");
+ exit;
+}
+
+# Combine errors generated at step 2, in the same order.
+$node->append_conf($hba_file, "include_dir ../hba_inc_fail");
+push @hba_raw_errors, @hba_raw_errors_step1;
+
+$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf");
+push @hba_raw_errors, @hba_raw_errors_step2;
+
+my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules'
+ . ' WHERE error IS NOT NULL ORDER BY rule_number'),
+ qq($hba_expected),
+ 'Detected all error in hba file');
+
+# and do the same for pg_ident
+my @ident_raw_errors;
+
+push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+$node->append_conf($ident_file, "include_dir ../ident_inc_fail");
+push @ident_raw_errors, @ident_raw_errors_step1;
+
+$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf");
+push @ident_raw_errors, @ident_raw_errors_step2;
+
+my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings'
+ . ' WHERE error IS NOT NULL ORDER BY mapping_number'),
+ qq($ident_expected),
+ 'Detected all error in ident file');
+
+done_testing();
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 79408710e0..5ed2fe3704 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname,
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
a.line_number,
a.type,
a.database,
@@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
a.line_number,
a.map_name,
a.sys_name,
a.pg_username,
a.error
- FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.37.0
--fy3hxyvyv2spu3wq
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v10-0003-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v10 2/3] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-05-30 03:15 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-05-30 03:15 UTC (permalink / raw)
pg_hba.conf file now has support for "include", "include_dir" and
"include_if_exists" directives, which work similarly to the same directives in
the postgresql.conf file.
This fixes a possible crash if a secondary file tries to include itself as
there's now a nesting depth check in the inclusion code path, same as the
postgresql.conf.
Many regression tests added to cover both the new directives, but also error
detection for the whole pg_hba / pg_ident files.
Catversion is bumped.
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/client-auth.sgml | 86 ++-
doc/src/sgml/system-views.sgml | 22 +-
src/backend/libpq/hba.c | 483 ++++++++++---
src/backend/libpq/pg_hba.conf.sample | 25 +-
src/backend/libpq/pg_ident.conf.sample | 15 +-
src/backend/utils/adt/hbafuncs.c | 43 +-
src/backend/utils/misc/guc-file.l | 229 +++---
src/include/catalog/pg_proc.dat | 12 +-
src/include/libpq/hba.h | 5 +-
src/include/utils/guc.h | 2 +
.../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++
src/test/regress/expected/rules.out | 6 +-
12 files changed, 1312 insertions(+), 273 deletions(-)
create mode 100644 src/test/authentication/t/003_file_inclusion.pl
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index c6f1b70fd3..42ceb6f3e6 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,23 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication
+ record. Inclusion directives specify files that can be included, which
+ contains additional records. The records will be inserted in lieu of the
+ inclusion records. Those records only contains two fields: the
+ <literal>include</literal>, <literal>include_if_exists</literal> or
+ <literal>include_dir</literal> directive and the file or directory to be
+ included. The file or directory can be a relative of absolute path, and can
+ be double quoted if needed. For the <literal>include_dir</literal> form,
+ all files not starting with a <literal>.</literal> and ending with
+ <literal>.conf</literal> will be included. Multiple files within an include
+ directory are processed in file name order (according to C locale rules,
+ i.e., numbers before letters, and uppercase letters before lowercase ones).
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,21 +118,57 @@
<para>
A record can have several formats:
<synopsis>
-local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+include <replaceable>file</replaceable>
+include_if_exists <replaceable>file</replaceable>
+include_dir <replaceable>directory</replaceable>
+local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
</synopsis>
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_if_exists</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file if the
+ file exists and can be read. Otherwise, a message will be logged to
+ indicate that the file is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_dir</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of all the files found in
+ the directory, if they don't start with a <literal>.</literal> and end
+ with <literal>.conf</literal>, processed in file name order (according
+ to C locale rules, i.e., numbers before letters, and uppercase letters
+ before lowercase ones).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -835,8 +886,10 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
+<replaceable>include_dir</replaceable> <replaceable>directory</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -847,6 +900,11 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ As for <filename>pg_hba.conf</filename>, the lines in this file can either
+ be inclusion directives or user name map records, and follow the same
+ rules.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 1d619427c1..4e63dec74c 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -1003,12 +1003,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule the given <literal>file_name</literal>
</para></entry>
</row>
@@ -1153,12 +1162,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this mapping
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_ident.conf</filename>
+ Line number of this mapping in the given <literal>file_name</literal>
</para></entry>
</row>
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index 4637426d62..071bf1ff95 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -22,6 +22,7 @@
#include <sys/param.h>
#include <sys/socket.h>
#include <netdb.h>
+#include <sys/stat.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
@@ -69,6 +70,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -113,10 +120,22 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int depth,
+ int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
- const char *inc_filename, int elevel, char **err_msg);
+ const char *inc_filename, int depth, int elevel,
+ char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ bool strict, const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
+static char *process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth,
+ int elevel, MemoryContext linecxt,
+ List **tok_lines);
/*
@@ -303,7 +322,7 @@ copy_auth_token(AuthToken *in)
*/
static List *
next_field_expand(const char *filename, char **lineptr,
- int elevel, char **err_msg)
+ int depth, int elevel, char **err_msg)
{
char buf[MAX_TOKEN];
bool trailing_comma;
@@ -319,7 +338,7 @@ next_field_expand(const char *filename, char **lineptr,
/* Is this referencing a file? */
if (!initial_quote && buf[0] == '@' && buf[1] != '\0')
- tokens = tokenize_inc_file(tokens, filename, buf + 1,
+ tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1,
elevel, err_msg);
else
tokens = lappend(tokens, make_auth_token(buf, initial_quote));
@@ -347,6 +366,7 @@ static List *
tokenize_inc_file(List *tokens,
const char *outer_filename,
const char *inc_filename,
+ int depth,
int elevel,
char **err_msg)
{
@@ -356,39 +376,30 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
{
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
+ *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", *err_msg)));
+ return tokens;
}
- inc_file = AllocateFile(inc_fullname, "r");
- if (inc_file == NULL)
- {
- int save_errno = errno;
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true,
+ outer_filename, elevel, err_msg, &inc_fullname);
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
+ if (inc_file == NULL)
return tokens;
- }
/* There is possible recursion here if the file contains @ */
- linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
+ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1,
+ elevel);
FreeFile(inc_file);
pfree(inc_fullname);
@@ -426,11 +437,38 @@ tokenize_inc_file(List *tokens,
/*
* tokenize_auth_file
- * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a dedicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
+ int depth, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, depth,
+ elevel);
+
+ return linecxt;
+}
+
+/*
+ * Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see the struct definition
* in libpq/hba.h.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -439,30 +477,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int depth, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -515,7 +545,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
{
List *current_field;
- current_field = next_field_expand(filename, &lineptr,
+ current_field = next_field_expand(filename, &lineptr, depth,
elevel, &err_msg);
/* add field to line, unless we are at EOL or comment start */
if (current_field != NIL)
@@ -523,29 +553,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ AuthToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
+
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
+
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, true,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
+ else if (strcmp(first->string, "include_dir") == 0)
+ {
+ char **filenames;
+ char *dir_name = second->string;
+ int num_filenames;
+ StringInfoData err_buf;
+
+ filenames = GetDirConfFiles(dir_name, filename, elevel,
+ &num_filenames, &err_msg);
+
+ if (!filenames)
+ {
+ /* We have the error in err_msg, simply process it */
+ goto process_line;
+ }
+
+ initStringInfo(&err_buf);
+ for (int i = 0; i < num_filenames; i++)
+ {
+ /*
+ * err_msg is used here as a temp buffer, it will be
+ * overwritten at the end of the loop with the
+ * cumulated errors, if any.
+ */
+ err_msg = process_included_authfile(filenames[i], true,
+ filename, depth + 1, elevel,
+ linecxt, tok_lines);
+
+ /* Cumulate errors if any. */
+ if (err_msg)
+ {
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfoString(&err_buf, err_msg);
+ }
+ }
+
+ /*
+ * If there were no errors, the line is fully processed, bypass
+ * the general TokenizedAuthLine processing.
+ */
+ if (err_buf.len == 0)
+ goto next_line;
+
+ /* Otherwise, process the cumulated errors, if any. */
+ err_msg = err_buf.data;
+ }
+ else if (strcmp(first->string, "include_if_exists") == 0)
+ {
+ char *inc_filename;
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, false,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
}
+process_line:
+ /*
+ * General processing: report the error if any and emit line to the
+ * TokenizedAuthLine
+ */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -855,7 +983,7 @@ do { \
errmsg("authentication option \"%s\" is only valid for authentication methods %s", \
optname, _(validmethods)), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \
optname, validmethods); \
return false; \
@@ -875,7 +1003,7 @@ do { \
errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname); \
return NULL; \
@@ -898,7 +1026,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("missing entry at end of line"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, file_name))); \
*err_msg = pstrdup("missing entry at end of line"); \
return NULL; \
} \
@@ -911,7 +1039,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("multiple values in ident field"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, file_name))); \
*err_msg = pstrdup("multiple values in ident field"); \
return NULL; \
} \
@@ -934,6 +1062,7 @@ HbaLine *
parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
{
int line_num = tok_line->line_num;
+ char *file_name = tok_line->file_name;
char **err_msg = &tok_line->err_msg;
char *str;
struct addrinfo *gai_result;
@@ -948,6 +1077,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -962,7 +1092,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for connection type"),
errhint("Specify exactly one connection type per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for connection type";
return NULL;
}
@@ -990,7 +1120,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("hostssl record cannot match because SSL is disabled"),
errhint("Set ssl = on in postgresql.conf."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostssl record cannot match because SSL is disabled";
}
#else
@@ -998,7 +1128,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("hostssl record cannot match because SSL is not supported by this build"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostssl record cannot match because SSL is not supported by this build";
#endif
}
@@ -1010,7 +1140,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("hostgssenc record cannot match because GSSAPI is not supported by this build"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostgssenc record cannot match because GSSAPI is not supported by this build";
#endif
}
@@ -1031,7 +1161,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid connection type \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid connection type \"%s\"", token->string);
return NULL;
}
@@ -1044,7 +1174,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before database specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before database specification";
return NULL;
}
@@ -1064,7 +1194,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before role specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before role specification";
return NULL;
}
@@ -1086,7 +1216,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before IP address specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before IP address specification";
return NULL;
}
@@ -1098,7 +1228,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for host address"),
errhint("Specify one address range per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for host address";
return NULL;
}
@@ -1157,7 +1287,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid IP address \"%s\": %s",
str, gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid IP address \"%s\": %s",
str, gai_strerror(ret));
if (gai_result)
@@ -1177,7 +1307,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("specifying both host name and CIDR mask is invalid: \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("specifying both host name and CIDR mask is invalid: \"%s\"",
token->string);
return NULL;
@@ -1191,7 +1321,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid CIDR mask in address \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid CIDR mask in address \"%s\"",
token->string);
return NULL;
@@ -1211,7 +1341,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("end-of-line before netmask specification"),
errhint("Specify an address range in CIDR notation, or provide a separate netmask."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before netmask specification";
return NULL;
}
@@ -1222,7 +1352,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("multiple values specified for netmask"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for netmask";
return NULL;
}
@@ -1237,7 +1367,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid IP mask \"%s\": %s",
token->string, gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid IP mask \"%s\": %s",
token->string, gai_strerror(ret));
if (gai_result)
@@ -1256,7 +1386,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("IP address and mask do not match"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "IP address and mask do not match";
return NULL;
}
@@ -1272,7 +1402,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before authentication method"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before authentication method";
return NULL;
}
@@ -1284,7 +1414,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for authentication type"),
errhint("Specify exactly one authentication type per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for authentication type";
return NULL;
}
@@ -1321,7 +1451,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("MD5 authentication is not supported when \"db_user_namespace\" is enabled"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "MD5 authentication is not supported when \"db_user_namespace\" is enabled";
return NULL;
}
@@ -1362,7 +1492,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid authentication method \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid authentication method \"%s\"",
token->string);
return NULL;
@@ -1375,7 +1505,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid authentication method \"%s\": not supported by this build",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid authentication method \"%s\": not supported by this build",
token->string);
return NULL;
@@ -1397,7 +1527,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("gssapi authentication is not supported on local sockets"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "gssapi authentication is not supported on local sockets";
return NULL;
}
@@ -1409,7 +1539,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("peer authentication is only supported on local sockets"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "peer authentication is only supported on local sockets";
return NULL;
}
@@ -1427,7 +1557,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cert authentication is only supported on hostssl connections"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cert authentication is only supported on hostssl connections";
return NULL;
}
@@ -1477,7 +1607,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("authentication option not in name=value format: %s", token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("authentication option not in name=value format: %s",
token->string);
return NULL;
@@ -1521,7 +1651,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix";
return NULL;
}
@@ -1532,7 +1662,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set";
return NULL;
}
@@ -1548,7 +1678,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cannot use ldapsearchattribute together with ldapsearchfilter"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cannot use ldapsearchattribute together with ldapsearchfilter";
return NULL;
}
@@ -1565,7 +1695,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("list of RADIUS servers cannot be empty"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "list of RADIUS servers cannot be empty";
return NULL;
}
@@ -1576,7 +1706,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("list of RADIUS secrets cannot be empty"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "list of RADIUS secrets cannot be empty";
return NULL;
}
@@ -1595,7 +1725,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiussecrets),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS secrets (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiussecrets),
list_length(parsedline->radiusservers));
@@ -1611,7 +1741,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiusports),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS ports (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiusports),
list_length(parsedline->radiusservers));
@@ -1627,7 +1757,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiusidentifiers),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS identifiers (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiusidentifiers),
list_length(parsedline->radiusservers));
@@ -1662,6 +1792,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg)
{
int line_num = hbaline->linenumber;
+ char *file_name = hbaline->sourcefile;
#ifdef USE_LDAP
hbaline->ldapscope = LDAP_SCOPE_SUBTREE;
@@ -1685,7 +1816,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientcert can only be configured for \"hostssl\" rows"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientcert can only be configured for \"hostssl\" rows";
return false;
}
@@ -1702,7 +1833,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientcert only accepts \"verify-full\" when using \"cert\" authentication"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientcert can only be set to \"verify-full\" when using \"cert\" authentication";
return false;
}
@@ -1715,7 +1846,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid value for clientcert: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
}
@@ -1727,7 +1858,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientname can only be configured for \"hostssl\" rows"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientname can only be configured for \"hostssl\" rows";
return false;
}
@@ -1746,7 +1877,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid value for clientname: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
}
@@ -1832,7 +1963,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid ldapscheme value: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
hbaline->ldapscheme = pstrdup(val);
}
else if (strcmp(name, "ldapserver") == 0)
@@ -1850,7 +1981,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid LDAP port number: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid LDAP port number: \"%s\"", val);
return false;
}
@@ -1944,7 +2075,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS server list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -1963,7 +2094,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not translate RADIUS server name \"%s\" to address: %s",
(char *) lfirst(l), gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
if (gai_result)
pg_freeaddrinfo_all(hints.ai_family, gai_result);
@@ -1992,7 +2123,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS port list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid RADIUS port number: \"%s\"", val);
return false;
}
@@ -2005,7 +2136,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid RADIUS port number: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2028,7 +2159,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS secret list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2050,7 +2181,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS identifiers list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2064,7 +2195,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("unrecognized authentication option name: \"%s\"",
name),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("unrecognized authentication option name: \"%s\"",
name);
return false;
@@ -2212,7 +2343,7 @@ load_hba(void)
return false;
}
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
@@ -2283,6 +2414,137 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ if (strict)
+ {
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ }
+ else
+ {
+ Assert(kind == IncludedAuthFile);
+ ereport(LOG,
+ (errmsg("skipping missing authentication file \"%s\"",
+ *inc_fullname)));
+ }
+
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
+
+/*
+ * Try to open an included file, and tokenize it using the given context.
+ * Returns NULL if no error happens during tokenization, otherwise the error.
+ */
+static char *
+process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth, int elevel,
+ MemoryContext linecxt, List **tok_lines)
+{
+ char *inc_fullname;
+ FILE *inc_file;
+ char *err_msg = NULL;
+
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
+ {
+ char *err_msg;
+
+ err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", err_msg)));
+ return err_msg;
+ }
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict,
+ outer_filename, elevel, &err_msg, &inc_fullname);
+
+ if (inc_file == NULL)
+ {
+ if (strict)
+ {
+ /* open_inc_file should have reported an error. */
+ Assert(err_msg != NULL);
+ return err_msg;
+ }
+ else
+ return NULL;
+ }
+ else
+ {
+ /* No error message should have been reported. */
+ Assert(err_msg == NULL);
+ }
+
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, depth, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ return NULL;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
@@ -2301,6 +2563,7 @@ load_hba(void)
IdentLine *
parse_ident_line(TokenizedAuthLine *tok_line, int elevel)
{
+ char *file_name = tok_line->file_name;
int line_num = tok_line->line_num;
char **err_msg = &tok_line->err_msg;
ListCell *field;
@@ -2361,7 +2624,7 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid regular expression \"%s\": %s",
parsedline->ident_user + 1, errstr),
errcontext("line %d of configuration file \"%s\"",
- line_num, IdentFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid regular expression \"%s\": %s",
parsedline->ident_user + 1, errstr);
@@ -2596,7 +2859,7 @@ load_ident(void)
return false;
}
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..7433050112 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,16 +9,27 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
-# local DATABASE USER METHOD [OPTIONS]
-# host DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# local DATABASE USER METHOD [OPTIONS]
+# host DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..8e3fa29135 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,23 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
-# MAPNAME SYSTEM-USERNAME PG-USERNAME
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index c9be4bff1f..15326a01e2 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,12 +26,12 @@
static ArrayType *get_hba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int rule_number, int lineno, HbaLine *hba,
- const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int mapping_number, int lineno, IdentLine *ident,
- const char *err_msg);
+ int mapping_number, const char *filename,
+ int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
@@ -159,7 +159,7 @@ get_hba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 10
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line
@@ -168,7 +168,8 @@ get_hba_options(HbaLine *hba)
* tuple_store: where to store data
* tupdesc: tuple descriptor for the view
* rule_number: unique rule identifier among all valid rules
- * lineno: pg_hba.conf line number (must always be valid)
+ * filename: name of the file containing that line
+ * lineno: line number in that file (must always be valid)
* hba: parsed line data (can be NULL, in which case err_msg should be set)
* err_msg: error message (NULL if none)
*
@@ -177,7 +178,7 @@ get_hba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int rule_number, int lineno, HbaLine *hba,
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -202,6 +203,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -345,7 +348,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -386,7 +389,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open configuration file \"%s\": %m",
HbaFileName)));
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -407,8 +410,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
rule_number++;
- fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -445,7 +448,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
}
/* Number of columns in pg_hba_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -454,7 +457,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
* tuple_store: where to store data
* tupdesc: tuple descriptor for the view
* mapping_number: unique rule identifier among all valid rules
- * lineno: pg_ident.conf line number (must always be valid)
+ * filename: name of the file containing that line
+ * lineno: line number in that file (must always be valid)
* ident: parsed line data (can be NULL, in which case err_msg should be set)
* err_msg: error message (NULL if none)
*
@@ -463,8 +467,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int mapping_number, int lineno, IdentLine *ident,
- const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -482,6 +486,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -494,7 +500,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -534,7 +540,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open usermap file \"%s\": %m",
IdentFileName)));
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -556,7 +562,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
mapping_number++;
fill_ident_line(tuple_store, tupdesc, mapping_number,
- tok_line->line_num, identline, tok_line->err_msg);
+ tok_line->file_name, tok_line->line_num, identline,
+ tok_line->err_msg);
}
/* Free tokenizer memory */
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index 4f43430db4..ecfb4a2a28 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -345,6 +345,110 @@ GUC_flex_fatal(const char *msg)
return 0; /* keep compiler quiet */
}
+/*
+ * Returns the list of config files located in a directory, in alphabetical
+ * order.
+ *
+ * We don't check for recursion or too-deep nesting depth here, its up to the
+ * caller to take care of that.
+ */
+char **
+GetDirConfFiles(const char *includedir, const char *calling_file, int elevel,
+ int *num_filenames, char **err_msg)
+{
+ char *directory;
+ DIR *d;
+ struct dirent *de;
+ char **filenames;
+ int size_filenames;
+
+ /*
+ * Reject directory name that is all-blank (including empty), as that
+ * leads to confusion --- we'd read the containing directory, typically
+ * resulting in recursive inclusion of the same file(s).
+ */
+ if (strspn(includedir, " \t\r\n") == strlen(includedir))
+ {
+ ereport(elevel,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("empty configuration directory name: \"%s\"",
+ includedir)));
+ *err_msg = "empty configuration directory name";
+ return NULL;
+ }
+
+ directory = AbsoluteConfigLocation(includedir, calling_file);
+ d = AllocateDir(directory);
+ if (d == NULL)
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not open configuration directory \"%s\": %m",
+ directory)));
+ *err_msg = psprintf("could not open directory \"%s\"", directory);
+ filenames = NULL;
+ goto cleanup;
+ }
+
+ /*
+ * Read the directory and put the filenames in an array, so we can sort
+ * them prior to caller processing the contents.
+ */
+ size_filenames = 32;
+ filenames = (char **) palloc(size_filenames * sizeof(char *));
+ *num_filenames = 0;
+
+ while ((de = ReadDir(d, directory)) != NULL)
+ {
+ PGFileType de_type;
+ char filename[MAXPGPATH];
+
+ /*
+ * Only parse files with names ending in ".conf". Explicitly reject
+ * files starting with ".". This excludes things like "." and "..",
+ * as well as typical hidden files, backup files, and editor debris.
+ */
+ if (strlen(de->d_name) < 6)
+ continue;
+ if (de->d_name[0] == '.')
+ continue;
+ if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
+ continue;
+
+ join_path_components(filename, directory, de->d_name);
+ canonicalize_path(filename);
+ de_type = get_dirent_type(filename, de, true, elevel);
+ if (de_type == PGFILETYPE_ERROR)
+ {
+ *err_msg = psprintf("could not stat file \"%s\"", filename);
+ pfree(filenames);
+ filenames = NULL;
+ goto cleanup;
+ }
+ else if (de_type != PGFILETYPE_DIR)
+ {
+ /* Add file to array, increasing its size in blocks of 32 */
+ if (*num_filenames >= size_filenames)
+ {
+ size_filenames += 32;
+ filenames = (char **) repalloc(filenames,
+ size_filenames * sizeof(char *));
+ }
+ filenames[*num_filenames] = pstrdup(filename);
+ (*num_filenames)++;
+ }
+ }
+
+ if (*num_filenames > 0)
+ qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp);
+
+cleanup:
+ if (d)
+ FreeDir(d);
+ pfree(directory);
+ return filenames;
+}
+
/*
* Read and parse a single configuration file. This function recurses
* to handle "include" directives.
@@ -606,127 +710,30 @@ ParseConfigDirectory(const char *includedir,
ConfigVariable **head_p,
ConfigVariable **tail_p)
{
- char *directory;
- DIR *d;
- struct dirent *de;
+ char *err_msg;
char **filenames;
int num_filenames;
- int size_filenames;
- bool status;
-
- /*
- * Reject directory name that is all-blank (including empty), as that
- * leads to confusion --- we'd read the containing directory, typically
- * resulting in recursive inclusion of the same file(s).
- */
- if (strspn(includedir, " \t\r\n") == strlen(includedir))
- {
- ereport(elevel,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("empty configuration directory name: \"%s\"",
- includedir)));
- record_config_file_error("empty configuration directory name",
- calling_file, calling_lineno,
- head_p, tail_p);
- return false;
- }
-
- /*
- * We don't check for recursion or too-deep nesting depth here; the
- * subsequent calls to ParseConfigFile will take care of that.
- */
-
- directory = AbsoluteConfigLocation(includedir, calling_file);
- d = AllocateDir(directory);
- if (d == NULL)
- {
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open configuration directory \"%s\": %m",
- directory)));
- record_config_file_error(psprintf("could not open directory \"%s\"",
- directory),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
- /*
- * Read the directory and put the filenames in an array, so we can sort
- * them prior to processing the contents.
- */
- size_filenames = 32;
- filenames = (char **) palloc(size_filenames * sizeof(char *));
- num_filenames = 0;
+ filenames = GetDirConfFiles(includedir, calling_file, elevel,
+ &num_filenames, &err_msg);
- while ((de = ReadDir(d, directory)) != NULL)
+ if (!filenames)
{
- PGFileType de_type;
- char filename[MAXPGPATH];
-
- /*
- * Only parse files with names ending in ".conf". Explicitly reject
- * files starting with ".". This excludes things like "." and "..",
- * as well as typical hidden files, backup files, and editor debris.
- */
- if (strlen(de->d_name) < 6)
- continue;
- if (de->d_name[0] == '.')
- continue;
- if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
- continue;
-
- join_path_components(filename, directory, de->d_name);
- canonicalize_path(filename);
- de_type = get_dirent_type(filename, de, true, elevel);
- if (de_type == PGFILETYPE_ERROR)
- {
- record_config_file_error(psprintf("could not stat file \"%s\"",
- filename),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
- else if (de_type != PGFILETYPE_DIR)
- {
- /* Add file to array, increasing its size in blocks of 32 */
- if (num_filenames >= size_filenames)
- {
- size_filenames += 32;
- filenames = (char **) repalloc(filenames,
- size_filenames * sizeof(char *));
- }
- filenames[num_filenames] = pstrdup(filename);
- num_filenames++;
- }
+ record_config_file_error(err_msg, calling_file, calling_lineno, head_p,
+ tail_p);
+ return false;
}
- if (num_filenames > 0)
+ for (int i = 0; i < num_filenames; i++)
{
- int i;
-
- qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp);
- for (i = 0; i < num_filenames; i++)
- {
- if (!ParseConfigFile(filenames[i], true,
- calling_file, calling_lineno,
- depth, elevel,
- head_p, tail_p))
- {
- status = false;
- goto cleanup;
- }
- }
+ if (!ParseConfigFile(filenames[i], true,
+ calling_file, calling_lineno,
+ depth, elevel,
+ head_p, tail_p))
+ return false;
}
- status = true;
-cleanup:
- if (d)
- FreeDir(d);
- pfree(directory);
- return status;
+ return true;
}
/*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 060bafaced..e6a54cc3d6 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6128,16 +6128,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '6250', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,text,text,text}',
- proargmodes => '{o,o,o,o,o,o}',
- proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index 90036f7bcd..0ea100d1b8 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -155,6 +156,7 @@ typedef struct AuthToken
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of AuthTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
@@ -174,6 +176,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel);
extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel);
extern bool pg_isblank(const char c);
extern MemoryContext tokenize_auth_file(const char *filename, FILE *file,
- List **tok_lines, int elevel);
+ List **tok_lines, int depth,
+ int elevel);
#endif /* HBA_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 45ae1b537f..cb3bbcf949 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -145,6 +145,8 @@ typedef struct ConfigVariable
struct ConfigVariable *next;
} ConfigVariable;
+extern char **GetDirConfFiles(const char *includedir, const char *calling_file,
+ int elevel, int *num_filenames, char **err_msg);
extern bool ParseConfigFile(const char *config_file, bool strict,
const char *calling_file, int calling_lineno,
int depth, int elevel,
diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl
new file mode 100644
index 0000000000..8eae72b8d4
--- /dev/null
+++ b/src/test/authentication/t/003_file_inclusion.pl
@@ -0,0 +1,657 @@
+
+# Copyright (c) 2021-2022, PostgreSQL Global Development Group
+
+# Set of tests for authentication and pg_hba.conf inclusion.
+# This test can only run with Unix-domain sockets.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Time::HiRes qw(usleep);
+use IPC::Run qw(pump finish timer);
+use Data::Dumper;
+
+if (!$use_unix_sockets)
+{
+ plan skip_all =>
+ "authentication tests cannot run without Unix-domain sockets";
+}
+
+# stores the current line counter for each file. hba_rule and ident_rule are
+# fake file names used for the global rule number for each auth view.
+my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+my $hba_file = 'subdir1/pg_hba_custom.conf';
+my $ident_file = 'subdir2/pg_ident_custom.conf';
+
+# Initialize primary node
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+$node->start;
+
+my $data_dir = $node->data_dir;
+
+# Normalize the data directory for Windows
+$data_dir =~ s/\/\.\//\//g; # reduce /./ to /
+$data_dir =~ s/\/\//\//g; # reduce // to /
+$data_dir =~ s/\/$//; # remove trailing /
+
+
+# Add the given payload to the given relative HBA file of the given node.
+# This function maintains the %cur_line metadata, so it has to be called in the
+# expected inclusion evaluation order in order to keep it in sync.
+#
+# If the payload starts with "include" or "ignore", the function doesn't
+# increase the general hba rule number.
+#
+# If an err_str is provided, it returns an arrayref containing the provided
+# filename, the current line number in that file and the provided err_str. The
+# err_str has to be a valid regex string.
+# Otherwise it only returns the line number of the payload in the wanted file.
+# This function has to be called in the expected inclusion evaluation order to
+# keep the %cur_line information in sync.
+sub add_hba_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'hba_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_hba_file_rules line
+ @tokens = split(/ /, $payload);
+ $tokens[1] = '{' . $tokens[1] . '}'; # database
+ $tokens[2] = '{' . $tokens[2] . '}'; # user_name
+
+ # add empty address and netmask betweed user_name and auth_method
+ splice @tokens, 3, 0, '';
+ splice @tokens, 3, 0, '';
+
+ # append empty options and error
+ push @tokens, '';
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Add the given payload to the given relative ident file of the given node.
+# Same as add_hba_line but for pg_ident files
+sub add_ident_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'ident_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_ident_file_mappings line
+ @tokens = split(/ /, $payload);
+
+ # append empty error
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Delete pg_hba.conf from the given node, add various entries to test the
+# include infrastructure and then execute a reload to refresh it.
+sub generate_valid_auth_files
+{
+ my $node = shift;
+ my $hba_expected = '';
+ my $ident_expected = '';
+
+ # customise main auth file names
+ $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'");
+ $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'");
+
+ # and make original ones invalid to be sure they're not used anywhere
+ $node->append_conf('pg_hba.conf', "some invalid line");
+ $node->append_conf('pg_ident.conf', "some invalid line");
+
+ # pg_hba stuff
+ mkdir("$data_dir/subdir1");
+ mkdir("$data_dir/hba_inc");
+ mkdir("$data_dir/hba_inc_if");
+ mkdir("$data_dir/hba_pos");
+
+ # Make sure we will still be able to connect
+ $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust');
+
+ # Add include data
+ add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf");
+ $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject");
+
+ $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject");
+
+ add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject");
+ # include is relative to current path
+ add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject");
+
+ # include_if_exists data
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none");
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some");
+ $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject");
+
+ # include_dir data
+ add_hba_line($node, "$hba_file", "include_dir ../hba_inc");
+ add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included");
+ $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject");
+ $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject");
+
+ # secondary auth file
+ add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject');
+ $node->append_conf('dbnames.conf', "db1");
+ $node->append_conf('dbnames.conf', "db3");
+ $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1)
+ . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1)
+ . '|local|{db1,db3}|{all}|||reject||';
+
+ # pg_ident stuff
+ mkdir("$data_dir/subdir2");
+ mkdir("$data_dir/ident_inc");
+ mkdir("$data_dir/ident_inc_if");
+ mkdir("$data_dir/ident_pos");
+
+ # Add include data
+ add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf");
+ $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar");
+
+ $ident_expected .= add_ident_line($node, "$ident_file", "test a b");
+
+ add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar");
+ # include is relative to current path
+ add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar");
+
+ # include_if_exists data
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none");
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some");
+ $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar");
+
+ # include_dir data
+ add_ident_line($node, "$ident_file", "include_dir ../ident_inc");
+ add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included");
+ $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar");
+ $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar");
+
+ $node->restart;
+ $node->connect_ok('dbname=postgres',
+ 'Connection ok after generating valid auth files');
+
+ return ($hba_expected, $ident_expected);
+}
+
+# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal
+# entries to allow authentication.
+sub reset_auth_files
+{
+ my $node = shift;
+
+ unlink("$data_dir/$hba_file");
+ unlink("$data_dir/$ident_file");
+
+ %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+ return add_hba_line($node, "$hba_file", 'local all all trust');
+}
+
+# Generate a list of expected error regex for the given array of error
+# conditions, as generated by add_hba_line/add_ident_line with an err_str.
+#
+# 2 regex are generated per array entry: one for the given err_str, and one for
+# the expected line in the specific file. Since all lines are independant,
+# there's no guarantee that a specific failure regex and the per-line regex
+# will match the same error. Calling code should add at least one test with a
+# single error to make sure that the line number / file name is correct.
+#
+# On top of that, an extra line is generated for the general failure to process
+# the main auth file.
+sub generate_log_err_patterns
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $is_hba_err = shift;
+ my @errors;
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ push @errors, qr/$err_str/;
+
+ # Context messages with the file / line location aren't always emitted
+ if ($err_str !~ /maximum nesting depth exceeded/ and
+ $err_str !~ /could not open secondary authentication file/)
+ {
+ push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/
+ }
+ }
+
+ push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err);
+
+ return \@errors;
+}
+
+# Generate the expected output for the auth file view error reporting (file
+# name, file line, error), for the given array of error conditions, as
+# generated generated by add_hba_line/add_ident_line with an err_str.
+sub generate_log_err_rows
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $exp_rows = '';
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ $exp_rows .= "\n" if ($exp_rows ne "");
+
+ # Unescape regex patterns if any
+ $err_str =~ s/\\([\(\)])/$1/g;
+ $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str"
+ }
+
+ return $exp_rows;
+}
+
+# Reset the main auth files, append the given payload to the given config file,
+# and check that the instance cannot start, raising the expected error line(s).
+sub start_errors_like
+{
+ my $node = shift;
+ my $file = shift;
+ my $payload = shift;
+ my $pattern = shift;
+ my $should_fail = shift;
+
+ reset_auth_files($node);
+ $node->append_conf($file, $payload);
+
+ unlink($node->logfile);
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir,
+ '-l', $node->logfile, 'start');
+
+ if ($should_fail)
+ {
+ ok($ret != 0, "Cannot start postgres with faulty $file");
+ }
+ else
+ {
+ ok($ret == 0, "postgres can start with faulty $file");
+ }
+
+ my $log_contents = slurp_file($node->logfile);
+
+ foreach (@{$pattern})
+ {
+ like($log_contents,
+ $_,
+ "Expected failure found in the logs");
+ }
+
+ if (not $should_fail)
+ {
+ # We can't simply call $node->stop here as the call is optimized out
+ # when the server isn't started with $node->start.
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D',
+ $data_dir, 'stop', '-m', 'fast');
+ ok($ret == 0, "Could stop postgres");
+ }
+}
+
+# We should be able to connect, and see an empty pg_ident.conf
+is($node->psql(
+ 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'),
+ qq(0),
+ 'pg_ident.conf is empty');
+
+############################################
+# part 1, test view reporting for valid data
+############################################
+my ($exp_hba, $exp_ident) = generate_valid_auth_files($node);
+
+$node->connect_ok('dbname=postgres', 'Connection still ok');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_hba_file_rules'),
+ qq($exp_hba),
+ 'pg_hba_file_rules content is expected');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_ident_file_mappings'),
+ qq($exp_ident),
+ 'pg_ident_file_mappings content is expected');
+
+#############################################
+# part 2, test log reporting for invalid data
+#############################################
+reset_auth_files($node);
+$node->restart('fast');
+$node->connect_ok('dbname=postgres',
+ 'Connection ok after resetting auth files');
+
+$node->stop('fast');
+
+start_errors_like($node, $hba_file, "include ../not_a_file",
+ [
+ qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file
+mkdir("$data_dir/hba_inc_fail");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token");
+start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail",
+ [
+ qr/invalid connection type "not_a_token"/,
+ qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file with nested inclusion
+unlink("$data_dir/hba_inc_fail/inc_dir.conf");
+my @hba_raw_errors_step1;
+
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1");
+
+add_hba_line($node, "hba_inc_fail/file1", "include file2");
+add_hba_line($node, "hba_inc_fail/file2", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file2", "include file3");
+
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3",
+ "local all all zuul",
+ 'invalid authentication method "zuul"');
+
+start_errors_like(
+ $node, $hba_file, "include_dir ../hba_inc_fail",
+ generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @hba_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local",
+ "end-of-line before database specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local,host",
+ "multiple values specified for connection type");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all",
+ "end-of-line before role specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all all",
+ "end-of-line before authentication method");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "host all all test/42",
+ 'specifying both host name and CIDR mask is invalid: "test/42"');
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ 'local @dbnames_fails.conf all reject',
+ "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory");
+
+add_hba_line($node, "hba_if_exists.conf", "include recurse.conf");
+push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf",
+ "include recurse.conf",
+ 'could not open configuration file "recurse.conf": maximum nesting depth exceeded');
+
+# Generate the regex for the expected errors in the logs. There's no guarantee
+# that the generated "line X of file..." will be emitted for the expected line,
+# but previous tests already ensured that the correct line number / file name
+# was emitted, so ensuring that there's an error in all expected lines is
+# enough here.
+my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2,
+ 1);
+
+# Not an error, but it should raise a message in the logs. Manually add an
+# extra log message to detect
+add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none");
+push @{$expected_errors},
+ qr/skipping missing authentication file "$data_dir\/if_exists_none"/;
+
+start_errors_like(
+ $node, $hba_file, "include_if_exists ../hba_if_exists.conf",
+ $expected_errors, 1);
+
+# Mostly the same, but for ident files
+reset_auth_files($node);
+
+my @ident_raw_errors_step1;
+
+# include_dir, single included file with nested inclusion
+mkdir("$data_dir/ident_inc_fail");
+add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1");
+
+add_ident_line($node, "ident_inc_fail/file1", "include file2");
+add_ident_line($node, "ident_inc_fail/file2", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file2", "include file3");
+
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3",
+ "failmap /(fail postgres",
+ 'invalid regular expression "\(fail": parentheses \(\) not balanced');
+
+start_errors_like(
+ $node, $ident_file, "include_dir ../ident_inc_fail",
+ generate_log_err_patterns($node, \@ident_raw_errors_step1, 0),
+ 0);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @ident_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map",
+ "missing entry at end of line");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2",
+ "multiple values in ident field");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf",
+ 'map @osnames_fails.conf postgres',
+ "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory");
+
+add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf",
+ 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded');
+
+start_errors_like(
+ $node, $ident_file, "include_if_exists ../ident_if_exists.conf",
+ # There's no guarantee that the generated "line X of file..." will be
+ # emitted for the expected line, but previous tests already ensured that
+ # the correct line number / file name was emitted, so ensuring that there's
+ # an error in all expected lines is enough here.
+ generate_log_err_patterns($node, \@ident_raw_errors_step2, 0),
+ 0);
+
+#####################################################
+# part 3, test reporting of various error scenario
+# NOTE: this will be bypassed -DEXEC_BACKEND or win32
+#####################################################
+reset_auth_files($node);
+
+$node->start;
+$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset');
+
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_hba_file_rules');
+
+add_ident_line($node, $ident_file, '');
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_ident_file_mappings');
+
+# The instance could be restarted and no error is detected. Now check if the
+# build is compatible with the view error reporting (EXEC_BACKEND / win32 will
+# fail when trying to connect as they always rely on the current auth files
+# content)
+my @hba_raw_errors;
+
+push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+my ($stdout, $stderr);
+my $cmdret = $node->psql('postgres', 'SELECT 1',
+ stdout => \$stdout, stderr => \$stderr);
+
+if ($cmdret != 0)
+{
+ # Connection failed. Bail out, but make sure to raise a failure if it
+ # didn't fail for the expected hba file modification.
+ like($stderr,
+ qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/,
+ "Connection failed due to loading an invalid hba file");
+
+ done_testing();
+ diag("Build not compatible with auth file view error reporting, bail out.\n");
+ exit;
+}
+
+# Combine errors generated at step 2, in the same order.
+$node->append_conf($hba_file, "include_dir ../hba_inc_fail");
+push @hba_raw_errors, @hba_raw_errors_step1;
+
+$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf");
+push @hba_raw_errors, @hba_raw_errors_step2;
+
+my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules'
+ . ' WHERE error IS NOT NULL ORDER BY rule_number'),
+ qq($hba_expected),
+ 'Detected all error in hba file');
+
+# and do the same for pg_ident
+my @ident_raw_errors;
+
+push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+$node->append_conf($ident_file, "include_dir ../ident_inc_fail");
+push @ident_raw_errors, @ident_raw_errors_step1;
+
+$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf");
+push @ident_raw_errors, @ident_raw_errors_step2;
+
+my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings'
+ . ' WHERE error IS NOT NULL ORDER BY mapping_number'),
+ qq($ident_expected),
+ 'Detected all error in ident file');
+
+done_testing();
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 79408710e0..5ed2fe3704 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname,
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
a.line_number,
a.type,
a.database,
@@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
a.line_number,
a.map_name,
a.sys_name,
a.pg_username,
a.error
- FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.37.0
--fy3hxyvyv2spu3wq
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v10-0003-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v11 2/3] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-05-30 03:15 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-05-30 03:15 UTC (permalink / raw)
pg_hba.conf file now has support for "include", "include_dir" and
"include_if_exists" directives, which work similarly to the same directives in
the postgresql.conf file.
This fixes a possible crash if a secondary file tries to include itself as
there's now a nesting depth check in the inclusion code path, same as the
postgresql.conf.
Many regression tests added to cover both the new directives, but also error
detection for the whole pg_hba / pg_ident files.
Catversion is bumped.
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/client-auth.sgml | 86 ++-
doc/src/sgml/system-views.sgml | 22 +-
src/backend/libpq/hba.c | 481 ++++++++++---
src/backend/libpq/pg_hba.conf.sample | 25 +-
src/backend/libpq/pg_ident.conf.sample | 15 +-
src/backend/utils/adt/hbafuncs.c | 43 +-
src/backend/utils/misc/guc-file.l | 229 +++---
src/include/catalog/pg_proc.dat | 12 +-
src/include/libpq/hba.h | 5 +-
src/include/utils/guc.h | 2 +
.../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++
src/test/regress/expected/rules.out | 6 +-
12 files changed, 1310 insertions(+), 273 deletions(-)
create mode 100644 src/test/authentication/t/003_file_inclusion.pl
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index c6f1b70fd3..42ceb6f3e6 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,23 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication
+ record. Inclusion directives specify files that can be included, which
+ contains additional records. The records will be inserted in lieu of the
+ inclusion records. Those records only contains two fields: the
+ <literal>include</literal>, <literal>include_if_exists</literal> or
+ <literal>include_dir</literal> directive and the file or directory to be
+ included. The file or directory can be a relative of absolute path, and can
+ be double quoted if needed. For the <literal>include_dir</literal> form,
+ all files not starting with a <literal>.</literal> and ending with
+ <literal>.conf</literal> will be included. Multiple files within an include
+ directory are processed in file name order (according to C locale rules,
+ i.e., numbers before letters, and uppercase letters before lowercase ones).
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,21 +118,57 @@
<para>
A record can have several formats:
<synopsis>
-local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+include <replaceable>file</replaceable>
+include_if_exists <replaceable>file</replaceable>
+include_dir <replaceable>directory</replaceable>
+local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
</synopsis>
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_if_exists</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file if the
+ file exists and can be read. Otherwise, a message will be logged to
+ indicate that the file is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_dir</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of all the files found in
+ the directory, if they don't start with a <literal>.</literal> and end
+ with <literal>.conf</literal>, processed in file name order (according
+ to C locale rules, i.e., numbers before letters, and uppercase letters
+ before lowercase ones).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -835,8 +886,10 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
+<replaceable>include_dir</replaceable> <replaceable>directory</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -847,6 +900,11 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ As for <filename>pg_hba.conf</filename>, the lines in this file can either
+ be inclusion directives or user name map records, and follow the same
+ rules.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 4723f712a7..7d1cec8b7f 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -1003,12 +1003,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule the given <literal>file_name</literal>
</para></entry>
</row>
@@ -1153,12 +1162,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this mapping
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_ident.conf</filename>
+ Line number of this mapping in the given <literal>file_name</literal>
</para></entry>
</row>
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index 4637426d62..4e4a45b793 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -22,6 +22,7 @@
#include <sys/param.h>
#include <sys/socket.h>
#include <netdb.h>
+#include <sys/stat.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
@@ -69,6 +70,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -113,10 +120,22 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int depth,
+ int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
- const char *inc_filename, int elevel, char **err_msg);
+ const char *inc_filename, int depth, int elevel,
+ char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ bool strict, const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
+static char *process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth,
+ int elevel, MemoryContext linecxt,
+ List **tok_lines);
/*
@@ -303,7 +322,7 @@ copy_auth_token(AuthToken *in)
*/
static List *
next_field_expand(const char *filename, char **lineptr,
- int elevel, char **err_msg)
+ int depth, int elevel, char **err_msg)
{
char buf[MAX_TOKEN];
bool trailing_comma;
@@ -319,7 +338,7 @@ next_field_expand(const char *filename, char **lineptr,
/* Is this referencing a file? */
if (!initial_quote && buf[0] == '@' && buf[1] != '\0')
- tokens = tokenize_inc_file(tokens, filename, buf + 1,
+ tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1,
elevel, err_msg);
else
tokens = lappend(tokens, make_auth_token(buf, initial_quote));
@@ -347,6 +366,7 @@ static List *
tokenize_inc_file(List *tokens,
const char *outer_filename,
const char *inc_filename,
+ int depth,
int elevel,
char **err_msg)
{
@@ -356,39 +376,30 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
{
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
+ *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", *err_msg)));
+ return tokens;
}
- inc_file = AllocateFile(inc_fullname, "r");
- if (inc_file == NULL)
- {
- int save_errno = errno;
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true,
+ outer_filename, elevel, err_msg, &inc_fullname);
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
+ if (inc_file == NULL)
return tokens;
- }
/* There is possible recursion here if the file contains @ */
- linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
+ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1,
+ elevel);
FreeFile(inc_file);
pfree(inc_fullname);
@@ -426,11 +437,38 @@ tokenize_inc_file(List *tokens,
/*
* tokenize_auth_file
- * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a dedicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
+ int depth, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, depth,
+ elevel);
+
+ return linecxt;
+}
+
+/*
+ * Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see the struct definition
* in libpq/hba.h.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -439,30 +477,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int depth, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -515,7 +545,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
{
List *current_field;
- current_field = next_field_expand(filename, &lineptr,
+ current_field = next_field_expand(filename, &lineptr, depth,
elevel, &err_msg);
/* add field to line, unless we are at EOL or comment start */
if (current_field != NIL)
@@ -523,29 +553,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ AuthToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
+
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
+
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, true,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
+ else if (strcmp(first->string, "include_dir") == 0)
+ {
+ char **filenames;
+ char *dir_name = second->string;
+ int num_filenames;
+ StringInfoData err_buf;
+
+ filenames = GetDirConfFiles(dir_name, filename, elevel,
+ &num_filenames, &err_msg);
+
+ if (!filenames)
+ {
+ /* We have the error in err_msg, simply process it */
+ goto process_line;
+ }
+
+ initStringInfo(&err_buf);
+ for (int i = 0; i < num_filenames; i++)
+ {
+ /*
+ * err_msg is used here as a temp buffer, it will be
+ * overwritten at the end of the loop with the
+ * cumulated errors, if any.
+ */
+ err_msg = process_included_authfile(filenames[i], true,
+ filename, depth + 1, elevel,
+ linecxt, tok_lines);
+
+ /* Cumulate errors if any. */
+ if (err_msg)
+ {
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfoString(&err_buf, err_msg);
+ }
+ }
+
+ /*
+ * If there were no errors, the line is fully processed, bypass
+ * the general TokenizedAuthLine processing.
+ */
+ if (err_buf.len == 0)
+ goto next_line;
+
+ /* Otherwise, process the cumulated errors, if any. */
+ err_msg = err_buf.data;
+ }
+ else if (strcmp(first->string, "include_if_exists") == 0)
+ {
+ char *inc_filename;
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, false,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
}
+process_line:
+ /*
+ * General processing: report the error if any and emit line to the
+ * TokenizedAuthLine
+ */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -855,7 +983,7 @@ do { \
errmsg("authentication option \"%s\" is only valid for authentication methods %s", \
optname, _(validmethods)), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \
optname, validmethods); \
return false; \
@@ -875,7 +1003,7 @@ do { \
errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname); \
return NULL; \
@@ -898,7 +1026,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("missing entry at end of line"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, file_name))); \
*err_msg = pstrdup("missing entry at end of line"); \
return NULL; \
} \
@@ -911,7 +1039,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("multiple values in ident field"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, file_name))); \
*err_msg = pstrdup("multiple values in ident field"); \
return NULL; \
} \
@@ -934,6 +1062,7 @@ HbaLine *
parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
{
int line_num = tok_line->line_num;
+ char *file_name = tok_line->file_name;
char **err_msg = &tok_line->err_msg;
char *str;
struct addrinfo *gai_result;
@@ -948,6 +1077,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -962,7 +1092,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for connection type"),
errhint("Specify exactly one connection type per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for connection type";
return NULL;
}
@@ -990,7 +1120,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("hostssl record cannot match because SSL is disabled"),
errhint("Set ssl = on in postgresql.conf."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostssl record cannot match because SSL is disabled";
}
#else
@@ -998,7 +1128,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("hostssl record cannot match because SSL is not supported by this build"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostssl record cannot match because SSL is not supported by this build";
#endif
}
@@ -1010,7 +1140,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("hostgssenc record cannot match because GSSAPI is not supported by this build"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostgssenc record cannot match because GSSAPI is not supported by this build";
#endif
}
@@ -1031,7 +1161,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid connection type \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid connection type \"%s\"", token->string);
return NULL;
}
@@ -1044,7 +1174,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before database specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before database specification";
return NULL;
}
@@ -1064,7 +1194,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before role specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before role specification";
return NULL;
}
@@ -1086,7 +1216,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before IP address specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before IP address specification";
return NULL;
}
@@ -1098,7 +1228,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for host address"),
errhint("Specify one address range per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for host address";
return NULL;
}
@@ -1157,7 +1287,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid IP address \"%s\": %s",
str, gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid IP address \"%s\": %s",
str, gai_strerror(ret));
if (gai_result)
@@ -1177,7 +1307,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("specifying both host name and CIDR mask is invalid: \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("specifying both host name and CIDR mask is invalid: \"%s\"",
token->string);
return NULL;
@@ -1191,7 +1321,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid CIDR mask in address \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid CIDR mask in address \"%s\"",
token->string);
return NULL;
@@ -1211,7 +1341,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("end-of-line before netmask specification"),
errhint("Specify an address range in CIDR notation, or provide a separate netmask."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before netmask specification";
return NULL;
}
@@ -1222,7 +1352,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("multiple values specified for netmask"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for netmask";
return NULL;
}
@@ -1237,7 +1367,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid IP mask \"%s\": %s",
token->string, gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid IP mask \"%s\": %s",
token->string, gai_strerror(ret));
if (gai_result)
@@ -1256,7 +1386,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("IP address and mask do not match"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "IP address and mask do not match";
return NULL;
}
@@ -1272,7 +1402,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before authentication method"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before authentication method";
return NULL;
}
@@ -1284,7 +1414,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for authentication type"),
errhint("Specify exactly one authentication type per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for authentication type";
return NULL;
}
@@ -1321,7 +1451,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("MD5 authentication is not supported when \"db_user_namespace\" is enabled"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "MD5 authentication is not supported when \"db_user_namespace\" is enabled";
return NULL;
}
@@ -1362,7 +1492,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid authentication method \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid authentication method \"%s\"",
token->string);
return NULL;
@@ -1375,7 +1505,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid authentication method \"%s\": not supported by this build",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid authentication method \"%s\": not supported by this build",
token->string);
return NULL;
@@ -1397,7 +1527,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("gssapi authentication is not supported on local sockets"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "gssapi authentication is not supported on local sockets";
return NULL;
}
@@ -1409,7 +1539,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("peer authentication is only supported on local sockets"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "peer authentication is only supported on local sockets";
return NULL;
}
@@ -1427,7 +1557,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cert authentication is only supported on hostssl connections"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cert authentication is only supported on hostssl connections";
return NULL;
}
@@ -1477,7 +1607,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("authentication option not in name=value format: %s", token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("authentication option not in name=value format: %s",
token->string);
return NULL;
@@ -1521,7 +1651,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix";
return NULL;
}
@@ -1532,7 +1662,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set";
return NULL;
}
@@ -1548,7 +1678,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cannot use ldapsearchattribute together with ldapsearchfilter"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cannot use ldapsearchattribute together with ldapsearchfilter";
return NULL;
}
@@ -1565,7 +1695,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("list of RADIUS servers cannot be empty"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "list of RADIUS servers cannot be empty";
return NULL;
}
@@ -1576,7 +1706,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("list of RADIUS secrets cannot be empty"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "list of RADIUS secrets cannot be empty";
return NULL;
}
@@ -1595,7 +1725,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiussecrets),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS secrets (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiussecrets),
list_length(parsedline->radiusservers));
@@ -1611,7 +1741,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiusports),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS ports (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiusports),
list_length(parsedline->radiusservers));
@@ -1627,7 +1757,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiusidentifiers),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS identifiers (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiusidentifiers),
list_length(parsedline->radiusservers));
@@ -1662,6 +1792,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg)
{
int line_num = hbaline->linenumber;
+ char *file_name = hbaline->sourcefile;
#ifdef USE_LDAP
hbaline->ldapscope = LDAP_SCOPE_SUBTREE;
@@ -1685,7 +1816,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientcert can only be configured for \"hostssl\" rows"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientcert can only be configured for \"hostssl\" rows";
return false;
}
@@ -1702,7 +1833,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientcert only accepts \"verify-full\" when using \"cert\" authentication"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientcert can only be set to \"verify-full\" when using \"cert\" authentication";
return false;
}
@@ -1715,7 +1846,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid value for clientcert: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
}
@@ -1727,7 +1858,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientname can only be configured for \"hostssl\" rows"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientname can only be configured for \"hostssl\" rows";
return false;
}
@@ -1746,7 +1877,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid value for clientname: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
}
@@ -1832,7 +1963,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid ldapscheme value: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
hbaline->ldapscheme = pstrdup(val);
}
else if (strcmp(name, "ldapserver") == 0)
@@ -1850,7 +1981,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid LDAP port number: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid LDAP port number: \"%s\"", val);
return false;
}
@@ -1944,7 +2075,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS server list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -1963,7 +2094,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not translate RADIUS server name \"%s\" to address: %s",
(char *) lfirst(l), gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
if (gai_result)
pg_freeaddrinfo_all(hints.ai_family, gai_result);
@@ -1992,7 +2123,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS port list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid RADIUS port number: \"%s\"", val);
return false;
}
@@ -2005,7 +2136,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid RADIUS port number: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2028,7 +2159,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS secret list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2050,7 +2181,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS identifiers list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2064,7 +2195,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("unrecognized authentication option name: \"%s\"",
name),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("unrecognized authentication option name: \"%s\"",
name);
return false;
@@ -2212,7 +2343,7 @@ load_hba(void)
return false;
}
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
@@ -2283,6 +2414,135 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ if (strict)
+ {
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ }
+ else
+ {
+ Assert(kind == IncludedAuthFile);
+ ereport(LOG,
+ (errmsg("skipping missing authentication file \"%s\"",
+ *inc_fullname)));
+ }
+
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
+
+/*
+ * Try to open an included file, and tokenize it using the given context.
+ * Returns NULL if no error happens during tokenization, otherwise the error.
+ */
+static char *
+process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth, int elevel,
+ MemoryContext linecxt, List **tok_lines)
+{
+ char *inc_fullname;
+ FILE *inc_file;
+ char *err_msg = NULL;
+
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
+ {
+ err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", err_msg)));
+ return err_msg;
+ }
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict,
+ outer_filename, elevel, &err_msg, &inc_fullname);
+
+ if (inc_file == NULL)
+ {
+ if (strict)
+ {
+ /* open_inc_file should have reported an error. */
+ Assert(err_msg != NULL);
+ return err_msg;
+ }
+ else
+ return NULL;
+ }
+ else
+ {
+ /* No error message should have been reported. */
+ Assert(err_msg == NULL);
+ }
+
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, depth, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ return NULL;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
@@ -2301,6 +2561,7 @@ load_hba(void)
IdentLine *
parse_ident_line(TokenizedAuthLine *tok_line, int elevel)
{
+ char *file_name = tok_line->file_name;
int line_num = tok_line->line_num;
char **err_msg = &tok_line->err_msg;
ListCell *field;
@@ -2361,7 +2622,7 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid regular expression \"%s\": %s",
parsedline->ident_user + 1, errstr),
errcontext("line %d of configuration file \"%s\"",
- line_num, IdentFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid regular expression \"%s\": %s",
parsedline->ident_user + 1, errstr);
@@ -2596,7 +2857,7 @@ load_ident(void)
return false;
}
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..7433050112 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,16 +9,27 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
-# local DATABASE USER METHOD [OPTIONS]
-# host DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# local DATABASE USER METHOD [OPTIONS]
+# host DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..8e3fa29135 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,23 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
-# MAPNAME SYSTEM-USERNAME PG-USERNAME
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index c9be4bff1f..15326a01e2 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,12 +26,12 @@
static ArrayType *get_hba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int rule_number, int lineno, HbaLine *hba,
- const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int mapping_number, int lineno, IdentLine *ident,
- const char *err_msg);
+ int mapping_number, const char *filename,
+ int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
@@ -159,7 +159,7 @@ get_hba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 10
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line
@@ -168,7 +168,8 @@ get_hba_options(HbaLine *hba)
* tuple_store: where to store data
* tupdesc: tuple descriptor for the view
* rule_number: unique rule identifier among all valid rules
- * lineno: pg_hba.conf line number (must always be valid)
+ * filename: name of the file containing that line
+ * lineno: line number in that file (must always be valid)
* hba: parsed line data (can be NULL, in which case err_msg should be set)
* err_msg: error message (NULL if none)
*
@@ -177,7 +178,7 @@ get_hba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int rule_number, int lineno, HbaLine *hba,
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -202,6 +203,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -345,7 +348,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -386,7 +389,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open configuration file \"%s\": %m",
HbaFileName)));
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -407,8 +410,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
rule_number++;
- fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -445,7 +448,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
}
/* Number of columns in pg_hba_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -454,7 +457,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
* tuple_store: where to store data
* tupdesc: tuple descriptor for the view
* mapping_number: unique rule identifier among all valid rules
- * lineno: pg_ident.conf line number (must always be valid)
+ * filename: name of the file containing that line
+ * lineno: line number in that file (must always be valid)
* ident: parsed line data (can be NULL, in which case err_msg should be set)
* err_msg: error message (NULL if none)
*
@@ -463,8 +467,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int mapping_number, int lineno, IdentLine *ident,
- const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -482,6 +486,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -494,7 +500,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -534,7 +540,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open usermap file \"%s\": %m",
IdentFileName)));
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -556,7 +562,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
mapping_number++;
fill_ident_line(tuple_store, tupdesc, mapping_number,
- tok_line->line_num, identline, tok_line->err_msg);
+ tok_line->file_name, tok_line->line_num, identline,
+ tok_line->err_msg);
}
/* Free tokenizer memory */
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index 721628c0cf..86b6cc1c8a 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -345,6 +345,110 @@ GUC_flex_fatal(const char *msg)
return 0; /* keep compiler quiet */
}
+/*
+ * Returns the list of config files located in a directory, in alphabetical
+ * order.
+ *
+ * We don't check for recursion or too-deep nesting depth here, its up to the
+ * caller to take care of that.
+ */
+char **
+GetDirConfFiles(const char *includedir, const char *calling_file, int elevel,
+ int *num_filenames, char **err_msg)
+{
+ char *directory;
+ DIR *d;
+ struct dirent *de;
+ char **filenames;
+ int size_filenames;
+
+ /*
+ * Reject directory name that is all-blank (including empty), as that
+ * leads to confusion --- we'd read the containing directory, typically
+ * resulting in recursive inclusion of the same file(s).
+ */
+ if (strspn(includedir, " \t\r\n") == strlen(includedir))
+ {
+ ereport(elevel,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("empty configuration directory name: \"%s\"",
+ includedir)));
+ *err_msg = "empty configuration directory name";
+ return NULL;
+ }
+
+ directory = AbsoluteConfigLocation(includedir, calling_file);
+ d = AllocateDir(directory);
+ if (d == NULL)
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not open configuration directory \"%s\": %m",
+ directory)));
+ *err_msg = psprintf("could not open directory \"%s\"", directory);
+ filenames = NULL;
+ goto cleanup;
+ }
+
+ /*
+ * Read the directory and put the filenames in an array, so we can sort
+ * them prior to caller processing the contents.
+ */
+ size_filenames = 32;
+ filenames = (char **) palloc(size_filenames * sizeof(char *));
+ *num_filenames = 0;
+
+ while ((de = ReadDir(d, directory)) != NULL)
+ {
+ PGFileType de_type;
+ char filename[MAXPGPATH];
+
+ /*
+ * Only parse files with names ending in ".conf". Explicitly reject
+ * files starting with ".". This excludes things like "." and "..",
+ * as well as typical hidden files, backup files, and editor debris.
+ */
+ if (strlen(de->d_name) < 6)
+ continue;
+ if (de->d_name[0] == '.')
+ continue;
+ if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
+ continue;
+
+ join_path_components(filename, directory, de->d_name);
+ canonicalize_path(filename);
+ de_type = get_dirent_type(filename, de, true, elevel);
+ if (de_type == PGFILETYPE_ERROR)
+ {
+ *err_msg = psprintf("could not stat file \"%s\"", filename);
+ pfree(filenames);
+ filenames = NULL;
+ goto cleanup;
+ }
+ else if (de_type != PGFILETYPE_DIR)
+ {
+ /* Add file to array, increasing its size in blocks of 32 */
+ if (*num_filenames >= size_filenames)
+ {
+ size_filenames += 32;
+ filenames = (char **) repalloc(filenames,
+ size_filenames * sizeof(char *));
+ }
+ filenames[*num_filenames] = pstrdup(filename);
+ (*num_filenames)++;
+ }
+ }
+
+ if (*num_filenames > 0)
+ qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp);
+
+cleanup:
+ if (d)
+ FreeDir(d);
+ pfree(directory);
+ return filenames;
+}
+
/*
* Read and parse a single configuration file. This function recurses
* to handle "include" directives.
@@ -606,127 +710,30 @@ ParseConfigDirectory(const char *includedir,
ConfigVariable **head_p,
ConfigVariable **tail_p)
{
- char *directory;
- DIR *d;
- struct dirent *de;
+ char *err_msg;
char **filenames;
int num_filenames;
- int size_filenames;
- bool status;
-
- /*
- * Reject directory name that is all-blank (including empty), as that
- * leads to confusion --- we'd read the containing directory, typically
- * resulting in recursive inclusion of the same file(s).
- */
- if (strspn(includedir, " \t\r\n") == strlen(includedir))
- {
- ereport(elevel,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("empty configuration directory name: \"%s\"",
- includedir)));
- record_config_file_error("empty configuration directory name",
- calling_file, calling_lineno,
- head_p, tail_p);
- return false;
- }
-
- /*
- * We don't check for recursion or too-deep nesting depth here; the
- * subsequent calls to ParseConfigFile will take care of that.
- */
-
- directory = AbsoluteConfigLocation(includedir, calling_file);
- d = AllocateDir(directory);
- if (d == NULL)
- {
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open configuration directory \"%s\": %m",
- directory)));
- record_config_file_error(psprintf("could not open directory \"%s\"",
- directory),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
- /*
- * Read the directory and put the filenames in an array, so we can sort
- * them prior to processing the contents.
- */
- size_filenames = 32;
- filenames = (char **) palloc(size_filenames * sizeof(char *));
- num_filenames = 0;
+ filenames = GetDirConfFiles(includedir, calling_file, elevel,
+ &num_filenames, &err_msg);
- while ((de = ReadDir(d, directory)) != NULL)
+ if (!filenames)
{
- PGFileType de_type;
- char filename[MAXPGPATH];
-
- /*
- * Only parse files with names ending in ".conf". Explicitly reject
- * files starting with ".". This excludes things like "." and "..",
- * as well as typical hidden files, backup files, and editor debris.
- */
- if (strlen(de->d_name) < 6)
- continue;
- if (de->d_name[0] == '.')
- continue;
- if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
- continue;
-
- join_path_components(filename, directory, de->d_name);
- canonicalize_path(filename);
- de_type = get_dirent_type(filename, de, true, elevel);
- if (de_type == PGFILETYPE_ERROR)
- {
- record_config_file_error(psprintf("could not stat file \"%s\"",
- filename),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
- else if (de_type != PGFILETYPE_DIR)
- {
- /* Add file to array, increasing its size in blocks of 32 */
- if (num_filenames >= size_filenames)
- {
- size_filenames += 32;
- filenames = (char **) repalloc(filenames,
- size_filenames * sizeof(char *));
- }
- filenames[num_filenames] = pstrdup(filename);
- num_filenames++;
- }
+ record_config_file_error(err_msg, calling_file, calling_lineno, head_p,
+ tail_p);
+ return false;
}
- if (num_filenames > 0)
+ for (int i = 0; i < num_filenames; i++)
{
- int i;
-
- qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp);
- for (i = 0; i < num_filenames; i++)
- {
- if (!ParseConfigFile(filenames[i], true,
- calling_file, calling_lineno,
- depth, elevel,
- head_p, tail_p))
- {
- status = false;
- goto cleanup;
- }
- }
+ if (!ParseConfigFile(filenames[i], true,
+ calling_file, calling_lineno,
+ depth, elevel,
+ head_p, tail_p))
+ return false;
}
- status = true;
-cleanup:
- if (d)
- FreeDir(d);
- pfree(directory);
- return status;
+ return true;
}
/*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index e10d5124ab..807fe3890b 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6131,16 +6131,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '6250', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,text,text,text}',
- proargmodes => '{o,o,o,o,o,o}',
- proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index d06da81806..b45b5cff38 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -155,6 +156,7 @@ typedef struct AuthToken
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of AuthTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
@@ -174,6 +176,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel);
extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel);
extern bool pg_isblank(const char c);
extern MemoryContext tokenize_auth_file(const char *filename, FILE *file,
- List **tok_lines, int elevel);
+ List **tok_lines, int depth,
+ int elevel);
#endif /* HBA_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 1788361974..b465dc0356 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -144,6 +144,8 @@ typedef struct ConfigVariable
struct ConfigVariable *next;
} ConfigVariable;
+extern char **GetDirConfFiles(const char *includedir, const char *calling_file,
+ int elevel, int *num_filenames, char **err_msg);
extern bool ParseConfigFile(const char *config_file, bool strict,
const char *calling_file, int calling_lineno,
int depth, int elevel,
diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl
new file mode 100644
index 0000000000..8eae72b8d4
--- /dev/null
+++ b/src/test/authentication/t/003_file_inclusion.pl
@@ -0,0 +1,657 @@
+
+# Copyright (c) 2021-2022, PostgreSQL Global Development Group
+
+# Set of tests for authentication and pg_hba.conf inclusion.
+# This test can only run with Unix-domain sockets.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Time::HiRes qw(usleep);
+use IPC::Run qw(pump finish timer);
+use Data::Dumper;
+
+if (!$use_unix_sockets)
+{
+ plan skip_all =>
+ "authentication tests cannot run without Unix-domain sockets";
+}
+
+# stores the current line counter for each file. hba_rule and ident_rule are
+# fake file names used for the global rule number for each auth view.
+my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+my $hba_file = 'subdir1/pg_hba_custom.conf';
+my $ident_file = 'subdir2/pg_ident_custom.conf';
+
+# Initialize primary node
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+$node->start;
+
+my $data_dir = $node->data_dir;
+
+# Normalize the data directory for Windows
+$data_dir =~ s/\/\.\//\//g; # reduce /./ to /
+$data_dir =~ s/\/\//\//g; # reduce // to /
+$data_dir =~ s/\/$//; # remove trailing /
+
+
+# Add the given payload to the given relative HBA file of the given node.
+# This function maintains the %cur_line metadata, so it has to be called in the
+# expected inclusion evaluation order in order to keep it in sync.
+#
+# If the payload starts with "include" or "ignore", the function doesn't
+# increase the general hba rule number.
+#
+# If an err_str is provided, it returns an arrayref containing the provided
+# filename, the current line number in that file and the provided err_str. The
+# err_str has to be a valid regex string.
+# Otherwise it only returns the line number of the payload in the wanted file.
+# This function has to be called in the expected inclusion evaluation order to
+# keep the %cur_line information in sync.
+sub add_hba_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'hba_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_hba_file_rules line
+ @tokens = split(/ /, $payload);
+ $tokens[1] = '{' . $tokens[1] . '}'; # database
+ $tokens[2] = '{' . $tokens[2] . '}'; # user_name
+
+ # add empty address and netmask betweed user_name and auth_method
+ splice @tokens, 3, 0, '';
+ splice @tokens, 3, 0, '';
+
+ # append empty options and error
+ push @tokens, '';
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Add the given payload to the given relative ident file of the given node.
+# Same as add_hba_line but for pg_ident files
+sub add_ident_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'ident_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_ident_file_mappings line
+ @tokens = split(/ /, $payload);
+
+ # append empty error
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Delete pg_hba.conf from the given node, add various entries to test the
+# include infrastructure and then execute a reload to refresh it.
+sub generate_valid_auth_files
+{
+ my $node = shift;
+ my $hba_expected = '';
+ my $ident_expected = '';
+
+ # customise main auth file names
+ $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'");
+ $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'");
+
+ # and make original ones invalid to be sure they're not used anywhere
+ $node->append_conf('pg_hba.conf', "some invalid line");
+ $node->append_conf('pg_ident.conf', "some invalid line");
+
+ # pg_hba stuff
+ mkdir("$data_dir/subdir1");
+ mkdir("$data_dir/hba_inc");
+ mkdir("$data_dir/hba_inc_if");
+ mkdir("$data_dir/hba_pos");
+
+ # Make sure we will still be able to connect
+ $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust');
+
+ # Add include data
+ add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf");
+ $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject");
+
+ $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject");
+
+ add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject");
+ # include is relative to current path
+ add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject");
+
+ # include_if_exists data
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none");
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some");
+ $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject");
+
+ # include_dir data
+ add_hba_line($node, "$hba_file", "include_dir ../hba_inc");
+ add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included");
+ $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject");
+ $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject");
+
+ # secondary auth file
+ add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject');
+ $node->append_conf('dbnames.conf', "db1");
+ $node->append_conf('dbnames.conf', "db3");
+ $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1)
+ . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1)
+ . '|local|{db1,db3}|{all}|||reject||';
+
+ # pg_ident stuff
+ mkdir("$data_dir/subdir2");
+ mkdir("$data_dir/ident_inc");
+ mkdir("$data_dir/ident_inc_if");
+ mkdir("$data_dir/ident_pos");
+
+ # Add include data
+ add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf");
+ $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar");
+
+ $ident_expected .= add_ident_line($node, "$ident_file", "test a b");
+
+ add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar");
+ # include is relative to current path
+ add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar");
+
+ # include_if_exists data
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none");
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some");
+ $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar");
+
+ # include_dir data
+ add_ident_line($node, "$ident_file", "include_dir ../ident_inc");
+ add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included");
+ $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar");
+ $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar");
+
+ $node->restart;
+ $node->connect_ok('dbname=postgres',
+ 'Connection ok after generating valid auth files');
+
+ return ($hba_expected, $ident_expected);
+}
+
+# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal
+# entries to allow authentication.
+sub reset_auth_files
+{
+ my $node = shift;
+
+ unlink("$data_dir/$hba_file");
+ unlink("$data_dir/$ident_file");
+
+ %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+ return add_hba_line($node, "$hba_file", 'local all all trust');
+}
+
+# Generate a list of expected error regex for the given array of error
+# conditions, as generated by add_hba_line/add_ident_line with an err_str.
+#
+# 2 regex are generated per array entry: one for the given err_str, and one for
+# the expected line in the specific file. Since all lines are independant,
+# there's no guarantee that a specific failure regex and the per-line regex
+# will match the same error. Calling code should add at least one test with a
+# single error to make sure that the line number / file name is correct.
+#
+# On top of that, an extra line is generated for the general failure to process
+# the main auth file.
+sub generate_log_err_patterns
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $is_hba_err = shift;
+ my @errors;
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ push @errors, qr/$err_str/;
+
+ # Context messages with the file / line location aren't always emitted
+ if ($err_str !~ /maximum nesting depth exceeded/ and
+ $err_str !~ /could not open secondary authentication file/)
+ {
+ push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/
+ }
+ }
+
+ push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err);
+
+ return \@errors;
+}
+
+# Generate the expected output for the auth file view error reporting (file
+# name, file line, error), for the given array of error conditions, as
+# generated generated by add_hba_line/add_ident_line with an err_str.
+sub generate_log_err_rows
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $exp_rows = '';
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ $exp_rows .= "\n" if ($exp_rows ne "");
+
+ # Unescape regex patterns if any
+ $err_str =~ s/\\([\(\)])/$1/g;
+ $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str"
+ }
+
+ return $exp_rows;
+}
+
+# Reset the main auth files, append the given payload to the given config file,
+# and check that the instance cannot start, raising the expected error line(s).
+sub start_errors_like
+{
+ my $node = shift;
+ my $file = shift;
+ my $payload = shift;
+ my $pattern = shift;
+ my $should_fail = shift;
+
+ reset_auth_files($node);
+ $node->append_conf($file, $payload);
+
+ unlink($node->logfile);
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir,
+ '-l', $node->logfile, 'start');
+
+ if ($should_fail)
+ {
+ ok($ret != 0, "Cannot start postgres with faulty $file");
+ }
+ else
+ {
+ ok($ret == 0, "postgres can start with faulty $file");
+ }
+
+ my $log_contents = slurp_file($node->logfile);
+
+ foreach (@{$pattern})
+ {
+ like($log_contents,
+ $_,
+ "Expected failure found in the logs");
+ }
+
+ if (not $should_fail)
+ {
+ # We can't simply call $node->stop here as the call is optimized out
+ # when the server isn't started with $node->start.
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D',
+ $data_dir, 'stop', '-m', 'fast');
+ ok($ret == 0, "Could stop postgres");
+ }
+}
+
+# We should be able to connect, and see an empty pg_ident.conf
+is($node->psql(
+ 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'),
+ qq(0),
+ 'pg_ident.conf is empty');
+
+############################################
+# part 1, test view reporting for valid data
+############################################
+my ($exp_hba, $exp_ident) = generate_valid_auth_files($node);
+
+$node->connect_ok('dbname=postgres', 'Connection still ok');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_hba_file_rules'),
+ qq($exp_hba),
+ 'pg_hba_file_rules content is expected');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_ident_file_mappings'),
+ qq($exp_ident),
+ 'pg_ident_file_mappings content is expected');
+
+#############################################
+# part 2, test log reporting for invalid data
+#############################################
+reset_auth_files($node);
+$node->restart('fast');
+$node->connect_ok('dbname=postgres',
+ 'Connection ok after resetting auth files');
+
+$node->stop('fast');
+
+start_errors_like($node, $hba_file, "include ../not_a_file",
+ [
+ qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file
+mkdir("$data_dir/hba_inc_fail");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token");
+start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail",
+ [
+ qr/invalid connection type "not_a_token"/,
+ qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file with nested inclusion
+unlink("$data_dir/hba_inc_fail/inc_dir.conf");
+my @hba_raw_errors_step1;
+
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1");
+
+add_hba_line($node, "hba_inc_fail/file1", "include file2");
+add_hba_line($node, "hba_inc_fail/file2", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file2", "include file3");
+
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3",
+ "local all all zuul",
+ 'invalid authentication method "zuul"');
+
+start_errors_like(
+ $node, $hba_file, "include_dir ../hba_inc_fail",
+ generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @hba_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local",
+ "end-of-line before database specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local,host",
+ "multiple values specified for connection type");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all",
+ "end-of-line before role specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all all",
+ "end-of-line before authentication method");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "host all all test/42",
+ 'specifying both host name and CIDR mask is invalid: "test/42"');
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ 'local @dbnames_fails.conf all reject',
+ "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory");
+
+add_hba_line($node, "hba_if_exists.conf", "include recurse.conf");
+push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf",
+ "include recurse.conf",
+ 'could not open configuration file "recurse.conf": maximum nesting depth exceeded');
+
+# Generate the regex for the expected errors in the logs. There's no guarantee
+# that the generated "line X of file..." will be emitted for the expected line,
+# but previous tests already ensured that the correct line number / file name
+# was emitted, so ensuring that there's an error in all expected lines is
+# enough here.
+my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2,
+ 1);
+
+# Not an error, but it should raise a message in the logs. Manually add an
+# extra log message to detect
+add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none");
+push @{$expected_errors},
+ qr/skipping missing authentication file "$data_dir\/if_exists_none"/;
+
+start_errors_like(
+ $node, $hba_file, "include_if_exists ../hba_if_exists.conf",
+ $expected_errors, 1);
+
+# Mostly the same, but for ident files
+reset_auth_files($node);
+
+my @ident_raw_errors_step1;
+
+# include_dir, single included file with nested inclusion
+mkdir("$data_dir/ident_inc_fail");
+add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1");
+
+add_ident_line($node, "ident_inc_fail/file1", "include file2");
+add_ident_line($node, "ident_inc_fail/file2", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file2", "include file3");
+
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3",
+ "failmap /(fail postgres",
+ 'invalid regular expression "\(fail": parentheses \(\) not balanced');
+
+start_errors_like(
+ $node, $ident_file, "include_dir ../ident_inc_fail",
+ generate_log_err_patterns($node, \@ident_raw_errors_step1, 0),
+ 0);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @ident_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map",
+ "missing entry at end of line");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2",
+ "multiple values in ident field");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf",
+ 'map @osnames_fails.conf postgres',
+ "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory");
+
+add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf",
+ 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded');
+
+start_errors_like(
+ $node, $ident_file, "include_if_exists ../ident_if_exists.conf",
+ # There's no guarantee that the generated "line X of file..." will be
+ # emitted for the expected line, but previous tests already ensured that
+ # the correct line number / file name was emitted, so ensuring that there's
+ # an error in all expected lines is enough here.
+ generate_log_err_patterns($node, \@ident_raw_errors_step2, 0),
+ 0);
+
+#####################################################
+# part 3, test reporting of various error scenario
+# NOTE: this will be bypassed -DEXEC_BACKEND or win32
+#####################################################
+reset_auth_files($node);
+
+$node->start;
+$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset');
+
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_hba_file_rules');
+
+add_ident_line($node, $ident_file, '');
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_ident_file_mappings');
+
+# The instance could be restarted and no error is detected. Now check if the
+# build is compatible with the view error reporting (EXEC_BACKEND / win32 will
+# fail when trying to connect as they always rely on the current auth files
+# content)
+my @hba_raw_errors;
+
+push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+my ($stdout, $stderr);
+my $cmdret = $node->psql('postgres', 'SELECT 1',
+ stdout => \$stdout, stderr => \$stderr);
+
+if ($cmdret != 0)
+{
+ # Connection failed. Bail out, but make sure to raise a failure if it
+ # didn't fail for the expected hba file modification.
+ like($stderr,
+ qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/,
+ "Connection failed due to loading an invalid hba file");
+
+ done_testing();
+ diag("Build not compatible with auth file view error reporting, bail out.\n");
+ exit;
+}
+
+# Combine errors generated at step 2, in the same order.
+$node->append_conf($hba_file, "include_dir ../hba_inc_fail");
+push @hba_raw_errors, @hba_raw_errors_step1;
+
+$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf");
+push @hba_raw_errors, @hba_raw_errors_step2;
+
+my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules'
+ . ' WHERE error IS NOT NULL ORDER BY rule_number'),
+ qq($hba_expected),
+ 'Detected all error in hba file');
+
+# and do the same for pg_ident
+my @ident_raw_errors;
+
+push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+$node->append_conf($ident_file, "include_dir ../ident_inc_fail");
+push @ident_raw_errors, @ident_raw_errors_step1;
+
+$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf");
+push @ident_raw_errors, @ident_raw_errors_step2;
+
+my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings'
+ . ' WHERE error IS NOT NULL ORDER BY mapping_number'),
+ qq($ident_expected),
+ 'Detected all error in ident file');
+
+done_testing();
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 6e29a6acc8..ef17bf1e8a 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname,
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
a.line_number,
a.type,
a.database,
@@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
a.line_number,
a.map_name,
a.sys_name,
a.pg_username,
a.error
- FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.37.0
--z5mbrmnmd43fw5pt
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v11-0003-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v8 5/6] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-05-30 03:15 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-05-30 03:15 UTC (permalink / raw)
pg_hba.conf file now has support for "include", "include_dir" and
"include_if_exists" directives, which work similarly to the same directives in
the postgresql.conf file.
This fixes a possible crash if a secondary file tries to include itself as
there's now a nesting depth check in the inclusion code path, same as the
postgresql.conf.
Many regression tests added to cover both the new directives, but also error
detection for the whole pg_hba / pg_ident files.
Catversion is bumped.
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/client-auth.sgml | 86 ++-
doc/src/sgml/system-views.sgml | 22 +-
src/backend/libpq/hba.c | 485 ++++++++++---
src/backend/libpq/pg_hba.conf.sample | 25 +-
src/backend/libpq/pg_ident.conf.sample | 15 +-
src/backend/utils/adt/hbafuncs.c | 43 +-
src/backend/utils/misc/guc-file.l | 250 +++----
src/include/catalog/pg_proc.dat | 12 +-
src/include/libpq/hba.h | 5 +-
src/include/utils/guc.h | 2 +
.../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++
src/test/regress/expected/rules.out | 6 +-
12 files changed, 1325 insertions(+), 283 deletions(-)
create mode 100644 src/test/authentication/t/003_file_inclusion.pl
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index 433759928b..e4eacab4a5 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,23 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication
+ record. Inclusion directives specify files that can be included, which
+ contains additional records. The records will be inserted in lieu of the
+ inclusion records. Those records only contains two fields: the
+ <literal>include</literal>, <literal>include_if_exists</literal> or
+ <literal>include_dir</literal> directive and the file or directory to be
+ included. The file or directory can be a relative of absolute path, and can
+ be double quoted if needed. For the <literal>include_dir</literal> form,
+ all files not starting with a <literal>.</literal> and ending with
+ <literal>.conf</literal> will be included. Multiple files within an include
+ directory are processed in file name order (according to C locale rules,
+ i.e., numbers before letters, and uppercase letters before lowercase ones).
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,21 +118,57 @@
<para>
A record can have several formats:
<synopsis>
-local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+include <replaceable>file</replaceable>
+include_if_exists <replaceable>file</replaceable>
+include_dir <replaceable>directory</replaceable>
+local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
</synopsis>
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_if_exists</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file if the
+ file exists and can be read. Otherwise, a message will be logged to
+ indicate that the file is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_dir</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of all the files found in
+ the directory, if they don't start with a <literal>.</literal> and end
+ with <literal>.conf</literal>, processed in file name order (according
+ to C locale rules, i.e., numbers before letters, and uppercase letters
+ before lowercase ones).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -837,8 +888,10 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
+<replaceable>include_dir</replaceable> <replaceable>directory</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -849,6 +902,11 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ As for <filename>pg_hba.conf</filename>, the lines in this file can either
+ be inclusion directives or user name map records, and follow the same
+ rules.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index b4ee40c042..35ed69a64a 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -1003,12 +1003,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule the given <literal>file_name</literal>
</para></entry>
</row>
@@ -1153,12 +1162,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this mapping
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_ident.conf</filename>
+ Line number of this mapping in the given <literal>file_name</literal>
</para></entry>
</row>
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index deee05c197..814bfcf30d 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -21,6 +21,7 @@
#include <fcntl.h>
#include <sys/param.h>
#include <sys/socket.h>
+#include <sys/stat.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
@@ -68,6 +69,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -112,10 +119,22 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int depth,
+ int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
- const char *inc_filename, int elevel, char **err_msg);
+ const char *inc_filename, int depth, int elevel,
+ char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ bool strict, const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
+static char *process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth,
+ int elevel, MemoryContext linecxt,
+ List **tok_lines);
/*
@@ -302,7 +321,7 @@ copy_auth_token(AuthToken *in)
*/
static List *
next_field_expand(const char *filename, char **lineptr,
- int elevel, char **err_msg)
+ int depth, int elevel, char **err_msg)
{
char buf[MAX_TOKEN];
bool trailing_comma;
@@ -318,7 +337,7 @@ next_field_expand(const char *filename, char **lineptr,
/* Is this referencing a file? */
if (!initial_quote && buf[0] == '@' && buf[1] != '\0')
- tokens = tokenize_inc_file(tokens, filename, buf + 1,
+ tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1,
elevel, err_msg);
else
tokens = lappend(tokens, make_auth_token(buf, initial_quote));
@@ -346,6 +365,7 @@ static List *
tokenize_inc_file(List *tokens,
const char *outer_filename,
const char *inc_filename,
+ int depth,
int elevel,
char **err_msg)
{
@@ -355,39 +375,30 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
{
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
+ *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", *err_msg)));
+ return tokens;
}
- inc_file = AllocateFile(inc_fullname, "r");
- if (inc_file == NULL)
- {
- int save_errno = errno;
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true,
+ outer_filename, elevel, err_msg, &inc_fullname);
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
+ if (inc_file == NULL)
return tokens;
- }
/* There is possible recursion here if the file contains @ */
- linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
+ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1,
+ elevel);
FreeFile(inc_file);
pfree(inc_fullname);
@@ -425,11 +436,38 @@ tokenize_inc_file(List *tokens,
/*
* tokenize_auth_file
- * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a dedicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
+ int depth, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, depth,
+ elevel);
+
+ return linecxt;
+}
+
+/*
+ * Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see the struct definition
* in libpq/hba.h.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -438,30 +476,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int depth, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -514,7 +544,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
{
List *current_field;
- current_field = next_field_expand(filename, &lineptr,
+ current_field = next_field_expand(filename, &lineptr, depth,
elevel, &err_msg);
/* add field to line, unless we are at EOL or comment start */
if (current_field != NIL)
@@ -522,29 +552,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ AuthToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
+
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
+
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, true,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
+ else if (strcmp(first->string, "include_dir") == 0)
+ {
+ char **filenames;
+ char *dir_name = second->string;
+ int num_filenames;
+ StringInfoData err_buf;
+
+ filenames = GetDirConfFiles(dir_name, filename, elevel,
+ &num_filenames, &err_msg);
+
+ if (!filenames)
+ {
+ /* We have the error in err_msg, simply process it */
+ goto process_line;
+ }
+
+ initStringInfo(&err_buf);
+ for (int i = 0; i < num_filenames; i++)
+ {
+ /*
+ * err_msg is used here as a temp buffer, it will be
+ * overwritten at the end of the loop with the
+ * cumulated errors, if any.
+ */
+ err_msg = process_included_authfile(filenames[i], true,
+ filename, depth + 1, elevel,
+ linecxt, tok_lines);
+
+ /* Cumulate errors if any. */
+ if (err_msg)
+ {
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfoString(&err_buf, err_msg);
+ }
+ }
+
+ /*
+ * If there were no errors, the line is fully processed, bypass
+ * the general TokenizedAuthLine processing.
+ */
+ if (err_buf.len == 0)
+ goto next_line;
+
+ /* Otherwise, process the cumulated errors, if any. */
+ err_msg = err_buf.data;
+ }
+ else if (strcmp(first->string, "include_if_exists") == 0)
+ {
+ char *inc_filename;
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, false,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
}
+process_line:
+ /*
+ * General processing: report the error if any and emit line to the
+ * TokenizedAuthLine
+ */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -859,7 +987,7 @@ do { \
errmsg("authentication option \"%s\" is only valid for authentication methods %s", \
optname, _(validmethods)), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \
optname, validmethods); \
return false; \
@@ -879,7 +1007,7 @@ do { \
errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, HbaFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \
authname, argname); \
return NULL; \
@@ -902,7 +1030,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("missing entry at end of line"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("missing entry at end of line"); \
return NULL; \
} \
@@ -915,7 +1043,7 @@ do { \
(errcode(ERRCODE_CONFIG_FILE_ERROR), \
errmsg("multiple values in ident field"), \
errcontext("line %d of configuration file \"%s\"", \
- line_num, IdentFileName))); \
+ line_num, file_name))); \
*err_msg = psprintf("multiple values in ident field"); \
return NULL; \
} \
@@ -938,6 +1066,7 @@ HbaLine *
parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
{
int line_num = tok_line->line_num;
+ char *file_name = tok_line->file_name;
char **err_msg = &tok_line->err_msg;
char *str;
struct addrinfo *gai_result;
@@ -952,6 +1081,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
HbaLine *parsedline;
parsedline = palloc0(sizeof(HbaLine));
+ parsedline->sourcefile = pstrdup(file_name);
parsedline->linenumber = line_num;
parsedline->rawline = pstrdup(tok_line->raw_line);
@@ -966,7 +1096,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for connection type"),
errhint("Specify exactly one connection type per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for connection type";
return NULL;
}
@@ -980,7 +1110,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("local connections are not supported by this build"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "local connections are not supported by this build";
return NULL;
#endif
@@ -1004,7 +1134,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("hostssl record cannot match because SSL is disabled"),
errhint("Set ssl = on in postgresql.conf."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostssl record cannot match because SSL is disabled";
}
#else
@@ -1012,7 +1142,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("hostssl record cannot match because SSL is not supported by this build"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostssl record cannot match because SSL is not supported by this build";
#endif
}
@@ -1024,7 +1154,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("hostgssenc record cannot match because GSSAPI is not supported by this build"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "hostgssenc record cannot match because GSSAPI is not supported by this build";
#endif
}
@@ -1045,7 +1175,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid connection type \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid connection type \"%s\"", token->string);
return NULL;
}
@@ -1058,7 +1188,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before database specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before database specification";
return NULL;
}
@@ -1078,7 +1208,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before role specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before role specification";
return NULL;
}
@@ -1100,7 +1230,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before IP address specification"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before IP address specification";
return NULL;
}
@@ -1112,7 +1242,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for host address"),
errhint("Specify one address range per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for host address";
return NULL;
}
@@ -1171,7 +1301,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid IP address \"%s\": %s",
str, gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid IP address \"%s\": %s",
str, gai_strerror(ret));
if (gai_result)
@@ -1191,7 +1321,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("specifying both host name and CIDR mask is invalid: \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("specifying both host name and CIDR mask is invalid: \"%s\"",
token->string);
return NULL;
@@ -1205,7 +1335,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid CIDR mask in address \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid CIDR mask in address \"%s\"",
token->string);
return NULL;
@@ -1225,7 +1355,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("end-of-line before netmask specification"),
errhint("Specify an address range in CIDR notation, or provide a separate netmask."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before netmask specification";
return NULL;
}
@@ -1236,7 +1366,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("multiple values specified for netmask"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for netmask";
return NULL;
}
@@ -1251,7 +1381,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid IP mask \"%s\": %s",
token->string, gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid IP mask \"%s\": %s",
token->string, gai_strerror(ret));
if (gai_result)
@@ -1270,7 +1400,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("IP address and mask do not match"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "IP address and mask do not match";
return NULL;
}
@@ -1286,7 +1416,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("end-of-line before authentication method"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "end-of-line before authentication method";
return NULL;
}
@@ -1298,7 +1428,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("multiple values specified for authentication type"),
errhint("Specify exactly one authentication type per line."),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "multiple values specified for authentication type";
return NULL;
}
@@ -1335,7 +1465,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("MD5 authentication is not supported when \"db_user_namespace\" is enabled"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "MD5 authentication is not supported when \"db_user_namespace\" is enabled";
return NULL;
}
@@ -1376,7 +1506,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid authentication method \"%s\"",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid authentication method \"%s\"",
token->string);
return NULL;
@@ -1389,7 +1519,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid authentication method \"%s\": not supported by this build",
token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid authentication method \"%s\": not supported by this build",
token->string);
return NULL;
@@ -1411,7 +1541,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("gssapi authentication is not supported on local sockets"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "gssapi authentication is not supported on local sockets";
return NULL;
}
@@ -1423,7 +1553,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("peer authentication is only supported on local sockets"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "peer authentication is only supported on local sockets";
return NULL;
}
@@ -1441,7 +1571,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cert authentication is only supported on hostssl connections"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cert authentication is only supported on hostssl connections";
return NULL;
}
@@ -1491,7 +1621,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("authentication option not in name=value format: %s", token->string),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("authentication option not in name=value format: %s",
token->string);
return NULL;
@@ -1535,7 +1665,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix";
return NULL;
}
@@ -1546,7 +1676,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set";
return NULL;
}
@@ -1562,7 +1692,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("cannot use ldapsearchattribute together with ldapsearchfilter"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "cannot use ldapsearchattribute together with ldapsearchfilter";
return NULL;
}
@@ -1579,7 +1709,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("list of RADIUS servers cannot be empty"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "list of RADIUS servers cannot be empty";
return NULL;
}
@@ -1590,7 +1720,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("list of RADIUS secrets cannot be empty"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "list of RADIUS secrets cannot be empty";
return NULL;
}
@@ -1609,7 +1739,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiussecrets),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS secrets (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiussecrets),
list_length(parsedline->radiusservers));
@@ -1625,7 +1755,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiusports),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS ports (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiusports),
list_length(parsedline->radiusservers));
@@ -1641,7 +1771,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel)
list_length(parsedline->radiusidentifiers),
list_length(parsedline->radiusservers)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("the number of RADIUS identifiers (%d) must be 1 or the same as the number of RADIUS servers (%d)",
list_length(parsedline->radiusidentifiers),
list_length(parsedline->radiusservers));
@@ -1676,6 +1806,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg)
{
int line_num = hbaline->linenumber;
+ char *file_name = hbaline->sourcefile;
#ifdef USE_LDAP
hbaline->ldapscope = LDAP_SCOPE_SUBTREE;
@@ -1699,7 +1830,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientcert can only be configured for \"hostssl\" rows"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientcert can only be configured for \"hostssl\" rows";
return false;
}
@@ -1716,7 +1847,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientcert only accepts \"verify-full\" when using \"cert\" authentication"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientcert can only be set to \"verify-full\" when using \"cert\" authentication";
return false;
}
@@ -1729,7 +1860,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid value for clientcert: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
}
@@ -1741,7 +1872,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("clientname can only be configured for \"hostssl\" rows"),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = "clientname can only be configured for \"hostssl\" rows";
return false;
}
@@ -1760,7 +1891,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid value for clientname: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
}
@@ -1846,7 +1977,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid ldapscheme value: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
hbaline->ldapscheme = pstrdup(val);
}
else if (strcmp(name, "ldapserver") == 0)
@@ -1864,7 +1995,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid LDAP port number: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid LDAP port number: \"%s\"", val);
return false;
}
@@ -1958,7 +2089,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS server list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -1977,7 +2108,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not translate RADIUS server name \"%s\" to address: %s",
(char *) lfirst(l), gai_strerror(ret)),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
if (gai_result)
pg_freeaddrinfo_all(hints.ai_family, gai_result);
@@ -2006,7 +2137,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS port list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid RADIUS port number: \"%s\"", val);
return false;
}
@@ -2019,7 +2150,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid RADIUS port number: \"%s\"", val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2042,7 +2173,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS secret list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2064,7 +2195,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("could not parse RADIUS identifiers list \"%s\"",
val),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
return false;
}
@@ -2078,7 +2209,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
errmsg("unrecognized authentication option name: \"%s\"",
name),
errcontext("line %d of configuration file \"%s\"",
- line_num, HbaFileName)));
+ line_num, file_name)));
*err_msg = psprintf("unrecognized authentication option name: \"%s\"",
name);
return false;
@@ -2226,7 +2357,7 @@ load_hba(void)
return false;
}
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
@@ -2297,6 +2428,137 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ if (strict)
+ {
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ }
+ else
+ {
+ Assert(kind == IncludedAuthFile);
+ ereport(LOG,
+ (errmsg("skipping missing authentication file \"%s\"",
+ *inc_fullname)));
+ }
+
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
+
+/*
+ * Try to open an included file, and tokenize it using the given context.
+ * Returns NULL if no error happens during tokenization, otherwise the error.
+ */
+static char *
+process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth, int elevel,
+ MemoryContext linecxt, List **tok_lines)
+{
+ char *inc_fullname;
+ FILE *inc_file;
+ char *err_msg = NULL;
+
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
+ {
+ char *err_msg;
+
+ err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", err_msg)));
+ return err_msg;
+ }
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict,
+ outer_filename, elevel, &err_msg, &inc_fullname);
+
+ if (inc_file == NULL)
+ {
+ if (strict)
+ {
+ /* open_inc_file should have reported an error. */
+ Assert(err_msg != NULL);
+ return err_msg;
+ }
+ else
+ return NULL;
+ }
+ else
+ {
+ /* No error message should have been reported. */
+ Assert(err_msg == NULL);
+ }
+
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, depth, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ return NULL;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
@@ -2315,6 +2577,7 @@ load_hba(void)
IdentLine *
parse_ident_line(TokenizedAuthLine *tok_line, int elevel)
{
+ char *file_name = tok_line->file_name;
int line_num = tok_line->line_num;
char **err_msg = &tok_line->err_msg;
ListCell *field;
@@ -2375,7 +2638,7 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel)
errmsg("invalid regular expression \"%s\": %s",
parsedline->ident_user + 1, errstr),
errcontext("line %d of configuration file \"%s\"",
- line_num, IdentFileName)));
+ line_num, file_name)));
*err_msg = psprintf("invalid regular expression \"%s\": %s",
parsedline->ident_user + 1, errstr);
@@ -2610,7 +2873,7 @@ load_ident(void)
return false;
}
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..7433050112 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,16 +9,27 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
-# local DATABASE USER METHOD [OPTIONS]
-# host DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# local DATABASE USER METHOD [OPTIONS]
+# host DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..8e3fa29135 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,23 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
-# MAPNAME SYSTEM-USERNAME PG-USERNAME
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index c9be4bff1f..15326a01e2 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -26,12 +26,12 @@
static ArrayType *get_hba_options(HbaLine *hba);
static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int rule_number, int lineno, HbaLine *hba,
- const char *err_msg);
+ int rule_number, const char *filename, int lineno,
+ HbaLine *hba, const char *err_msg);
static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int mapping_number, int lineno, IdentLine *ident,
- const char *err_msg);
+ int mapping_number, const char *filename,
+ int lineno, IdentLine *ident, const char *err_msg);
static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc);
@@ -159,7 +159,7 @@ get_hba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 10
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line
@@ -168,7 +168,8 @@ get_hba_options(HbaLine *hba)
* tuple_store: where to store data
* tupdesc: tuple descriptor for the view
* rule_number: unique rule identifier among all valid rules
- * lineno: pg_hba.conf line number (must always be valid)
+ * filename: name of the file containing that line
+ * lineno: line number in that file (must always be valid)
* hba: parsed line data (can be NULL, in which case err_msg should be set)
* err_msg: error message (NULL if none)
*
@@ -177,7 +178,7 @@ get_hba_options(HbaLine *hba)
*/
static void
fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int rule_number, int lineno, HbaLine *hba,
+ int rule_number, const char *filename, int lineno, HbaLine *hba,
const char *err_msg)
{
Datum values[NUM_PG_HBA_FILE_RULES_ATTS];
@@ -202,6 +203,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -345,7 +348,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -386,7 +389,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open configuration file \"%s\": %m",
HbaFileName)));
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -407,8 +410,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
rule_number++;
- fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->line_num,
- hbaline, tok_line->err_msg);
+ fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name,
+ tok_line->line_num, hbaline, tok_line->err_msg);
}
/* Free tokenizer memory */
@@ -445,7 +448,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
}
/* Number of columns in pg_hba_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -454,7 +457,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
* tuple_store: where to store data
* tupdesc: tuple descriptor for the view
* mapping_number: unique rule identifier among all valid rules
- * lineno: pg_ident.conf line number (must always be valid)
+ * filename: name of the file containing that line
+ * lineno: line number in that file (must always be valid)
* ident: parsed line data (can be NULL, in which case err_msg should be set)
* err_msg: error message (NULL if none)
*
@@ -463,8 +467,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
*/
static void
fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
- int mapping_number, int lineno, IdentLine *ident,
- const char *err_msg)
+ int mapping_number, const char *filename, int lineno,
+ IdentLine *ident, const char *err_msg)
{
Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS];
@@ -482,6 +486,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(filename);
/* line_number */
values[index++] = Int32GetDatum(lineno);
@@ -494,7 +500,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -534,7 +540,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open usermap file \"%s\": %m",
IdentFileName)));
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -556,7 +562,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
mapping_number++;
fill_ident_line(tuple_store, tupdesc, mapping_number,
- tok_line->line_num, identline, tok_line->err_msg);
+ tok_line->file_name, tok_line->line_num, identline,
+ tok_line->err_msg);
}
/* Free tokenizer memory */
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index ce5633844c..b3e18e48cf 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -700,6 +700,122 @@ GUC_flex_fatal(const char *msg)
return 0; /* keep compiler quiet */
}
+/*
+ * Returns the list of config files located in a directory, in alphabetical
+ * order.
+ *
+ * We don't check for recursion or too-deep nesting depth here, its up to the
+ * caller to take care of that.
+ */
+char **
+GetDirConfFiles(const char *includedir, const char *calling_file, int elevel,
+ int *num_filenames, char **err_msg)
+{
+ char *directory;
+ DIR *d;
+ struct dirent *de;
+ char **filenames;
+ int size_filenames;
+
+ /*
+ * Reject directory name that is all-blank (including empty), as that
+ * leads to confusion --- we'd read the containing directory, typically
+ * resulting in recursive inclusion of the same file(s).
+ */
+ if (strspn(includedir, " \t\r\n") == strlen(includedir))
+ {
+ ereport(elevel,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("empty configuration directory name: \"%s\"",
+ includedir)));
+ *err_msg = "empty configuration directory name";
+ return NULL;
+ }
+
+ directory = AbsoluteConfigLocation(includedir, calling_file);
+ d = AllocateDir(directory);
+ if (d == NULL)
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not open configuration directory \"%s\": %m",
+ directory)));
+ *err_msg = psprintf("could not open directory \"%s\"", directory);
+ filenames = NULL;
+ goto cleanup;
+ }
+
+ /*
+ * Read the directory and put the filenames in an array, so we can sort
+ * them prior to caller processing the contents.
+ */
+ size_filenames = 32;
+ filenames = (char **) palloc(size_filenames * sizeof(char *));
+ *num_filenames = 0;
+
+ while ((de = ReadDir(d, directory)) != NULL)
+ {
+ struct stat st;
+ char filename[MAXPGPATH];
+
+ /*
+ * Only parse files with names ending in ".conf". Explicitly reject
+ * files starting with ".". This excludes things like "." and "..",
+ * as well as typical hidden files, backup files, and editor debris.
+ */
+ if (strlen(de->d_name) < 6)
+ continue;
+ if (de->d_name[0] == '.')
+ continue;
+ if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
+ continue;
+
+ join_path_components(filename, directory, de->d_name);
+ canonicalize_path(filename);
+ if (stat(filename, &st) == 0)
+ {
+ /* Ignore directories. */
+ if (S_ISDIR(st.st_mode))
+ continue;
+
+ /* Add file to array, increasing its size in blocks of 32 */
+ if (*num_filenames >= size_filenames)
+ {
+ size_filenames += 32;
+ filenames = (char **) repalloc(filenames,
+ size_filenames * sizeof(char *));
+ }
+ filenames[*num_filenames] = pstrdup(filename);
+ (*num_filenames)++;
+ }
+ else
+ {
+ /*
+ * stat does not care about permissions, so the most likely reason
+ * a file can't be accessed now is if it was removed between the
+ * directory listing and now.
+ */
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not stat file \"%s\": %m",
+ filename)));
+ *err_msg = psprintf("could not stat file \"%s\"", filename);
+ pfree(filenames);
+ filenames = NULL;
+ goto cleanup;
+ }
+ }
+
+ if (*num_filenames > 0)
+ qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp);
+
+cleanup:
+ if (d)
+ FreeDir(d);
+ pfree(directory);
+ return filenames;
+}
+
/*
* Read and parse a single configuration file. This function recurses
* to handle "include" directives.
@@ -961,138 +1077,32 @@ ParseConfigDirectory(const char *includedir,
ConfigVariable **head_p,
ConfigVariable **tail_p)
{
- char *directory;
- DIR *d;
- struct dirent *de;
+ char *err_msg;
char **filenames;
int num_filenames;
- int size_filenames;
- bool status;
- /*
- * Reject directory name that is all-blank (including empty), as that
- * leads to confusion --- we'd read the containing directory, typically
- * resulting in recursive inclusion of the same file(s).
- */
- if (strspn(includedir, " \t\r\n") == strlen(includedir))
- {
- ereport(elevel,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("empty configuration directory name: \"%s\"",
- includedir)));
- record_config_file_error("empty configuration directory name",
- calling_file, calling_lineno,
- head_p, tail_p);
- return false;
- }
+ filenames = GetDirConfFiles(includedir, calling_file, elevel,
+ &num_filenames, &err_msg);
- /*
- * We don't check for recursion or too-deep nesting depth here; the
- * subsequent calls to ParseConfigFile will take care of that.
- */
-
- directory = AbsoluteConfigLocation(includedir, calling_file);
- d = AllocateDir(directory);
- if (d == NULL)
+ if (!filenames)
{
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open configuration directory \"%s\": %m",
- directory)));
- record_config_file_error(psprintf("could not open directory \"%s\"",
- directory),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
-
- /*
- * Read the directory and put the filenames in an array, so we can sort
- * them prior to processing the contents.
- */
- size_filenames = 32;
- filenames = (char **) palloc(size_filenames * sizeof(char *));
- num_filenames = 0;
-
- while ((de = ReadDir(d, directory)) != NULL)
- {
- struct stat st;
- char filename[MAXPGPATH];
-
- /*
- * Only parse files with names ending in ".conf". Explicitly reject
- * files starting with ".". This excludes things like "." and "..",
- * as well as typical hidden files, backup files, and editor debris.
- */
- if (strlen(de->d_name) < 6)
- continue;
- if (de->d_name[0] == '.')
- continue;
- if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
- continue;
-
- join_path_components(filename, directory, de->d_name);
- canonicalize_path(filename);
- if (stat(filename, &st) == 0)
- {
- if (!S_ISDIR(st.st_mode))
- {
- /* Add file to array, increasing its size in blocks of 32 */
- if (num_filenames >= size_filenames)
- {
- size_filenames += 32;
- filenames = (char **) repalloc(filenames,
- size_filenames * sizeof(char *));
- }
- filenames[num_filenames] = pstrdup(filename);
- num_filenames++;
- }
- }
- else
- {
- /*
- * stat does not care about permissions, so the most likely reason
- * a file can't be accessed now is if it was removed between the
- * directory listing and now.
- */
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not stat file \"%s\": %m",
- filename)));
- record_config_file_error(psprintf("could not stat file \"%s\"",
- filename),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
+ record_config_file_error(err_msg, calling_file, calling_lineno, head_p,
+ tail_p);
+ return false;
}
- if (num_filenames > 0)
+ for (int i = 0; i < num_filenames; i++)
{
- int i;
-
- qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp);
- for (i = 0; i < num_filenames; i++)
+ if (!ParseConfigFile(filenames[i], true,
+ calling_file, calling_lineno,
+ depth, elevel,
+ head_p, tail_p))
{
- if (!ParseConfigFile(filenames[i], true,
- calling_file, calling_lineno,
- depth, elevel,
- head_p, tail_p))
- {
- status = false;
- goto cleanup;
- }
+ return false;
}
}
- status = true;
-cleanup:
- if (d)
- FreeDir(d);
- pfree(directory);
- return status;
+ return true;
}
/*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index e544f9f758..d66b2443a4 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6128,16 +6128,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '6250', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,text,text,text}',
- proargmodes => '{o,o,o,o,o,o}',
- proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index 90036f7bcd..0ea100d1b8 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -79,6 +79,7 @@ typedef enum ClientCertName
typedef struct HbaLine
{
+ char *sourcefile;
int linenumber;
char *rawline;
ConnType conntype;
@@ -155,6 +156,7 @@ typedef struct AuthToken
typedef struct TokenizedAuthLine
{
List *fields; /* List of lists of AuthTokens */
+ char *file_name; /* File name */
int line_num; /* Line number */
char *raw_line; /* Raw line text */
char *err_msg; /* Error message if any */
@@ -174,6 +176,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel);
extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel);
extern bool pg_isblank(const char c);
extern MemoryContext tokenize_auth_file(const char *filename, FILE *file,
- List **tok_lines, int elevel);
+ List **tok_lines, int depth,
+ int elevel);
#endif /* HBA_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index e734493a48..1a3ab6306d 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -145,6 +145,8 @@ typedef struct ConfigVariable
struct ConfigVariable *next;
} ConfigVariable;
+extern char **GetDirConfFiles(const char *includedir, const char *calling_file,
+ int elevel, int *num_filenames, char **err_msg);
extern bool ParseConfigFile(const char *config_file, bool strict,
const char *calling_file, int calling_lineno,
int depth, int elevel,
diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl
new file mode 100644
index 0000000000..8eae72b8d4
--- /dev/null
+++ b/src/test/authentication/t/003_file_inclusion.pl
@@ -0,0 +1,657 @@
+
+# Copyright (c) 2021-2022, PostgreSQL Global Development Group
+
+# Set of tests for authentication and pg_hba.conf inclusion.
+# This test can only run with Unix-domain sockets.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Time::HiRes qw(usleep);
+use IPC::Run qw(pump finish timer);
+use Data::Dumper;
+
+if (!$use_unix_sockets)
+{
+ plan skip_all =>
+ "authentication tests cannot run without Unix-domain sockets";
+}
+
+# stores the current line counter for each file. hba_rule and ident_rule are
+# fake file names used for the global rule number for each auth view.
+my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+my $hba_file = 'subdir1/pg_hba_custom.conf';
+my $ident_file = 'subdir2/pg_ident_custom.conf';
+
+# Initialize primary node
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+$node->start;
+
+my $data_dir = $node->data_dir;
+
+# Normalize the data directory for Windows
+$data_dir =~ s/\/\.\//\//g; # reduce /./ to /
+$data_dir =~ s/\/\//\//g; # reduce // to /
+$data_dir =~ s/\/$//; # remove trailing /
+
+
+# Add the given payload to the given relative HBA file of the given node.
+# This function maintains the %cur_line metadata, so it has to be called in the
+# expected inclusion evaluation order in order to keep it in sync.
+#
+# If the payload starts with "include" or "ignore", the function doesn't
+# increase the general hba rule number.
+#
+# If an err_str is provided, it returns an arrayref containing the provided
+# filename, the current line number in that file and the provided err_str. The
+# err_str has to be a valid regex string.
+# Otherwise it only returns the line number of the payload in the wanted file.
+# This function has to be called in the expected inclusion evaluation order to
+# keep the %cur_line information in sync.
+sub add_hba_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'hba_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_hba_file_rules line
+ @tokens = split(/ /, $payload);
+ $tokens[1] = '{' . $tokens[1] . '}'; # database
+ $tokens[2] = '{' . $tokens[2] . '}'; # user_name
+
+ # add empty address and netmask betweed user_name and auth_method
+ splice @tokens, 3, 0, '';
+ splice @tokens, 3, 0, '';
+
+ # append empty options and error
+ push @tokens, '';
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Add the given payload to the given relative ident file of the given node.
+# Same as add_hba_line but for pg_ident files
+sub add_ident_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'ident_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_ident_file_mappings line
+ @tokens = split(/ /, $payload);
+
+ # append empty error
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Delete pg_hba.conf from the given node, add various entries to test the
+# include infrastructure and then execute a reload to refresh it.
+sub generate_valid_auth_files
+{
+ my $node = shift;
+ my $hba_expected = '';
+ my $ident_expected = '';
+
+ # customise main auth file names
+ $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'");
+ $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'");
+
+ # and make original ones invalid to be sure they're not used anywhere
+ $node->append_conf('pg_hba.conf', "some invalid line");
+ $node->append_conf('pg_ident.conf', "some invalid line");
+
+ # pg_hba stuff
+ mkdir("$data_dir/subdir1");
+ mkdir("$data_dir/hba_inc");
+ mkdir("$data_dir/hba_inc_if");
+ mkdir("$data_dir/hba_pos");
+
+ # Make sure we will still be able to connect
+ $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust');
+
+ # Add include data
+ add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf");
+ $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject");
+
+ $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject");
+
+ add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject");
+ # include is relative to current path
+ add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject");
+
+ # include_if_exists data
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none");
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some");
+ $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject");
+
+ # include_dir data
+ add_hba_line($node, "$hba_file", "include_dir ../hba_inc");
+ add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included");
+ $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject");
+ $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject");
+
+ # secondary auth file
+ add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject');
+ $node->append_conf('dbnames.conf', "db1");
+ $node->append_conf('dbnames.conf', "db3");
+ $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1)
+ . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1)
+ . '|local|{db1,db3}|{all}|||reject||';
+
+ # pg_ident stuff
+ mkdir("$data_dir/subdir2");
+ mkdir("$data_dir/ident_inc");
+ mkdir("$data_dir/ident_inc_if");
+ mkdir("$data_dir/ident_pos");
+
+ # Add include data
+ add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf");
+ $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar");
+
+ $ident_expected .= add_ident_line($node, "$ident_file", "test a b");
+
+ add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar");
+ # include is relative to current path
+ add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar");
+
+ # include_if_exists data
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none");
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some");
+ $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar");
+
+ # include_dir data
+ add_ident_line($node, "$ident_file", "include_dir ../ident_inc");
+ add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included");
+ $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar");
+ $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar");
+
+ $node->restart;
+ $node->connect_ok('dbname=postgres',
+ 'Connection ok after generating valid auth files');
+
+ return ($hba_expected, $ident_expected);
+}
+
+# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal
+# entries to allow authentication.
+sub reset_auth_files
+{
+ my $node = shift;
+
+ unlink("$data_dir/$hba_file");
+ unlink("$data_dir/$ident_file");
+
+ %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+ return add_hba_line($node, "$hba_file", 'local all all trust');
+}
+
+# Generate a list of expected error regex for the given array of error
+# conditions, as generated by add_hba_line/add_ident_line with an err_str.
+#
+# 2 regex are generated per array entry: one for the given err_str, and one for
+# the expected line in the specific file. Since all lines are independant,
+# there's no guarantee that a specific failure regex and the per-line regex
+# will match the same error. Calling code should add at least one test with a
+# single error to make sure that the line number / file name is correct.
+#
+# On top of that, an extra line is generated for the general failure to process
+# the main auth file.
+sub generate_log_err_patterns
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $is_hba_err = shift;
+ my @errors;
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ push @errors, qr/$err_str/;
+
+ # Context messages with the file / line location aren't always emitted
+ if ($err_str !~ /maximum nesting depth exceeded/ and
+ $err_str !~ /could not open secondary authentication file/)
+ {
+ push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/
+ }
+ }
+
+ push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err);
+
+ return \@errors;
+}
+
+# Generate the expected output for the auth file view error reporting (file
+# name, file line, error), for the given array of error conditions, as
+# generated generated by add_hba_line/add_ident_line with an err_str.
+sub generate_log_err_rows
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $exp_rows = '';
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ $exp_rows .= "\n" if ($exp_rows ne "");
+
+ # Unescape regex patterns if any
+ $err_str =~ s/\\([\(\)])/$1/g;
+ $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str"
+ }
+
+ return $exp_rows;
+}
+
+# Reset the main auth files, append the given payload to the given config file,
+# and check that the instance cannot start, raising the expected error line(s).
+sub start_errors_like
+{
+ my $node = shift;
+ my $file = shift;
+ my $payload = shift;
+ my $pattern = shift;
+ my $should_fail = shift;
+
+ reset_auth_files($node);
+ $node->append_conf($file, $payload);
+
+ unlink($node->logfile);
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir,
+ '-l', $node->logfile, 'start');
+
+ if ($should_fail)
+ {
+ ok($ret != 0, "Cannot start postgres with faulty $file");
+ }
+ else
+ {
+ ok($ret == 0, "postgres can start with faulty $file");
+ }
+
+ my $log_contents = slurp_file($node->logfile);
+
+ foreach (@{$pattern})
+ {
+ like($log_contents,
+ $_,
+ "Expected failure found in the logs");
+ }
+
+ if (not $should_fail)
+ {
+ # We can't simply call $node->stop here as the call is optimized out
+ # when the server isn't started with $node->start.
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D',
+ $data_dir, 'stop', '-m', 'fast');
+ ok($ret == 0, "Could stop postgres");
+ }
+}
+
+# We should be able to connect, and see an empty pg_ident.conf
+is($node->psql(
+ 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'),
+ qq(0),
+ 'pg_ident.conf is empty');
+
+############################################
+# part 1, test view reporting for valid data
+############################################
+my ($exp_hba, $exp_ident) = generate_valid_auth_files($node);
+
+$node->connect_ok('dbname=postgres', 'Connection still ok');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_hba_file_rules'),
+ qq($exp_hba),
+ 'pg_hba_file_rules content is expected');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_ident_file_mappings'),
+ qq($exp_ident),
+ 'pg_ident_file_mappings content is expected');
+
+#############################################
+# part 2, test log reporting for invalid data
+#############################################
+reset_auth_files($node);
+$node->restart('fast');
+$node->connect_ok('dbname=postgres',
+ 'Connection ok after resetting auth files');
+
+$node->stop('fast');
+
+start_errors_like($node, $hba_file, "include ../not_a_file",
+ [
+ qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file
+mkdir("$data_dir/hba_inc_fail");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token");
+start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail",
+ [
+ qr/invalid connection type "not_a_token"/,
+ qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file with nested inclusion
+unlink("$data_dir/hba_inc_fail/inc_dir.conf");
+my @hba_raw_errors_step1;
+
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1");
+
+add_hba_line($node, "hba_inc_fail/file1", "include file2");
+add_hba_line($node, "hba_inc_fail/file2", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file2", "include file3");
+
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3",
+ "local all all zuul",
+ 'invalid authentication method "zuul"');
+
+start_errors_like(
+ $node, $hba_file, "include_dir ../hba_inc_fail",
+ generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @hba_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local",
+ "end-of-line before database specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local,host",
+ "multiple values specified for connection type");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all",
+ "end-of-line before role specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all all",
+ "end-of-line before authentication method");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "host all all test/42",
+ 'specifying both host name and CIDR mask is invalid: "test/42"');
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ 'local @dbnames_fails.conf all reject',
+ "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory");
+
+add_hba_line($node, "hba_if_exists.conf", "include recurse.conf");
+push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf",
+ "include recurse.conf",
+ 'could not open configuration file "recurse.conf": maximum nesting depth exceeded');
+
+# Generate the regex for the expected errors in the logs. There's no guarantee
+# that the generated "line X of file..." will be emitted for the expected line,
+# but previous tests already ensured that the correct line number / file name
+# was emitted, so ensuring that there's an error in all expected lines is
+# enough here.
+my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2,
+ 1);
+
+# Not an error, but it should raise a message in the logs. Manually add an
+# extra log message to detect
+add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none");
+push @{$expected_errors},
+ qr/skipping missing authentication file "$data_dir\/if_exists_none"/;
+
+start_errors_like(
+ $node, $hba_file, "include_if_exists ../hba_if_exists.conf",
+ $expected_errors, 1);
+
+# Mostly the same, but for ident files
+reset_auth_files($node);
+
+my @ident_raw_errors_step1;
+
+# include_dir, single included file with nested inclusion
+mkdir("$data_dir/ident_inc_fail");
+add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1");
+
+add_ident_line($node, "ident_inc_fail/file1", "include file2");
+add_ident_line($node, "ident_inc_fail/file2", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file2", "include file3");
+
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3",
+ "failmap /(fail postgres",
+ 'invalid regular expression "\(fail": parentheses \(\) not balanced');
+
+start_errors_like(
+ $node, $ident_file, "include_dir ../ident_inc_fail",
+ generate_log_err_patterns($node, \@ident_raw_errors_step1, 0),
+ 0);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @ident_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map",
+ "missing entry at end of line");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2",
+ "multiple values in ident field");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf",
+ 'map @osnames_fails.conf postgres',
+ "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory");
+
+add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf",
+ 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded');
+
+start_errors_like(
+ $node, $ident_file, "include_if_exists ../ident_if_exists.conf",
+ # There's no guarantee that the generated "line X of file..." will be
+ # emitted for the expected line, but previous tests already ensured that
+ # the correct line number / file name was emitted, so ensuring that there's
+ # an error in all expected lines is enough here.
+ generate_log_err_patterns($node, \@ident_raw_errors_step2, 0),
+ 0);
+
+#####################################################
+# part 3, test reporting of various error scenario
+# NOTE: this will be bypassed -DEXEC_BACKEND or win32
+#####################################################
+reset_auth_files($node);
+
+$node->start;
+$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset');
+
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_hba_file_rules');
+
+add_ident_line($node, $ident_file, '');
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_ident_file_mappings');
+
+# The instance could be restarted and no error is detected. Now check if the
+# build is compatible with the view error reporting (EXEC_BACKEND / win32 will
+# fail when trying to connect as they always rely on the current auth files
+# content)
+my @hba_raw_errors;
+
+push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+my ($stdout, $stderr);
+my $cmdret = $node->psql('postgres', 'SELECT 1',
+ stdout => \$stdout, stderr => \$stderr);
+
+if ($cmdret != 0)
+{
+ # Connection failed. Bail out, but make sure to raise a failure if it
+ # didn't fail for the expected hba file modification.
+ like($stderr,
+ qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/,
+ "Connection failed due to loading an invalid hba file");
+
+ done_testing();
+ diag("Build not compatible with auth file view error reporting, bail out.\n");
+ exit;
+}
+
+# Combine errors generated at step 2, in the same order.
+$node->append_conf($hba_file, "include_dir ../hba_inc_fail");
+push @hba_raw_errors, @hba_raw_errors_step1;
+
+$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf");
+push @hba_raw_errors, @hba_raw_errors_step2;
+
+my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules'
+ . ' WHERE error IS NOT NULL ORDER BY rule_number'),
+ qq($hba_expected),
+ 'Detected all error in hba file');
+
+# and do the same for pg_ident
+my @ident_raw_errors;
+
+push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+$node->append_conf($ident_file, "include_dir ../ident_inc_fail");
+push @ident_raw_errors, @ident_raw_errors_step1;
+
+$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf");
+push @ident_raw_errors, @ident_raw_errors_step2;
+
+my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings'
+ . ' WHERE error IS NOT NULL ORDER BY mapping_number'),
+ qq($ident_expected),
+ 'Detected all error in ident file');
+
+done_testing();
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 79408710e0..5ed2fe3704 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname,
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
a.line_number,
a.type,
a.database,
@@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
a.line_number,
a.map_name,
a.sys_name,
a.pg_username,
a.error
- FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.37.0
--lqayot32rzm7n4sf
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment;
filename="v8-0006-POC-Add-a-pg_hba_matches-function.patch"
^ permalink raw reply [nested|flat] 33+ messages in thread
* [PATCH v14 4/5] Allow file inclusion in pg_hba and pg_ident files.
@ 2022-10-25 06:26 Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 33+ messages in thread
From: Julien Rouhaud @ 2022-10-25 06:26 UTC (permalink / raw)
pg_hba.conf file now has support for "include", "include_dir" and
"include_if_exists" directives, which work similarly to the same directives in
the postgresql.conf file.
This fixes a possible crash if a secondary file tries to include itself as
there's now a nesting depth check in the inclusion code path, same as the
postgresql.conf.
Many regression tests added to cover both the new directives, but also error
detection for the whole pg_hba / pg_ident files.
Catversion is bumped.
Author: Julien Rouhaud
Reviewed-by: FIXME
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud
---
doc/src/sgml/client-auth.sgml | 86 ++-
doc/src/sgml/system-views.sgml | 22 +-
src/backend/libpq/hba.c | 374 ++++++++--
src/backend/libpq/pg_hba.conf.sample | 25 +-
src/backend/libpq/pg_ident.conf.sample | 15 +-
src/backend/utils/adt/hbafuncs.c | 20 +-
src/backend/utils/misc/guc-file.l | 229 +++---
src/include/catalog/pg_proc.dat | 12 +-
src/include/libpq/hba.h | 3 +-
src/include/utils/guc.h | 2 +
.../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++
src/test/regress/expected/rules.out | 6 +-
12 files changed, 1239 insertions(+), 212 deletions(-)
create mode 100644 src/test/authentication/t/003_file_inclusion.pl
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index 32d5d45863..2ae723de66 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,23 @@
</para>
<para>
- Each record specifies a connection type, a client IP address range
- (if relevant for the connection type), a database name, a user name,
+ Each record can either be an inclusion directive or an authentication
+ record. Inclusion directives specify files that can be included, which
+ contains additional records. The records will be inserted in lieu of the
+ inclusion records. Those records only contains two fields: the
+ <literal>include</literal>, <literal>include_if_exists</literal> or
+ <literal>include_dir</literal> directive and the file or directory to be
+ included. The file or directory can be a relative of absolute path, and can
+ be double quoted if needed. For the <literal>include_dir</literal> form,
+ all files not starting with a <literal>.</literal> and ending with
+ <literal>.conf</literal> will be included. Multiple files within an include
+ directory are processed in file name order (according to C locale rules,
+ i.e., numbers before letters, and uppercase letters before lowercase ones).
+ </para>
+
+ <para>
+ Each authentication record specifies a connection type, a client IP address
+ range (if relevant for the connection type), a database name, a user name,
and the authentication method to be used for connections matching
these parameters. The first record with a matching connection type,
client address, requested database, and user name is used to perform
@@ -103,21 +118,57 @@
<para>
A record can have several formats:
<synopsis>
-local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+include <replaceable>file</replaceable>
+include_if_exists <replaceable>file</replaceable>
+include_dir <replaceable>directory</replaceable>
+local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
</synopsis>
The meaning of the fields is as follows:
<variablelist>
+ <varlistentry>
+ <term><literal>include</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_if_exists</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of the given file if the
+ file exists and can be read. Otherwise, a message will be logged to
+ indicate that the file is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>include_dir</literal></term>
+ <listitem>
+ <para>
+ This line will be replaced with the content of all the files found in
+ the directory, if they don't start with a <literal>.</literal> and end
+ with <literal>.conf</literal>, processed in file name order (according
+ to C locale rules, i.e., numbers before letters, and uppercase letters
+ before lowercase ones).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>local</literal></term>
<listitem>
@@ -863,8 +914,10 @@ local db1,db2,@demodbs all md5
cluster's data directory. (It is possible to place the map file
elsewhere, however; see the <xref linkend="guc-ident-file"/>
configuration parameter.)
- The ident map file contains lines of the general form:
+ The ident map file contains lines of two general form:
<synopsis>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
+<replaceable>include_dir</replaceable> <replaceable>directory</replaceable>
<replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
</synopsis>
Comments, whitespace and line continuations are handled in the same way as in
@@ -875,6 +928,11 @@ local db1,db2,@demodbs all md5
database user name. The same <replaceable>map-name</replaceable> can be
used repeatedly to specify multiple user-mappings within a single map.
</para>
+ <para>
+ As for <filename>pg_hba.conf</filename>, the lines in this file can either
+ be inclusion directives or user name map records, and follow the same
+ rules.
+ </para>
<para>
There is no restriction regarding how many database users a given
operating system user can correspond to, nor vice versa. Thus, entries
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 4723f712a7..7d1cec8b7f 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -1003,12 +1003,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this rule
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_hba.conf</filename>
+ Line number of this rule the given <literal>file_name</literal>
</para></entry>
</row>
@@ -1153,12 +1162,21 @@
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>file_name</structfield> <type>text</type>
+ </para>
+ <para>
+ Name of the file containing this mapping
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>line_number</structfield> <type>int4</type>
</para>
<para>
- Line number of this rule in <filename>pg_ident.conf</filename>
+ Line number of this mapping in the given <literal>file_name</literal>
</para></entry>
</row>
diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c
index 56bbe31dfd..74f46ffb04 100644
--- a/src/backend/libpq/hba.c
+++ b/src/backend/libpq/hba.c
@@ -22,6 +22,7 @@
#include <sys/param.h>
#include <sys/socket.h>
#include <netdb.h>
+#include <sys/stat.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <unistd.h>
@@ -70,6 +71,12 @@ typedef struct check_network_data
#define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0)
#define token_matches(t, k) (strcmp(t->string, k) == 0)
+typedef enum HbaIncludeKind
+{
+ SecondaryAuthFile,
+ IncludedAuthFile
+} HbaIncludeKind;
+
/*
* pre-parsed content of HBA config file: list of HbaLine structs.
* parsed_hba_context is the memory context where it lives.
@@ -115,14 +122,26 @@ static const char *const UserAuthName[] =
};
+static void tokenize_file_with_context(MemoryContext linecxt,
+ const char *filename, FILE *file,
+ List **tok_lines, int depth,
+ int elevel);
static List *tokenize_inc_file(List *tokens, const char *outer_filename,
- const char *inc_filename, int elevel, char **err_msg);
+ const char *inc_filename, int depth, int elevel,
+ char **err_msg);
static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline,
int elevel, char **err_msg);
static int regcomp_auth_token(AuthToken *token, char *filename, int line_num,
char **err_msg, int elevel);
static int regexec_auth_token(const char *match, AuthToken *token,
size_t nmatch, regmatch_t pmatch[]);
+static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename,
+ bool strict, const char *outer_filename, int elevel,
+ char **err_msg, char **inc_fullname);
+static char *process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth,
+ int elevel, MemoryContext linecxt,
+ List **tok_lines);
/*
@@ -413,7 +432,7 @@ regexec_auth_token(const char *match, AuthToken *token, size_t nmatch,
*/
static List *
next_field_expand(const char *filename, char **lineptr,
- int elevel, char **err_msg)
+ int depth, int elevel, char **err_msg)
{
char buf[MAX_TOKEN];
bool trailing_comma;
@@ -429,7 +448,7 @@ next_field_expand(const char *filename, char **lineptr,
/* Is this referencing a file? */
if (!initial_quote && buf[0] == '@' && buf[1] != '\0')
- tokens = tokenize_inc_file(tokens, filename, buf + 1,
+ tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1,
elevel, err_msg);
else
tokens = lappend(tokens, make_auth_token(buf, initial_quote));
@@ -457,6 +476,7 @@ static List *
tokenize_inc_file(List *tokens,
const char *outer_filename,
const char *inc_filename,
+ int depth,
int elevel,
char **err_msg)
{
@@ -466,39 +486,30 @@ tokenize_inc_file(List *tokens,
ListCell *inc_line;
MemoryContext linecxt;
- if (is_absolute_path(inc_filename))
- {
- /* absolute path is taken as-is */
- inc_fullname = pstrdup(inc_filename);
- }
- else
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
{
- /* relative path is relative to dir of calling file */
- inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
- strlen(inc_filename) + 1);
- strcpy(inc_fullname, outer_filename);
- get_parent_directory(inc_fullname);
- join_path_components(inc_fullname, inc_fullname, inc_filename);
- canonicalize_path(inc_fullname);
+ *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", *err_msg)));
+ return tokens;
}
- inc_file = AllocateFile(inc_fullname, "r");
- if (inc_file == NULL)
- {
- int save_errno = errno;
+ inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true,
+ outer_filename, elevel, err_msg, &inc_fullname);
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m",
- inc_filename, inc_fullname)));
- *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s",
- inc_filename, inc_fullname, strerror(save_errno));
- pfree(inc_fullname);
+ if (inc_file == NULL)
return tokens;
- }
/* There is possible recursion here if the file contains @ */
- linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel);
+ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1,
+ elevel);
FreeFile(inc_file);
pfree(inc_fullname);
@@ -536,11 +547,38 @@ tokenize_inc_file(List *tokens,
/*
* tokenize_auth_file
- * Tokenize the given file.
+ *
+ * Wrapper around tokenize_file_with_context, creating a dedicated memory
+ * context.
+ *
+ * Return value is this memory context which contains all memory allocated by
+ * this function (it's a child of caller's context).
+ */
+MemoryContext
+tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
+ int depth, int elevel)
+{
+ MemoryContext linecxt;
+ linecxt = AllocSetContextCreate(CurrentMemoryContext,
+ "tokenize_auth_file",
+ ALLOCSET_SMALL_SIZES);
+
+ *tok_lines = NIL;
+
+ tokenize_file_with_context(linecxt, filename, file, tok_lines, depth,
+ elevel);
+
+ return linecxt;
+}
+
+/*
+ * Tokenize the given file.
*
* The output is a list of TokenizedAuthLine structs; see the struct definition
* in libpq/hba.h.
*
+ * linecxt: memory context which must contain all memory allocated by the
+ * function
* filename: the absolute path to the target file
* file: the already-opened target file
* tok_lines: receives output list
@@ -549,30 +587,22 @@ tokenize_inc_file(List *tokens,
* Errors are reported by logging messages at ereport level elevel and by
* adding TokenizedAuthLine structs containing non-null err_msg fields to the
* output list.
- *
- * Return value is a memory context which contains all memory allocated by
- * this function (it's a child of caller's context).
*/
-MemoryContext
-tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
- int elevel)
+static void
+tokenize_file_with_context(MemoryContext linecxt, const char *filename,
+ FILE *file, List **tok_lines, int depth, int elevel)
{
- int line_number = 1;
StringInfoData buf;
- MemoryContext linecxt;
+ int line_number = 1;
MemoryContext oldcxt;
- linecxt = AllocSetContextCreate(CurrentMemoryContext,
- "tokenize_auth_file",
- ALLOCSET_SMALL_SIZES);
oldcxt = MemoryContextSwitchTo(linecxt);
initStringInfo(&buf);
- *tok_lines = NIL;
-
while (!feof(file) && !ferror(file))
{
+ TokenizedAuthLine *tok_line;
char *lineptr;
List *current_line = NIL;
char *err_msg = NULL;
@@ -625,7 +655,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
{
List *current_field;
- current_field = next_field_expand(filename, &lineptr,
+ current_field = next_field_expand(filename, &lineptr, depth,
elevel, &err_msg);
/* add field to line, unless we are at EOL or comment start */
if (current_field != NIL)
@@ -633,30 +663,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines,
}
/*
- * Reached EOL; emit line to TokenizedAuthLine list unless it's boring
+ * Reached EOL; no need to emit line to TokenizedAuthLine list if it's
+ * boring.
*/
- if (current_line != NIL || err_msg != NULL)
+ if (current_line == NIL && err_msg == NULL)
+ goto next_line;
+
+ /* If the line is valid, check if that's an include directive */
+ if (err_msg == NULL && list_length(current_line) == 2)
{
- TokenizedAuthLine *tok_line;
+ AuthToken *first, *second;
+
+ first = linitial(linitial_node(List, current_line));
+ second = linitial(lsecond_node(List, current_line));
+
+ if (strcmp(first->string, "include") == 0)
+ {
+ char *inc_filename;
+
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, true,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
+ else if (strcmp(first->string, "include_dir") == 0)
+ {
+ char **filenames;
+ char *dir_name = second->string;
+ int num_filenames;
+ StringInfoData err_buf;
+
+ filenames = GetDirConfFiles(dir_name, filename, elevel,
+ &num_filenames, &err_msg);
+
+ if (!filenames)
+ {
+ /* We have the error in err_msg, simply process it */
+ goto process_line;
+ }
+
+ initStringInfo(&err_buf);
+ for (int i = 0; i < num_filenames; i++)
+ {
+ /*
+ * err_msg is used here as a temp buffer, it will be
+ * overwritten at the end of the loop with the
+ * cumulated errors, if any.
+ */
+ err_msg = process_included_authfile(filenames[i], true,
+ filename, depth + 1, elevel,
+ linecxt, tok_lines);
+
+ /* Cumulate errors if any. */
+ if (err_msg)
+ {
+ if (err_buf.len > 0)
+ appendStringInfoChar(&err_buf, '\n');
+ appendStringInfoString(&err_buf, err_msg);
+ }
+ }
+
+ /*
+ * If there were no errors, the line is fully processed, bypass
+ * the general TokenizedAuthLine processing.
+ */
+ if (err_buf.len == 0)
+ goto next_line;
+
+ /* Otherwise, process the cumulated errors, if any. */
+ err_msg = err_buf.data;
+ }
+ else if (strcmp(first->string, "include_if_exists") == 0)
+ {
+ char *inc_filename;
- tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
- tok_line->fields = current_line;
- tok_line->file_name = pstrdup(filename);
- tok_line->line_num = line_number;
- tok_line->raw_line = pstrdup(buf.data);
- tok_line->err_msg = err_msg;
- *tok_lines = lappend(*tok_lines, tok_line);
+ inc_filename = second->string;
+
+ err_msg = process_included_authfile(inc_filename, false,
+ filename, depth + 1, elevel, linecxt,
+ tok_lines);
+
+ if (!err_msg)
+ {
+ /*
+ * The line is fully processed, bypass the general
+ * TokenizedAuthLine processing.
+ */
+ goto next_line;
+ }
+ }
}
+process_line:
+ /*
+ * General processing: report the error if any and emit line to the
+ * TokenizedAuthLine
+ */
+ tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine));
+ tok_line->fields = current_line;
+ tok_line->file_name = pstrdup(filename);
+ tok_line->line_num = line_number;
+ tok_line->raw_line = pstrdup(buf.data);
+ tok_line->err_msg = err_msg;
+ *tok_lines = lappend(*tok_lines, tok_line);
+
+next_line:
line_number += continuations + 1;
}
MemoryContextSwitchTo(oldcxt);
-
- return linecxt;
}
-
/*
* Does user belong to role?
*
@@ -2355,7 +2482,7 @@ load_hba(void)
return false;
}
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
@@ -2445,6 +2572,135 @@ load_hba(void)
return true;
}
+/*
+ * Open the given file for inclusion in an authentication file, whether
+ * secondary or included.
+ */
+static FILE *
+open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict,
+ const char *outer_filename, int elevel, char **err_msg,
+ char **inc_fullname)
+{
+ FILE *inc_file;
+
+ if (is_absolute_path(inc_filename))
+ {
+ /* absolute path is taken as-is */
+ *inc_fullname = pstrdup(inc_filename);
+ }
+ else
+ {
+ /* relative path is relative to dir of calling file */
+ *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 +
+ strlen(inc_filename) + 1);
+ strcpy(*inc_fullname, outer_filename);
+ get_parent_directory(*inc_fullname);
+ join_path_components(*inc_fullname, *inc_fullname, inc_filename);
+ canonicalize_path(*inc_fullname);
+ }
+
+ inc_file = AllocateFile(*inc_fullname, "r");
+ if (inc_file == NULL)
+ {
+ int save_errno = errno;
+ const char *msglog;
+ const char *msgview;
+
+ if (strict)
+ {
+ switch (kind)
+ {
+ case SecondaryAuthFile:
+ msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m";
+ msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s";
+ break;
+ case IncludedAuthFile:
+ msglog = "could not open included authentication file \"%s\" as \"%s\": %m";
+ msgview = "could not open included authentication file \"%s\" as \"%s\": %s";
+ break;
+ default:
+ elog(ERROR, "unknown HbaIncludeKind: %d", kind);
+ break;
+ }
+
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg(msglog, inc_filename, *inc_fullname)));
+ *err_msg = psprintf(msgview, inc_filename, *inc_fullname,
+ strerror(save_errno));
+ }
+ else
+ {
+ Assert(kind == IncludedAuthFile);
+ ereport(LOG,
+ (errmsg("skipping missing authentication file \"%s\"",
+ *inc_fullname)));
+ }
+
+ pfree(*inc_fullname);
+ *inc_fullname = NULL;
+ return NULL;
+ }
+
+ return inc_file;
+}
+
+/*
+ * Try to open an included file, and tokenize it using the given context.
+ * Returns NULL if no error happens during tokenization, otherwise the error.
+ */
+static char *
+process_included_authfile(const char *inc_filename, bool strict,
+ const char *outer_filename, int depth, int elevel,
+ MemoryContext linecxt, List **tok_lines)
+{
+ char *inc_fullname;
+ FILE *inc_file;
+ char *err_msg = NULL;
+
+ /*
+ * Reject too-deep include nesting depth. This is just a safety check to
+ * avoid dumping core due to stack overflow if an include file loops back
+ * to itself. The maximum nesting depth is pretty arbitrary.
+ */
+ if (depth > 10)
+ {
+ err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded",
+ inc_filename);
+ ereport(elevel,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("%s", err_msg)));
+ return err_msg;
+ }
+
+ inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict,
+ outer_filename, elevel, &err_msg, &inc_fullname);
+
+ if (inc_file == NULL)
+ {
+ if (strict)
+ {
+ /* open_inc_file should have reported an error. */
+ Assert(err_msg != NULL);
+ return err_msg;
+ }
+ else
+ return NULL;
+ }
+ else
+ {
+ /* No error message should have been reported. */
+ Assert(err_msg == NULL);
+ }
+
+ tokenize_file_with_context(linecxt, inc_fullname, inc_file,
+ tok_lines, depth, elevel);
+
+ FreeFile(inc_file);
+ pfree(inc_fullname);
+
+ return NULL;
+}
/*
* Parse one tokenised line from the ident config file and store the result in
@@ -2728,7 +2984,7 @@ load_ident(void)
return false;
}
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG);
FreeFile(file);
/* Now parse all the lines */
diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample
index 5f3f63eb0c..7433050112 100644
--- a/src/backend/libpq/pg_hba.conf.sample
+++ b/src/backend/libpq/pg_hba.conf.sample
@@ -9,16 +9,27 @@
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
-# local DATABASE USER METHOD [OPTIONS]
-# host DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
-# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# local DATABASE USER METHOD [OPTIONS]
+# host DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS]
+# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS]
#
# (The uppercase items must be replaced by actual values.)
#
-# The first field is the connection type:
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
+# Otherwise the first field is the connection type:
# - "local" is a Unix-domain socket
# - "host" is a TCP/IP socket (encrypted or not)
# - "hostssl" is a TCP/IP socket that is SSL-encrypted
diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample
index a5870e6448..8e3fa29135 100644
--- a/src/backend/libpq/pg_ident.conf.sample
+++ b/src/backend/libpq/pg_ident.conf.sample
@@ -7,12 +7,23 @@
#
# This file controls PostgreSQL user name mapping. It maps external
# user names to their corresponding PostgreSQL user names. Records
-# are of the form:
+# are one of these forms:
#
-# MAPNAME SYSTEM-USERNAME PG-USERNAME
+# include FILE
+# include_if_exists FILE
+# include_dir DIRECTORY
+# MAPNAME SYSTEM-USERNAME PG-USERNAME
#
# (The uppercase quantities must be replaced by actual values.)
#
+# If the first field is "include", "include_if_exists" or "include_dir", it's
+# not a mapping record but a directive to include records from respectively
+# another file, another file if it exists or all the files in the given
+# directory ending in '.conf'. FILE is the file name to include, and
+# DIR is the directory name containing the file(s) to include. FILE and
+# DIRECTORY can be specified with a relative or absolute path, and can be
+# double quoted if they contains spaces.
+#
# MAPNAME is the (otherwise freely chosen) map name that was used in
# pg_hba.conf. SYSTEM-USERNAME is the detected user name of the
# client. PG-USERNAME is the requested PostgreSQL user name. The
diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c
index 255109ab07..d356804c5a 100644
--- a/src/backend/utils/adt/hbafuncs.c
+++ b/src/backend/utils/adt/hbafuncs.c
@@ -158,7 +158,7 @@ get_hba_options(HbaLine *hba)
}
/* Number of columns in pg_hba_file_rules view */
-#define NUM_PG_HBA_FILE_RULES_ATTS 10
+#define NUM_PG_HBA_FILE_RULES_ATTS 11
/*
* fill_hba_line
@@ -199,6 +199,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(rule_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(hba->sourcefile);
/* line_number */
values[index++] = Int32GetDatum(hba->linenumber);
@@ -342,7 +344,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -383,7 +385,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open configuration file \"%s\": %m",
HbaFileName)));
- linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3);
+ linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -442,7 +444,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS)
}
/* Number of columns in pg_hba_file_mappings view */
-#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6
+#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7
/*
* fill_ident_line: build one row of pg_ident_file_mappings view, add it to
@@ -477,6 +479,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
nulls[index++] = true;
else
values[index++] = Int32GetDatum(mapping_number);
+ /* file_name */
+ values[index++] = CStringGetTextDatum(ident->sourcefile);
/* line_number */
values[index++] = Int32GetDatum(ident->linenumber);
@@ -489,7 +493,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc,
else
{
/* no parsing result, so set relevant fields to nulls */
- memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool));
+ memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool));
}
/* error */
@@ -529,7 +533,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
errmsg("could not open usermap file \"%s\": %m",
IdentFileName)));
- linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3);
+ linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3);
FreeFile(file);
/* Now parse all the lines */
@@ -550,8 +554,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc)
if (tok_line->err_msg == NULL)
mapping_number++;
- fill_ident_line(tuple_store, tupdesc, mapping_number, identline,
- tok_line->err_msg);
+ fill_ident_line(tuple_store, tupdesc, mapping_number,
+ identline, tok_line->err_msg);
}
/* Free tokenizer memory */
diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l
index 721628c0cf..86b6cc1c8a 100644
--- a/src/backend/utils/misc/guc-file.l
+++ b/src/backend/utils/misc/guc-file.l
@@ -345,6 +345,110 @@ GUC_flex_fatal(const char *msg)
return 0; /* keep compiler quiet */
}
+/*
+ * Returns the list of config files located in a directory, in alphabetical
+ * order.
+ *
+ * We don't check for recursion or too-deep nesting depth here, its up to the
+ * caller to take care of that.
+ */
+char **
+GetDirConfFiles(const char *includedir, const char *calling_file, int elevel,
+ int *num_filenames, char **err_msg)
+{
+ char *directory;
+ DIR *d;
+ struct dirent *de;
+ char **filenames;
+ int size_filenames;
+
+ /*
+ * Reject directory name that is all-blank (including empty), as that
+ * leads to confusion --- we'd read the containing directory, typically
+ * resulting in recursive inclusion of the same file(s).
+ */
+ if (strspn(includedir, " \t\r\n") == strlen(includedir))
+ {
+ ereport(elevel,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("empty configuration directory name: \"%s\"",
+ includedir)));
+ *err_msg = "empty configuration directory name";
+ return NULL;
+ }
+
+ directory = AbsoluteConfigLocation(includedir, calling_file);
+ d = AllocateDir(directory);
+ if (d == NULL)
+ {
+ ereport(elevel,
+ (errcode_for_file_access(),
+ errmsg("could not open configuration directory \"%s\": %m",
+ directory)));
+ *err_msg = psprintf("could not open directory \"%s\"", directory);
+ filenames = NULL;
+ goto cleanup;
+ }
+
+ /*
+ * Read the directory and put the filenames in an array, so we can sort
+ * them prior to caller processing the contents.
+ */
+ size_filenames = 32;
+ filenames = (char **) palloc(size_filenames * sizeof(char *));
+ *num_filenames = 0;
+
+ while ((de = ReadDir(d, directory)) != NULL)
+ {
+ PGFileType de_type;
+ char filename[MAXPGPATH];
+
+ /*
+ * Only parse files with names ending in ".conf". Explicitly reject
+ * files starting with ".". This excludes things like "." and "..",
+ * as well as typical hidden files, backup files, and editor debris.
+ */
+ if (strlen(de->d_name) < 6)
+ continue;
+ if (de->d_name[0] == '.')
+ continue;
+ if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
+ continue;
+
+ join_path_components(filename, directory, de->d_name);
+ canonicalize_path(filename);
+ de_type = get_dirent_type(filename, de, true, elevel);
+ if (de_type == PGFILETYPE_ERROR)
+ {
+ *err_msg = psprintf("could not stat file \"%s\"", filename);
+ pfree(filenames);
+ filenames = NULL;
+ goto cleanup;
+ }
+ else if (de_type != PGFILETYPE_DIR)
+ {
+ /* Add file to array, increasing its size in blocks of 32 */
+ if (*num_filenames >= size_filenames)
+ {
+ size_filenames += 32;
+ filenames = (char **) repalloc(filenames,
+ size_filenames * sizeof(char *));
+ }
+ filenames[*num_filenames] = pstrdup(filename);
+ (*num_filenames)++;
+ }
+ }
+
+ if (*num_filenames > 0)
+ qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp);
+
+cleanup:
+ if (d)
+ FreeDir(d);
+ pfree(directory);
+ return filenames;
+}
+
/*
* Read and parse a single configuration file. This function recurses
* to handle "include" directives.
@@ -606,127 +710,30 @@ ParseConfigDirectory(const char *includedir,
ConfigVariable **head_p,
ConfigVariable **tail_p)
{
- char *directory;
- DIR *d;
- struct dirent *de;
+ char *err_msg;
char **filenames;
int num_filenames;
- int size_filenames;
- bool status;
-
- /*
- * Reject directory name that is all-blank (including empty), as that
- * leads to confusion --- we'd read the containing directory, typically
- * resulting in recursive inclusion of the same file(s).
- */
- if (strspn(includedir, " \t\r\n") == strlen(includedir))
- {
- ereport(elevel,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("empty configuration directory name: \"%s\"",
- includedir)));
- record_config_file_error("empty configuration directory name",
- calling_file, calling_lineno,
- head_p, tail_p);
- return false;
- }
-
- /*
- * We don't check for recursion or too-deep nesting depth here; the
- * subsequent calls to ParseConfigFile will take care of that.
- */
-
- directory = AbsoluteConfigLocation(includedir, calling_file);
- d = AllocateDir(directory);
- if (d == NULL)
- {
- ereport(elevel,
- (errcode_for_file_access(),
- errmsg("could not open configuration directory \"%s\": %m",
- directory)));
- record_config_file_error(psprintf("could not open directory \"%s\"",
- directory),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
- /*
- * Read the directory and put the filenames in an array, so we can sort
- * them prior to processing the contents.
- */
- size_filenames = 32;
- filenames = (char **) palloc(size_filenames * sizeof(char *));
- num_filenames = 0;
+ filenames = GetDirConfFiles(includedir, calling_file, elevel,
+ &num_filenames, &err_msg);
- while ((de = ReadDir(d, directory)) != NULL)
+ if (!filenames)
{
- PGFileType de_type;
- char filename[MAXPGPATH];
-
- /*
- * Only parse files with names ending in ".conf". Explicitly reject
- * files starting with ".". This excludes things like "." and "..",
- * as well as typical hidden files, backup files, and editor debris.
- */
- if (strlen(de->d_name) < 6)
- continue;
- if (de->d_name[0] == '.')
- continue;
- if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0)
- continue;
-
- join_path_components(filename, directory, de->d_name);
- canonicalize_path(filename);
- de_type = get_dirent_type(filename, de, true, elevel);
- if (de_type == PGFILETYPE_ERROR)
- {
- record_config_file_error(psprintf("could not stat file \"%s\"",
- filename),
- calling_file, calling_lineno,
- head_p, tail_p);
- status = false;
- goto cleanup;
- }
- else if (de_type != PGFILETYPE_DIR)
- {
- /* Add file to array, increasing its size in blocks of 32 */
- if (num_filenames >= size_filenames)
- {
- size_filenames += 32;
- filenames = (char **) repalloc(filenames,
- size_filenames * sizeof(char *));
- }
- filenames[num_filenames] = pstrdup(filename);
- num_filenames++;
- }
+ record_config_file_error(err_msg, calling_file, calling_lineno, head_p,
+ tail_p);
+ return false;
}
- if (num_filenames > 0)
+ for (int i = 0; i < num_filenames; i++)
{
- int i;
-
- qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp);
- for (i = 0; i < num_filenames; i++)
- {
- if (!ParseConfigFile(filenames[i], true,
- calling_file, calling_lineno,
- depth, elevel,
- head_p, tail_p))
- {
- status = false;
- goto cleanup;
- }
- }
+ if (!ParseConfigFile(filenames[i], true,
+ calling_file, calling_lineno,
+ depth, elevel,
+ head_p, tail_p))
+ return false;
}
- status = true;
-cleanup:
- if (d)
- FreeDir(d);
- pfree(directory);
- return status;
+ return true;
}
/*
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4f5d05d0ce..2ad06c4d3e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -6135,16 +6135,16 @@
{ oid => '3401', descr => 'show pg_hba.conf rules',
proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
+ proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}',
prosrc => 'pg_hba_file_rules' },
{ oid => '6250', descr => 'show pg_ident.conf mappings',
proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,int4,text,text,text,text}',
- proargmodes => '{o,o,o,o,o,o}',
- proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}',
+ proallargtypes => '{int4,text,int4,text,text,text,text}',
+ proargmodes => '{o,o,o,o,o,o,o}',
+ proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}',
prosrc => 'pg_ident_file_mappings' },
{ oid => '1371', descr => 'view system lock information',
proname => 'pg_lock_status', prorows => '1000', proretset => 't',
diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h
index bf896ac084..7108cd2dae 100644
--- a/src/include/libpq/hba.h
+++ b/src/include/libpq/hba.h
@@ -179,6 +179,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel);
extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel);
extern bool pg_isblank(const char c);
extern MemoryContext tokenize_auth_file(const char *filename, FILE *file,
- List **tok_lines, int elevel);
+ List **tok_lines, int depth,
+ int elevel);
#endif /* HBA_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index b3aaff9665..59ca39d908 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -144,6 +144,8 @@ typedef struct ConfigVariable
struct ConfigVariable *next;
} ConfigVariable;
+extern char **GetDirConfFiles(const char *includedir, const char *calling_file,
+ int elevel, int *num_filenames, char **err_msg);
extern bool ParseConfigFile(const char *config_file, bool strict,
const char *calling_file, int calling_lineno,
int depth, int elevel,
diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl
new file mode 100644
index 0000000000..8eae72b8d4
--- /dev/null
+++ b/src/test/authentication/t/003_file_inclusion.pl
@@ -0,0 +1,657 @@
+
+# Copyright (c) 2021-2022, PostgreSQL Global Development Group
+
+# Set of tests for authentication and pg_hba.conf inclusion.
+# This test can only run with Unix-domain sockets.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Time::HiRes qw(usleep);
+use IPC::Run qw(pump finish timer);
+use Data::Dumper;
+
+if (!$use_unix_sockets)
+{
+ plan skip_all =>
+ "authentication tests cannot run without Unix-domain sockets";
+}
+
+# stores the current line counter for each file. hba_rule and ident_rule are
+# fake file names used for the global rule number for each auth view.
+my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+my $hba_file = 'subdir1/pg_hba_custom.conf';
+my $ident_file = 'subdir2/pg_ident_custom.conf';
+
+# Initialize primary node
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+$node->start;
+
+my $data_dir = $node->data_dir;
+
+# Normalize the data directory for Windows
+$data_dir =~ s/\/\.\//\//g; # reduce /./ to /
+$data_dir =~ s/\/\//\//g; # reduce // to /
+$data_dir =~ s/\/$//; # remove trailing /
+
+
+# Add the given payload to the given relative HBA file of the given node.
+# This function maintains the %cur_line metadata, so it has to be called in the
+# expected inclusion evaluation order in order to keep it in sync.
+#
+# If the payload starts with "include" or "ignore", the function doesn't
+# increase the general hba rule number.
+#
+# If an err_str is provided, it returns an arrayref containing the provided
+# filename, the current line number in that file and the provided err_str. The
+# err_str has to be a valid regex string.
+# Otherwise it only returns the line number of the payload in the wanted file.
+# This function has to be called in the expected inclusion evaluation order to
+# keep the %cur_line information in sync.
+sub add_hba_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'hba_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_hba_file_rules line
+ @tokens = split(/ /, $payload);
+ $tokens[1] = '{' . $tokens[1] . '}'; # database
+ $tokens[2] = '{' . $tokens[2] . '}'; # user_name
+
+ # add empty address and netmask betweed user_name and auth_method
+ splice @tokens, 3, 0, '';
+ splice @tokens, 3, 0, '';
+
+ # append empty options and error
+ push @tokens, '';
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Add the given payload to the given relative ident file of the given node.
+# Same as add_hba_line but for pg_ident files
+sub add_ident_line
+{
+ my $node = shift;
+ my $filename = shift;
+ my $payload = shift;
+ my $err_str = shift;
+ my $globline;
+ my $fileline;
+ my @tokens;
+ my $line;
+
+ # Append the payload to the given file
+ $node->append_conf($filename, $payload);
+
+ # Get the current %cur_line counter for the file
+ if (not defined $cur_line{$filename})
+ {
+ $cur_line{$filename} = 1;
+ }
+ $fileline = $cur_line{$filename}++;
+
+ # Include directive, don't generate an underlying pg_hba_file_rules line
+ # but make sure we incremented the %cur_line counter.
+ # Also ignore line beginning with "ignore", for content of files that
+ # should not being included
+ if ($payload =~ qr/^(include|ignore)/)
+ {
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+ else
+ {
+ return $fileline;
+ }
+ }
+
+ # Get (and increment) the global rule number
+ $globline = $cur_line{'ident_rule'}++;
+
+ # If caller provided an err_str, just returns the needed metadata
+ if (defined $err_str)
+ {
+ return [$filename, $fileline, $err_str];
+ }
+
+ # Otherwise, generate the expected pg_ident_file_mappings line
+ @tokens = split(/ /, $payload);
+
+ # append empty error
+ push @tokens, '';
+
+ # generate the expected final line
+ $line = "";
+ $line .= "\n" if ($globline > 1);
+ $line .= "$globline|$data_dir/$filename|$fileline|";
+ $line .= join('|', @tokens);
+
+ return $line;
+}
+
+# Delete pg_hba.conf from the given node, add various entries to test the
+# include infrastructure and then execute a reload to refresh it.
+sub generate_valid_auth_files
+{
+ my $node = shift;
+ my $hba_expected = '';
+ my $ident_expected = '';
+
+ # customise main auth file names
+ $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'");
+ $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'");
+
+ # and make original ones invalid to be sure they're not used anywhere
+ $node->append_conf('pg_hba.conf', "some invalid line");
+ $node->append_conf('pg_ident.conf', "some invalid line");
+
+ # pg_hba stuff
+ mkdir("$data_dir/subdir1");
+ mkdir("$data_dir/hba_inc");
+ mkdir("$data_dir/hba_inc_if");
+ mkdir("$data_dir/hba_pos");
+
+ # Make sure we will still be able to connect
+ $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust');
+
+ # Add include data
+ add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf");
+ $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject");
+
+ $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject");
+
+ add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject");
+ # include is relative to current path
+ add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf");
+ $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject");
+
+ # include_if_exists data
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none");
+ add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some");
+ $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject");
+
+ # include_dir data
+ add_hba_line($node, "$hba_file", "include_dir ../hba_inc");
+ add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included");
+ $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject");
+ $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject");
+
+ # secondary auth file
+ add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject');
+ $node->append_conf('dbnames.conf', "db1");
+ $node->append_conf('dbnames.conf', "db3");
+ $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1)
+ . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1)
+ . '|local|{db1,db3}|{all}|||reject||';
+
+ # pg_ident stuff
+ mkdir("$data_dir/subdir2");
+ mkdir("$data_dir/ident_inc");
+ mkdir("$data_dir/ident_inc_if");
+ mkdir("$data_dir/ident_pos");
+
+ # Add include data
+ add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf");
+ $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar");
+
+ $ident_expected .= add_ident_line($node, "$ident_file", "test a b");
+
+ add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar");
+ # include is relative to current path
+ add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf");
+ $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar");
+
+ # include_if_exists data
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none");
+ add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some");
+ $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar");
+
+ # include_dir data
+ add_ident_line($node, "$ident_file", "include_dir ../ident_inc");
+ add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included");
+ $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar");
+ $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar");
+
+ $node->restart;
+ $node->connect_ok('dbname=postgres',
+ 'Connection ok after generating valid auth files');
+
+ return ($hba_expected, $ident_expected);
+}
+
+# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal
+# entries to allow authentication.
+sub reset_auth_files
+{
+ my $node = shift;
+
+ unlink("$data_dir/$hba_file");
+ unlink("$data_dir/$ident_file");
+
+ %cur_line = ('hba_rule' => 1, 'ident_rule' => 1);
+
+ return add_hba_line($node, "$hba_file", 'local all all trust');
+}
+
+# Generate a list of expected error regex for the given array of error
+# conditions, as generated by add_hba_line/add_ident_line with an err_str.
+#
+# 2 regex are generated per array entry: one for the given err_str, and one for
+# the expected line in the specific file. Since all lines are independant,
+# there's no guarantee that a specific failure regex and the per-line regex
+# will match the same error. Calling code should add at least one test with a
+# single error to make sure that the line number / file name is correct.
+#
+# On top of that, an extra line is generated for the general failure to process
+# the main auth file.
+sub generate_log_err_patterns
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $is_hba_err = shift;
+ my @errors;
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ push @errors, qr/$err_str/;
+
+ # Context messages with the file / line location aren't always emitted
+ if ($err_str !~ /maximum nesting depth exceeded/ and
+ $err_str !~ /could not open secondary authentication file/)
+ {
+ push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/
+ }
+ }
+
+ push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err);
+
+ return \@errors;
+}
+
+# Generate the expected output for the auth file view error reporting (file
+# name, file line, error), for the given array of error conditions, as
+# generated generated by add_hba_line/add_ident_line with an err_str.
+sub generate_log_err_rows
+{
+ my $node = shift;
+ my $raw_errors = shift;
+ my $exp_rows = '';
+
+ foreach my $arr (@{$raw_errors})
+ {
+ my $filename = @{$arr}[0];
+ my $fileline = @{$arr}[1];
+ my $err_str = @{$arr}[2];
+
+ $exp_rows .= "\n" if ($exp_rows ne "");
+
+ # Unescape regex patterns if any
+ $err_str =~ s/\\([\(\)])/$1/g;
+ $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str"
+ }
+
+ return $exp_rows;
+}
+
+# Reset the main auth files, append the given payload to the given config file,
+# and check that the instance cannot start, raising the expected error line(s).
+sub start_errors_like
+{
+ my $node = shift;
+ my $file = shift;
+ my $payload = shift;
+ my $pattern = shift;
+ my $should_fail = shift;
+
+ reset_auth_files($node);
+ $node->append_conf($file, $payload);
+
+ unlink($node->logfile);
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir,
+ '-l', $node->logfile, 'start');
+
+ if ($should_fail)
+ {
+ ok($ret != 0, "Cannot start postgres with faulty $file");
+ }
+ else
+ {
+ ok($ret == 0, "postgres can start with faulty $file");
+ }
+
+ my $log_contents = slurp_file($node->logfile);
+
+ foreach (@{$pattern})
+ {
+ like($log_contents,
+ $_,
+ "Expected failure found in the logs");
+ }
+
+ if (not $should_fail)
+ {
+ # We can't simply call $node->stop here as the call is optimized out
+ # when the server isn't started with $node->start.
+ my $ret =
+ PostgreSQL::Test::Utils::system_log('pg_ctl', '-D',
+ $data_dir, 'stop', '-m', 'fast');
+ ok($ret == 0, "Could stop postgres");
+ }
+}
+
+# We should be able to connect, and see an empty pg_ident.conf
+is($node->psql(
+ 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'),
+ qq(0),
+ 'pg_ident.conf is empty');
+
+############################################
+# part 1, test view reporting for valid data
+############################################
+my ($exp_hba, $exp_ident) = generate_valid_auth_files($node);
+
+$node->connect_ok('dbname=postgres', 'Connection still ok');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_hba_file_rules'),
+ qq($exp_hba),
+ 'pg_hba_file_rules content is expected');
+
+is($node->safe_psql(
+ 'postgres', 'SELECT * FROM pg_ident_file_mappings'),
+ qq($exp_ident),
+ 'pg_ident_file_mappings content is expected');
+
+#############################################
+# part 2, test log reporting for invalid data
+#############################################
+reset_auth_files($node);
+$node->restart('fast');
+$node->connect_ok('dbname=postgres',
+ 'Connection ok after resetting auth files');
+
+$node->stop('fast');
+
+start_errors_like($node, $hba_file, "include ../not_a_file",
+ [
+ qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file
+mkdir("$data_dir/hba_inc_fail");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject");
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token");
+start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail",
+ [
+ qr/invalid connection type "not_a_token"/,
+ qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/,
+ qr/could not load $data_dir\/$hba_file/
+ ], 1);
+
+# include_dir, single included file with nested inclusion
+unlink("$data_dir/hba_inc_fail/inc_dir.conf");
+my @hba_raw_errors_step1;
+
+add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1");
+
+add_hba_line($node, "hba_inc_fail/file1", "include file2");
+add_hba_line($node, "hba_inc_fail/file2", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file2", "include file3");
+
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+add_hba_line($node, "hba_inc_fail/file3", "local all all reject");
+push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3",
+ "local all all zuul",
+ 'invalid authentication method "zuul"');
+
+start_errors_like(
+ $node, $hba_file, "include_dir ../hba_inc_fail",
+ generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @hba_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local",
+ "end-of-line before database specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local,host",
+ "multiple values specified for connection type");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all",
+ "end-of-line before role specification");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "local all all",
+ "end-of-line before authentication method");
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ "host all all test/42",
+ 'specifying both host name and CIDR mask is invalid: "test/42"');
+push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf",
+ 'local @dbnames_fails.conf all reject',
+ "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory");
+
+add_hba_line($node, "hba_if_exists.conf", "include recurse.conf");
+push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf",
+ "include recurse.conf",
+ 'could not open configuration file "recurse.conf": maximum nesting depth exceeded');
+
+# Generate the regex for the expected errors in the logs. There's no guarantee
+# that the generated "line X of file..." will be emitted for the expected line,
+# but previous tests already ensured that the correct line number / file name
+# was emitted, so ensuring that there's an error in all expected lines is
+# enough here.
+my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2,
+ 1);
+
+# Not an error, but it should raise a message in the logs. Manually add an
+# extra log message to detect
+add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none");
+push @{$expected_errors},
+ qr/skipping missing authentication file "$data_dir\/if_exists_none"/;
+
+start_errors_like(
+ $node, $hba_file, "include_if_exists ../hba_if_exists.conf",
+ $expected_errors, 1);
+
+# Mostly the same, but for ident files
+reset_auth_files($node);
+
+my @ident_raw_errors_step1;
+
+# include_dir, single included file with nested inclusion
+mkdir("$data_dir/ident_inc_fail");
+add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1");
+
+add_ident_line($node, "ident_inc_fail/file1", "include file2");
+add_ident_line($node, "ident_inc_fail/file2", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file2", "include file3");
+
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+add_ident_line($node, "ident_inc_fail/file3", "ok ok ok");
+push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3",
+ "failmap /(fail postgres",
+ 'invalid regular expression "\(fail": parentheses \(\) not balanced');
+
+start_errors_like(
+ $node, $ident_file, "include_dir ../ident_inc_fail",
+ generate_log_err_patterns($node, \@ident_raw_errors_step1, 0),
+ 0);
+
+# start_errors_like will reset the main auth files, so the previous error won't
+# occur again. We keep it around as we will put back both bogus inclusions for
+# the tests at step 3.
+my @ident_raw_errors_step2;
+
+# include_if_exists, with various problems
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map",
+ "missing entry at end of line");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2",
+ "multiple values in ident field");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf",
+ 'map @osnames_fails.conf postgres',
+ "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory");
+
+add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf");
+push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf",
+ 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded');
+
+start_errors_like(
+ $node, $ident_file, "include_if_exists ../ident_if_exists.conf",
+ # There's no guarantee that the generated "line X of file..." will be
+ # emitted for the expected line, but previous tests already ensured that
+ # the correct line number / file name was emitted, so ensuring that there's
+ # an error in all expected lines is enough here.
+ generate_log_err_patterns($node, \@ident_raw_errors_step2, 0),
+ 0);
+
+#####################################################
+# part 3, test reporting of various error scenario
+# NOTE: this will be bypassed -DEXEC_BACKEND or win32
+#####################################################
+reset_auth_files($node);
+
+$node->start;
+$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset');
+
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_hba_file_rules');
+
+add_ident_line($node, $ident_file, '');
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'),
+ qq(0),
+ 'No error expected in pg_ident_file_mappings');
+
+# The instance could be restarted and no error is detected. Now check if the
+# build is compatible with the view error reporting (EXEC_BACKEND / win32 will
+# fail when trying to connect as they always rely on the current auth files
+# content)
+my @hba_raw_errors;
+
+push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+my ($stdout, $stderr);
+my $cmdret = $node->psql('postgres', 'SELECT 1',
+ stdout => \$stdout, stderr => \$stderr);
+
+if ($cmdret != 0)
+{
+ # Connection failed. Bail out, but make sure to raise a failure if it
+ # didn't fail for the expected hba file modification.
+ like($stderr,
+ qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/,
+ "Connection failed due to loading an invalid hba file");
+
+ done_testing();
+ diag("Build not compatible with auth file view error reporting, bail out.\n");
+ exit;
+}
+
+# Combine errors generated at step 2, in the same order.
+$node->append_conf($hba_file, "include_dir ../hba_inc_fail");
+push @hba_raw_errors, @hba_raw_errors_step1;
+
+$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf");
+push @hba_raw_errors, @hba_raw_errors_step2;
+
+my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules'
+ . ' WHERE error IS NOT NULL ORDER BY rule_number'),
+ qq($hba_expected),
+ 'Detected all error in hba file');
+
+# and do the same for pg_ident
+my @ident_raw_errors;
+
+push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file",
+ "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory");
+
+$node->append_conf($ident_file, "include_dir ../ident_inc_fail");
+push @ident_raw_errors, @ident_raw_errors_step1;
+
+$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf");
+push @ident_raw_errors, @ident_raw_errors_step2;
+
+my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors);
+is($node->safe_psql(
+ 'postgres',
+ 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings'
+ . ' WHERE error IS NOT NULL ORDER BY mapping_number'),
+ qq($ident_expected),
+ 'Detected all error in ident file');
+
+done_testing();
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 178e536e21..3d8f182674 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname,
FROM pg_authid
WHERE (NOT pg_authid.rolcanlogin);
pg_hba_file_rules| SELECT a.rule_number,
+ a.file_name,
a.line_number,
a.type,
a.database,
@@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number,
a.auth_method,
a.options,
a.error
- FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error);
+ FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error);
pg_ident_file_mappings| SELECT a.mapping_number,
+ a.file_name,
a.line_number,
a.map_name,
a.sys_name,
a.pg_username,
a.error
- FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error);
+ FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error);
pg_indexes| SELECT n.nspname AS schemaname,
c.relname AS tablename,
i.relname AS indexname,
--
2.37.0
--4aj2zfv7etn2tv3k--
^ permalink raw reply [nested|flat] 33+ messages in thread
end of thread, other threads:[~2022-10-25 06:26 UTC | newest]
Thread overview: 33+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-02-21 07:45 [PATCH v7 1/2] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-02-21 07:45 [PATCH v4 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-02-21 07:45 [PATCH v3 3/4] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-02-21 07:45 [PATCH v7 1/2] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-02-21 07:45 [PATCH v2 3/4] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-02-21 07:45 [PATCH v1 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-02-21 07:45 [PATCH v5 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-02-21 07:45 [PATCH v7 1/2] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-02-21 07:45 [PATCH v6 1/2] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-04-06 14:13 Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-04-07 18:34 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 00:36 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 13:18 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-07 19:12 ` Re: logical decoding and replication of sequences Noah Misch <[email protected]>
2022-08-07 21:09 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 06:15 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-08 08:33 ` Re: logical decoding and replication of sequences Kyotaro Horiguchi <[email protected]>
2022-08-08 08:56 ` Re: logical decoding and replication of sequences Kyotaro Horiguchi <[email protected]>
2022-08-29 10:21 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-29 18:04 ` Re: logical decoding and replication of sequences Tomas Vondra <[email protected]>
2022-08-29 21:42 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-23 04:21 ` Re: logical decoding and replication of sequences Michael Paquier <[email protected]>
2022-08-23 04:32 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-08-23 15:04 ` Re: logical decoding and replication of sequences Robert Haas <[email protected]>
2022-08-24 00:51 ` Re: logical decoding and replication of sequences Thomas Munro <[email protected]>
2022-05-25 20:42 ` Re: logical decoding and replication of sequences Zheng Li <[email protected]>
2022-05-30 03:15 [PATCH v12 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-05-30 03:15 [PATCH v9 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-05-30 03:15 [PATCH v10 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-05-30 03:15 [PATCH v10 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-05-30 03:15 [PATCH v11 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-05-30 03:15 [PATCH v8 5/6] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
2022-10-25 06:26 [PATCH v14 4/5] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox