public inbox for [email protected]help / color / mirror / Atom feed
Re: Continue work on changes to recovery.conf API 41+ messages / 6 participants [nested] [flat]
* Re: Continue work on changes to recovery.conf API @ 2018-10-31 17:53 Robert Haas <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Robert Haas @ 2018-10-31 17:53 UTC (permalink / raw) To: Peter Eisentraut <[email protected]>; +Cc: Sergei Kornilov <[email protected]>; pgsql-hackers On Fri, Sep 28, 2018 at 4:20 PM Peter Eisentraut <[email protected]> wrote: > > - recovery_target (immediate), recovery_target_name, recovery_target_time, recovery_target_xid, recovery_target_lsn are replaced to recovery_target_type and recovery_target_value (was discussed and changed in previous thread) > > I think this was the major point of contention. I reread the old > thread, and it's still not clear why we need to change this. _type and > _value look like an EAV system to me. GUC variables should be > verifiable independent of another variable. The other idea of using > only one variable seems better, but using two variables seems like a > poor compromise between one and five. +1. I like one best, I can live with five, and I think two stinks. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v7 1/2] Allow file inclusion in pg_hba and pg_ident files. @ 2022-02-21 07:45 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw) Catversion is bumped. Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/catalogs.sgml | 48 +++- doc/src/sgml/client-auth.sgml | 48 +++- src/backend/libpq/hba.c | 343 +++++++++++++++++++++---- src/backend/libpq/pg_hba.conf.sample | 10 +- src/backend/libpq/pg_ident.conf.sample | 12 +- src/backend/utils/adt/hbafuncs.c | 51 +++- src/include/catalog/pg_proc.dat | 11 +- src/include/libpq/hba.h | 2 + src/test/regress/expected/rules.out | 12 +- 9 files changed, 453 insertions(+), 84 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index a533a2153e..b44e62d388 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -10554,12 +10554,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>rule_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this rule if the rule is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule in the given file_name </para></entry> </row> @@ -10694,6 +10713,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>mapping_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this mapping if the mapping is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this mapping + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>line_number</structfield> <type>int4</type> + </para> + <para> + Line number of this mapping in the given file_name + </para></entry> + </row> <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index b2a459fb0d..f7d871e660 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,21 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> or <literal>include_dir</literal> directive and + the file or directory to be included. The file or directory can be a + relative of absolute path, and can be double quoted if needed. For the + <literal>include_dir</literal> form, all files not starting with a + <literal>.</literal> and ending with <literal>.conf</literal> will be + included. + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,6 +116,8 @@ <para> A record can have several formats: <synopsis> +include <replaceable>file</replaceable> +include_dir <replaceable>directory</replaceable> local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> @@ -118,6 +133,26 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_dir</literal></term> + <listitem> + <para> + This line will be replaced with the content of all the files found in + the directory, if they don't start with a <literal>.</literal> and end + with <literal>.conf</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -835,8 +870,10 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> +<replaceable>include_dir</replaceable> <replaceable>directory</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -847,6 +884,11 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + As for <filename>pg_hba.conf</filename>, the lines in this file can either + be inclusion directives or an authentication rule, and follow the same + rules. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index 327a4b42af..56b6cec9d5 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -21,6 +21,7 @@ #include <fcntl.h> #include <sys/param.h> #include <sys/socket.h> +#include <sys/stat.h> #include <netinet/in.h> #include <arpa/inet.h> #include <unistd.h> @@ -68,6 +69,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -112,10 +119,19 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, int elevel, char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); +static char *process_included_authfile(const char *inc_filename, + const char *outer_filename, int elevel, + MemoryContext linecxt, List **tok_lines); /* @@ -355,36 +371,11 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else - { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); - } + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename, + elevel, err_msg, &inc_fullname); - inc_file = AllocateFile(inc_fullname, "r"); if (inc_file == NULL) - { - int save_errno = errno; - - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); return tokens; - } /* There is possible recursion here if the file contains @ */ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); @@ -425,11 +416,36 @@ tokenize_inc_file(List *tokens, /* * tokenize_auth_file - * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a decicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel); + + return linecxt; +} + +/* + * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see the struct definition * in libpq/hba.h. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -438,30 +454,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -522,29 +530,151 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + AuthToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); + + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, filename, + elevel, linecxt, tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } + else if (strcmp(first->string, "include_dir") == 0) + { + char *dir_name; + DIR *d; + struct dirent *de; + StringInfoData err_buf; + + dir_name = second->string; + d = AllocateDir(dir_name); + if (d == NULL) + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not open directory \"%s\": %m", + dir_name))); + err_msg = psprintf("coud not open directory \"%s\": %m", + dir_name); + goto process_line; + } + + initStringInfo(&err_buf); + while ((de = ReadDir(d, dir_name)) != NULL) + { + struct stat st; + char inc_filename[MAXPGPATH]; + + /* + * Only parse files with names ending in ".conf". + * Explicitly reject files starting with ".". This + * excludes things like "." and "..", as well as typical + * hidden files, backup files, and editor debris. + */ + if (strlen(de->d_name) < 6) + continue; + if (de->d_name[0] == '.') + continue; + if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) + continue; + + join_path_components(inc_filename, dir_name, de->d_name); + canonicalize_path(inc_filename); + if (stat(inc_filename, &st) == 0) + { + /* Ignore directories. */ + if (S_ISDIR(st.st_mode)) + continue; + + /* + * err_msg is used here as a temp buffer, it will be + * overwritten at the end of the loop with the + * cumulated errors, if any. + */ + err_msg = process_included_authfile(inc_filename, + filename, elevel, linecxt, + tok_lines); + + /* Cumulate errors if any. */ + if (err_msg) + { + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfoString(&err_buf, err_msg); + } + } + else + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not stat file \"%s\": %m", + filename))); + + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfo(&err_buf, + "could not stat file \"%s\": %m", + inc_filename); + } + } + FreeDir(d); + + /* + * If there were no errors, the line is fully processed, bypass + * the general TokenizedAuthLine processing. + */ + if (err_buf.len == 0) + goto next_line; + + /* Otherwise, process the cumulated errors, if any. */ + err_msg = err_buf.data; + } } +process_line: + /* + * General processing: report the error if any and emit line to the + * TokenizedAuthLine + */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; + } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -859,7 +989,7 @@ do { \ errmsg("authentication option \"%s\" is only valid for authentication methods %s", \ optname, _(validmethods)), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \ optname, validmethods); \ return false; \ @@ -879,7 +1009,7 @@ do { \ errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname); \ return NULL; \ @@ -901,7 +1031,7 @@ do { \ ereport(elevel, \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("missing entry in file \"%s\" at end of line %d", \ - IdentFileName, line_num))); \ + tok_line->file_name, line_num))); \ *err_msg = psprintf("missing entry at end of line"); \ return NULL; \ } \ @@ -914,7 +1044,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("multiple values in ident field"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, tok_line->file_name))); \ *err_msg = psprintf("multiple values in ident field"); \ return NULL; \ } \ @@ -937,6 +1067,7 @@ HbaLine * parse_hba_line(TokenizedAuthLine *tok_line, int elevel) { int line_num = tok_line->line_num; + char *file_name = tok_line->file_name; char **err_msg = &tok_line->err_msg; char *str; struct addrinfo *gai_result; @@ -951,6 +1082,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -1675,6 +1807,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg) { int line_num = hbaline->linenumber; + char *file_name = hbaline->sourcefile; #ifdef USE_LDAP hbaline->ldapscope = LDAP_SCOPE_SUBTREE; @@ -2296,6 +2429,102 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} + +/* + * Try to open an included file, and tokenize it using the given context. + */ +static char * +process_included_authfile(const char *inc_filename, const char *outer_filename, + int elevel, MemoryContext linecxt, List **tok_lines) +{ + char *inc_fullname; + FILE *inc_file; + char *err_msg = NULL; + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, outer_filename, + elevel, &err_msg, &inc_fullname); + + if (inc_file == NULL) + { + /* open_inc_file should have reported an error. */ + Assert(err_msg != NULL); + return err_msg; + } + else + { + /* No error message should have been reported. */ + Assert(err_msg == NULL); + } + + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + return NULL; +} /* * Parse one tokenised line from the ident config file and store the result in diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..0050e24186 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,6 +9,8 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # +# include FILE +# include_dir DIRECTORY # local DATABASE USER METHOD [OPTIONS] # host DATABASE USER ADDRESS METHOD [OPTIONS] # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] @@ -18,7 +20,13 @@ # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include" or "include_dir", it's not a mapping record but a directive to +# include records from other file(s), as specified in the field. FILE is the +# file to include, and DIR is the directory containing the file(s) to include +# It can be specified with a relative or absolute path, and can be double +# quoted if it contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..9b700d9989 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,20 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # -# MAPNAME SYSTEM-USERNAME PG-USERNAME +# include FILE +# include_dir FILE +# MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include" or "include_dir", it's not an authentication +# record but a directive to include records from other file(s), specified in +# the field. FILE is the file to include and DIR is the directory containing +# the file(s) to include. It can be specified with a relative or absolute +# path, and can be double quoted if it contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index 9fe7b62c9a..4d902927e0 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,9 +26,11 @@ static ArrayType *get_hba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, + int mapping_number, const char *filename, int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); @@ -157,7 +159,7 @@ get_hba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 9 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line @@ -174,7 +176,8 @@ get_hba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg) + int rule_number, const char *filename, int lineno, HbaLine *hba, + const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; bool nulls[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -193,6 +196,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* rule_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -336,7 +346,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -359,6 +369,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *hba_lines = NIL; ListCell *line; + int rule_number = 0; MemoryContext linecxt; MemoryContext hbacxt; MemoryContext oldcxt; @@ -393,8 +404,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) hbaline = parse_hba_line(tok_line, DEBUG3); - fill_hba_line(tuple_store, tupdesc, tok_line->line_num, - hbaline, tok_line->err_msg); + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + rule_number++; + + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -430,8 +445,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) PG_RETURN_NULL(); } -/* Number of columns in pg_ident_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5 +/* Number of columns in pg_hba_file_mappings view */ +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -448,7 +463,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, IdentLine *ident, const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -461,6 +477,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* mapping_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -473,7 +496,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -495,6 +518,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *ident_lines = NIL; ListCell *line; + int mapping_number = 0; MemoryContext linecxt; MemoryContext identcxt; MemoryContext oldcxt; @@ -529,7 +553,12 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) identline = parse_ident_line(tok_line, DEBUG3); - fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline, + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + mapping_number++; + + fill_ident_line(tuple_store, tupdesc, mapping_number, + tok_line->file_name, tok_line->line_num, identline, tok_line->err_msg); } diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index babe16f00a..65bfd32753 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6128,15 +6128,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o}', - proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '6250', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,text,text,text}', proargmodes => '{o,o,o,o,o}', - proargnames => '{line_number,map_name,sys_name,pg_username,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index 90036f7bcd..59f6faf9f8 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -155,6 +156,7 @@ typedef struct AuthToken typedef struct TokenizedAuthLine { List *fields; /* List of lists of AuthTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 21effe8315..11ffe1743b 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname, WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); -pg_hba_file_rules| SELECT a.line_number, +pg_hba_file_rules| SELECT a.rule_number, + a.file_name, + a.line_number, a.type, a.database, a.user_name, @@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error); -pg_ident_file_mappings| SELECT a.line_number, + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); +pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, + a.line_number, a.map_name, a.sys_name, a.pg_username, a.error - FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_username, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.33.1 --o45jk7yo2ulu7ifa Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v7-0002-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v2 3/4] Allow file inclusion in pg_hba and pg_ident files. @ 2022-02-21 07:45 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw) Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/catalogs.sgml | 48 +++- doc/src/sgml/client-auth.sgml | 34 ++- src/backend/libpq/hba.c | 373 ++++++++++++++++--------- src/backend/libpq/pg_hba.conf.sample | 8 +- src/backend/libpq/pg_ident.conf.sample | 8 +- src/backend/utils/adt/hbafuncs.c | 53 +++- src/include/catalog/pg_proc.dat | 12 +- src/include/libpq/hba.h | 2 + src/test/regress/expected/rules.out | 12 +- 9 files changed, 395 insertions(+), 155 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 0de40a9626..07c6679a52 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -10430,12 +10430,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>rule_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this rule if the rule is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule in the given file_name </para></entry> </row> @@ -10571,6 +10590,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>mapping_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this mapping if the mapping is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this mapping + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>line_number</structfield> <type>int4</type> + </para> + <para> + Line number of this mapping in the given file_name + </para></entry> + </row> <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index 142b0affcb..e1d0e103b3 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,17 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> directive and the file to be included. The file + can be a relative of absolute path, and can be double quoted if needed. + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,6 +112,7 @@ <para> A record can have several formats: <synopsis> +include <replaceable>file</replaceable> local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> @@ -118,6 +128,15 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -835,8 +854,9 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -847,6 +867,14 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + The lines can record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> directive and the file to be included. The file + can be a relative of absolute path, and can be double quoted if needed. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index e673ff5474..85988fdeda 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -68,6 +68,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -112,6 +118,12 @@ static const char *const UserAuthName[] = }; +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); +static void tokenize_auth_file_with_context(MemoryContext Linecxt, + const char *filename, FILE *file, + List **tok_lines, int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, int elevel, char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, @@ -327,6 +339,68 @@ next_field_expand(const char *filename, char **lineptr, return tokens; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. +*/ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} + /* * tokenize_inc_file * Expand a file included from another file into an hba "field" @@ -355,36 +429,11 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else - { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); - } + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename, + elevel, err_msg, &inc_fullname); - inc_file = AllocateFile(inc_fullname, "r"); if (inc_file == NULL) - { - int save_errno = errno; - - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); return tokens; - } /* There is possible recursion here if the file contains @ */ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); @@ -423,6 +472,169 @@ tokenize_inc_file(List *tokens, return tokens; } +/* + * Tokenize the given file. + * + * The output is a list of TokenizedAuthLine structs; see struct definition + * above. + * + * linecxt: memory context which must contain all memory allocated by the + * function + * filename: the absolute path to the target file + * file: the already-opened target file + * tok_lines: receives output list + * elevel: message logging level + * + * Errors are reported by logging messages at ereport level elevel and by + * adding TokenizedAuthLine structs containing non-null err_msg fields to the + * output list. + * + */ +static void +tokenize_auth_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int elevel) +{ + StringInfoData buf; + int line_number = 1; + MemoryContext oldcxt; + + oldcxt = MemoryContextSwitchTo(linecxt); + + initStringInfo(&buf); + + while (!feof(file) && !ferror(file)) + { + TokenizedAuthLine *tok_line; + char *lineptr; + List *current_line = NIL; + char *err_msg = NULL; + int last_backslash_buflen = 0; + int continuations = 0; + + /* Collect the next input line, handling backslash continuations */ + resetStringInfo(&buf); + + while (pg_get_line_append(file, &buf, NULL)) + { + /* Strip trailing newline, including \r in case we're on Windows */ + buf.len = pg_strip_crlf(buf.data); + + /* + * Check for backslash continuation. The backslash must be after + * the last place we found a continuation, else two backslashes + * followed by two \n's would behave surprisingly. + */ + if (buf.len > last_backslash_buflen && + buf.data[buf.len - 1] == '\\') + { + /* Continuation, so strip it and keep reading */ + buf.data[--buf.len] = '\0'; + last_backslash_buflen = buf.len; + continuations++; + continue; + } + + /* Nope, so we have the whole line */ + break; + } + + if (ferror(file)) + { + /* I/O error! */ + int save_errno = errno; + + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not read file \"%s\": %m", filename))); + err_msg = psprintf("could not read file \"%s\": %s", + filename, strerror(save_errno)); + break; + } + + /* Parse fields */ + lineptr = buf.data; + while (*lineptr && err_msg == NULL) + { + List *current_field; + + current_field = next_field_expand(filename, &lineptr, + elevel, &err_msg); + /* add field to line, unless we are at EOL or comment start */ + if (current_field != NIL) + current_line = lappend(current_line, current_field); + } + + /* + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. + */ + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) + { + HbaToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); + + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; + char *inc_fullname; + FILE *inc_file; + + inc_filename = second->string; + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, + filename, elevel, &err_msg, + &inc_fullname); + + /* + * The included file could be open, now recursively process it. + * Errors will be reported in the general TokenizedAuthLine + * processing. + */ + if (inc_file != NULL) + { + tokenize_auth_file_with_context(linecxt, inc_fullname, + inc_file, tok_lines, + elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + /* + * The line is fulle processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + else + { + /* We should got an error */ + Assert(err_msg != NULL); + } + } + } + + /* General processing: Emit line to the TokenizedAuthLine */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: + line_number += continuations + 1; + + } + + MemoryContextSwitchTo(oldcxt); +} /* * Does user belong to role? @@ -1796,6 +2008,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(tok_line->file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -2602,118 +2815,26 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel) /* * Tokenize the given file. * - * The output is a list of TokenizedAuthLine structs; see struct definition - * above. + * Wrapper around tokenize_auth_file_with_context, creating a dedicated memory + * context. * - * filename: the absolute path to the target file - * file: the already-opened target file - * tok_lines: receives output list - * elevel: message logging level - * - * Errors are reported by logging messages at ereport level elevel and by - * adding TokenizedAuthLine structs containing non-null err_msg fields to the - * output list. - * - * Return value is a memory context which contains all memory allocated by + * Return value is this memory context which contains all memory allocated by * this function (it's a child of caller's context). */ MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel) +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, + int elevel) { - int line_number = 1; - StringInfoData buf; MemoryContext linecxt; - MemoryContext oldcxt; linecxt = AllocSetContextCreate(CurrentMemoryContext, "tokenize_auth_file", ALLOCSET_SMALL_SIZES); - oldcxt = MemoryContextSwitchTo(linecxt); - - initStringInfo(&buf); *tok_lines = NIL; - while (!feof(file) && !ferror(file)) - { - char *lineptr; - List *current_line = NIL; - char *err_msg = NULL; - int last_backslash_buflen = 0; - int continuations = 0; - - /* Collect the next input line, handling backslash continuations */ - resetStringInfo(&buf); - - while (pg_get_line_append(file, &buf, NULL)) - { - /* Strip trailing newline, including \r in case we're on Windows */ - buf.len = pg_strip_crlf(buf.data); - - /* - * Check for backslash continuation. The backslash must be after - * the last place we found a continuation, else two backslashes - * followed by two \n's would behave surprisingly. - */ - if (buf.len > last_backslash_buflen && - buf.data[buf.len - 1] == '\\') - { - /* Continuation, so strip it and keep reading */ - buf.data[--buf.len] = '\0'; - last_backslash_buflen = buf.len; - continuations++; - continue; - } - - /* Nope, so we have the whole line */ - break; - } - - if (ferror(file)) - { - /* I/O error! */ - int save_errno = errno; - - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not read file \"%s\": %m", filename))); - err_msg = psprintf("could not read file \"%s\": %s", - filename, strerror(save_errno)); - break; - } - - /* Parse fields */ - lineptr = buf.data; - while (*lineptr && err_msg == NULL) - { - List *current_field; - - current_field = next_field_expand(filename, &lineptr, - elevel, &err_msg); - /* add field to line, unless we are at EOL or comment start */ - if (current_field != NIL) - current_line = lappend(current_line, current_field); - } - - /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring - * */ - if (current_line != NIL || err_msg != NULL) - { - TokenizedAuthLine *tok_line; - - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); - } - - line_number += continuations + 1; - } - - MemoryContextSwitchTo(oldcxt); + tokenize_auth_file_with_context(linecxt, filename, file, tok_lines, + elevel); return linecxt; } diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..0b6589a7b9 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,6 +9,7 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # +# include FILE # local DATABASE USER METHOD [OPTIONS] # host DATABASE USER ADDRESS METHOD [OPTIONS] # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] @@ -18,7 +19,12 @@ # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include", it's not a mapping record but a directive to +# include records from another file, specified in the field. FILE is the file +# to include. It can be specified with a relative or absolute path, and can be +# double quoted if it contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..138359cf03 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,18 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # +# include FILE # MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include", it's not an authentication record but a +# directive to include records from another file, specified in the field. FILE +# is the file to include. It can be specified with a relative or absolute +# path, and can be double quoted if it contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index 2189c298b8..4c9df29427 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,16 +26,18 @@ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg); + int tule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, + int mappint_number, const char *filename, int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static ArrayType *gethba_options(HbaLine *hba); /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 9 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line: build one row of pg_hba_file_rules view, add it to tuplestore @@ -51,7 +53,8 @@ static ArrayType *gethba_options(HbaLine *hba); */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg) + int rule_number, const char *filename, int lineno, HbaLine *hba, + const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; bool nulls[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -70,6 +73,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* rule_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -213,7 +223,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -235,6 +245,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *hba_lines = NIL; ListCell *line; + int rule_number = 0; MemoryContext linecxt; MemoryContext hbacxt; MemoryContext oldcxt; @@ -265,12 +276,15 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) TokenizedAuthLine *tok_line = (TokenizedAuthLine *) lfirst(line); HbaLine *hbaline = NULL; - /* don't parse lines that already have errors */ + /* Only parse lines that don't already have an error */ if (tok_line->err_msg == NULL) + { hbaline = parse_hba_line(tok_line, DEBUG3); + rule_number++; + } - fill_hba_line(tuple_store, tupdesc, tok_line->line_num, - hbaline, tok_line->err_msg); + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -281,7 +295,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) } /* Number of columns in pg_hba_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5 +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -298,7 +312,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, IdentLine *ident, const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -311,6 +326,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* mapping_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -323,7 +345,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -345,6 +367,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *ident_lines = NIL; ListCell *line; + int mapping_number = 0; MemoryContext linecxt; MemoryContext identcxt; MemoryContext oldcxt; @@ -375,12 +398,16 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) TokenizedAuthLine *tok_line = (TokenizedAuthLine *) lfirst(line); IdentLine *identline = NULL; - /* don't parse lines that already have errors */ + /* Only parse lines that don't already have an error */ if (tok_line->err_msg == NULL) + { identline = parse_ident_line(tok_line, DEBUG3); + mapping_number++; + } - fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline, - tok_line->err_msg); + fill_ident_line(tuple_store, tupdesc, mapping_number, + tok_line->file_name, tok_line->line_num, identline, + tok_line->err_msg); } /* Free tokenizer memory */ diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 04d4edb228..e3baafc3e3 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6116,16 +6116,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o}', - proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '9556', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,text,text,text}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{line_number,map_name,sys_name,pg_usernamee,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_usernamee,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index fce7db248b..551e961585 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -153,6 +154,7 @@ typedef struct IdentLine typedef struct TokenizedAuthLine { List *fields; /* List of lists of HbaTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 62cf0d8674..e6f274cb59 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname, WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); -pg_hba_file_rules| SELECT a.line_number, +pg_hba_file_rules| SELECT a.rule_number, + a.file_name, + a.line_number, a.type, a.database, a.user_name, @@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error); -pg_ident_file_mappings| SELECT a.line_number, + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); +pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, + a.line_number, a.map_name, a.sys_name, a.pg_usernamee, a.error - FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_usernamee, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_usernamee, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.33.1 --uqjaptwdgaq4xou6 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v2-0004-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v5 2/3] Allow file inclusion in pg_hba and pg_ident files. @ 2022-02-21 07:45 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw) Catversion is bumped. Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/catalogs.sgml | 48 +++++- doc/src/sgml/client-auth.sgml | 34 +++- src/backend/libpq/hba.c | 229 +++++++++++++++++++------ src/backend/libpq/pg_hba.conf.sample | 8 +- src/backend/libpq/pg_ident.conf.sample | 8 +- src/backend/utils/adt/hbafuncs.c | 53 ++++-- src/include/catalog/pg_proc.dat | 12 +- src/include/libpq/hba.h | 2 + src/test/regress/expected/rules.out | 12 +- 9 files changed, 321 insertions(+), 85 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 75fedfa07e..bd1c9a8d21 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -10496,12 +10496,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>rule_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this rule if the rule is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule in the given file_name </para></entry> </row> @@ -10637,6 +10656,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>mapping_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this mapping if the mapping is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this mapping + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>line_number</structfield> <type>int4</type> + </para> + <para> + Line number of this mapping in the given file_name + </para></entry> + </row> <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index 142b0affcb..e1d0e103b3 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,17 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> directive and the file to be included. The file + can be a relative of absolute path, and can be double quoted if needed. + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,6 +112,7 @@ <para> A record can have several formats: <synopsis> +include <replaceable>file</replaceable> local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> @@ -118,6 +128,15 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -835,8 +854,9 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -847,6 +867,14 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + The lines can record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> directive and the file to be included. The file + can be a relative of absolute path, and can be double quoted if needed. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index 556f473b41..1551b34c53 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -68,6 +68,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -112,10 +118,16 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, int elevel, char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); /* @@ -355,36 +367,11 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else - { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); - } + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename, + elevel, err_msg, &inc_fullname); - inc_file = AllocateFile(inc_fullname, "r"); if (inc_file == NULL) - { - int save_errno = errno; - - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); return tokens; - } /* There is possible recursion here if the file contains @ */ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); @@ -425,11 +412,36 @@ tokenize_inc_file(List *tokens, /* * tokenize_auth_file - * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a decicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel); + + return linecxt; +} + +/* + * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see the struct definition * in libpq/hba.h. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -438,30 +450,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -522,29 +526,76 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + AuthToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; + char *inc_fullname; + FILE *inc_file; + + inc_filename = second->string; + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, + filename, elevel, &err_msg, + &inc_fullname); + + /* + * The included file could be open, now recursively process it. + * Errors will be reported in the general TokenizedAuthLine + * processing. + */ + if (inc_file != NULL) + { + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + else + { + /* We should got an error */ + Assert(err_msg != NULL); + } + } } + /* General processing: emit line to the TokenizedAuthLine */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; + } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -859,7 +910,7 @@ do { \ errmsg("authentication option \"%s\" is only valid for authentication methods %s", \ optname, _(validmethods)), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \ optname, validmethods); \ return false; \ @@ -879,7 +930,7 @@ do { \ errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname); \ return NULL; \ @@ -901,7 +952,7 @@ do { \ ereport(elevel, \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("missing entry in file \"%s\" at end of line %d", \ - IdentFileName, line_num))); \ + tok_line->file_name, line_num))); \ *err_msg = psprintf("missing entry at end of line"); \ return NULL; \ } \ @@ -914,7 +965,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("multiple values in ident field"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, tok_line->file_name))); \ *err_msg = psprintf("multiple values in ident field"); \ return NULL; \ } \ @@ -937,6 +988,7 @@ HbaLine * parse_hba_line(TokenizedAuthLine *tok_line, int elevel) { int line_num = tok_line->line_num; + char *file_name = tok_line->file_name; char **err_msg = &tok_line->err_msg; char *str; struct addrinfo *gai_result; @@ -951,6 +1003,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -1677,6 +1730,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg) { int line_num = hbaline->linenumber; + char *file_name = hbaline->sourcefile; #ifdef USE_LDAP hbaline->ldapscope = LDAP_SCOPE_SUBTREE; @@ -2299,6 +2353,67 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} /* * Parse one tokenised line from the ident config file and store the result in diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..0b6589a7b9 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,6 +9,7 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # +# include FILE # local DATABASE USER METHOD [OPTIONS] # host DATABASE USER ADDRESS METHOD [OPTIONS] # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] @@ -18,7 +19,12 @@ # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include", it's not a mapping record but a directive to +# include records from another file, specified in the field. FILE is the file +# to include. It can be specified with a relative or absolute path, and can be +# double quoted if it contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..138359cf03 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,18 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # +# include FILE # MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include", it's not an authentication record but a +# directive to include records from another file, specified in the field. FILE +# is the file to include. It can be specified with a relative or absolute +# path, and can be double quoted if it contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index 1970b4c497..33cf5fb954 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,9 +26,11 @@ static ArrayType *get_hba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, + int mapping_number, const char *filename, int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); @@ -157,7 +159,7 @@ get_hba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 9 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line @@ -174,7 +176,8 @@ get_hba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg) + int rule_number, const char *filename, int lineno, HbaLine *hba, + const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; bool nulls[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -193,6 +196,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* rule_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -336,7 +346,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -359,6 +369,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *hba_lines = NIL; ListCell *line; + int rule_number = 0; MemoryContext linecxt; MemoryContext hbacxt; MemoryContext oldcxt; @@ -393,8 +404,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) hbaline = parse_hba_line(tok_line, DEBUG3); - fill_hba_line(tuple_store, tupdesc, tok_line->line_num, - hbaline, tok_line->err_msg); + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + rule_number++; + + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -430,8 +445,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) PG_RETURN_NULL(); } -/* Number of columns in pg_ident_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5 +/* Number of columns in pg_hba_file_mappings view */ +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -448,7 +463,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, IdentLine *ident, const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -461,6 +477,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* mapping_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -473,7 +496,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -495,6 +518,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *ident_lines = NIL; ListCell *line; + int mapping_number = 0; MemoryContext linecxt; MemoryContext identcxt; MemoryContext oldcxt; @@ -529,8 +553,13 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) identline = parse_ident_line(tok_line, DEBUG3); - fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline, - tok_line->err_msg); + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + mapping_number++; + + fill_ident_line(tuple_store, tupdesc, mapping_number, + tok_line->file_name, tok_line->line_num, identline, + tok_line->err_msg); } /* Free tokenizer memory */ diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index efd48741d8..04cab629f5 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6111,16 +6111,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o}', - proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '9556', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,text,text,text}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{line_number,map_name,sys_name,pg_username,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index 90036f7bcd..59f6faf9f8 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -155,6 +156,7 @@ typedef struct AuthToken typedef struct TokenizedAuthLine { List *fields; /* List of lists of AuthTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 5a20e5a7e1..e2c6aa8ffb 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname, WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); -pg_hba_file_rules| SELECT a.line_number, +pg_hba_file_rules| SELECT a.rule_number, + a.file_name, + a.line_number, a.type, a.database, a.user_name, @@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error); -pg_ident_file_mappings| SELECT a.line_number, + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); +pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, + a.line_number, a.map_name, a.sys_name, a.pg_username, a.error - FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_username, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.33.1 --w64sw4oo7jkfb67p Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v5-0003-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v1 2/3] Allow file inclusion in pg_hba and pg_ident files. @ 2022-02-21 07:45 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw) Author: Julien Rouhaud Reviewed-by: FIXME Discussion: FIXME --- doc/src/sgml/catalogs.sgml | 48 ++++- doc/src/sgml/client-auth.sgml | 34 +++- src/backend/libpq/hba.c | 252 +++++++++++++++++++------ src/backend/libpq/pg_hba.conf.sample | 8 +- src/backend/libpq/pg_ident.conf.sample | 8 +- src/include/catalog/pg_proc.dat | 12 +- src/include/libpq/hba.h | 1 + src/test/regress/expected/rules.out | 12 +- 8 files changed, 304 insertions(+), 71 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 0de40a9626..07c6679a52 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -10430,12 +10430,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>rule_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this rule if the rule is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule in the given file_name </para></entry> </row> @@ -10571,6 +10590,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>mapping_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this mapping if the mapping is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this mapping + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>line_number</structfield> <type>int4</type> + </para> + <para> + Line number of this mapping in the given file_name + </para></entry> + </row> <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index 142b0affcb..e1d0e103b3 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,17 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> directive and the file to be included. The file + can be a relative of absolute path, and can be double quoted if needed. + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,6 +112,7 @@ <para> A record can have several formats: <synopsis> +include <replaceable>file</replaceable> local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> @@ -118,6 +128,15 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -835,8 +854,9 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -847,6 +867,14 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + The lines can record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> directive and the file to be included. The file + can be a relative of absolute path, and can be double quoted if needed. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index 9a9ce96fee..8b72141342 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -89,11 +89,18 @@ typedef struct HbaToken typedef struct TokenizedLine { List *fields; /* List of lists of HbaTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ } TokenizedLine; +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -140,17 +147,25 @@ static const char *const UserAuthName[] = static MemoryContext tokenize_file(const char *filename, FILE *file, List **tok_lines, int elevel); +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, int elevel, char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); static ArrayType *gethba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, + int mapping_number, const char *filename, int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); /* @@ -390,36 +405,11 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else - { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); - } + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename, + elevel, err_msg, &inc_fullname); - inc_file = AllocateFile(inc_fullname, "r"); if (inc_file == NULL) - { - int save_errno = errno; - - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); return tokens; - } /* There is possible recursion here if the file contains @ */ linecxt = tokenize_file(inc_fullname, inc_file, &inc_lines, elevel); @@ -458,11 +448,37 @@ tokenize_inc_file(List *tokens, return tokens; } +/* + * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a decicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +static MemoryContext +tokenize_file(const char *filename, FILE *file, List **tok_lines, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel); + + return linecxt; +} + /* * Tokenize the given file. * * The output is a list of TokenizedLine structs; see struct definition above. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -471,27 +487,19 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -static MemoryContext -tokenize_file(const char *filename, FILE *file, List **tok_lines, int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { char *lineptr; @@ -553,28 +561,76 @@ tokenize_file(const char *filename, FILE *file, List **tok_lines, int elevel) current_line = lappend(current_line, current_field); } - /* Reached EOL; emit line to TokenizedLine list unless it's boring */ - if (current_line != NIL || err_msg != NULL) + /* + * Reached EOL; no need to emit line to TokenizedLine list if it's + * boring + */ + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) + { + HbaToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); + + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; + char *inc_fullname; + FILE *inc_file; + + inc_filename = second->string; + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, + filename, elevel, &err_msg, + &inc_fullname); + + /* + * We could open the file, recursively process it. Errors will + * be reported in the general TokenizedLine processing. + */ + if (inc_file != NULL) + { + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + goto next_line; + } + else + { + /* We should got an error */ + Assert(err_msg != NULL); + } + } + } + + /* Emit line to TokenizedLine */ { TokenizedLine *tok_line; tok_line = (TokenizedLine *) palloc(sizeof(TokenizedLine)); tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); tok_line->line_num = line_number; tok_line->raw_line = pstrdup(buf.data); tok_line->err_msg = err_msg; *tok_lines = lappend(*tok_lines, tok_line); } +next_line: line_number += continuations + 1; + } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -981,6 +1037,7 @@ parse_hba_line(TokenizedLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(tok_line->file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -2453,7 +2510,7 @@ gethba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 9 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line: build one row of pg_hba_file_rules view, add it to tuplestore @@ -2469,7 +2526,8 @@ gethba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg) + int rule_number, const char *filename, int lineno, HbaLine *hba, + const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; bool nulls[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -2488,6 +2546,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* rule_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -2631,7 +2696,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -2653,6 +2718,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *hba_lines = NIL; ListCell *line; + int rule_number = 0; MemoryContext linecxt; MemoryContext hbacxt; MemoryContext oldcxt; @@ -2687,8 +2753,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) hbaline = parse_hba_line(tok_line, DEBUG3); - fill_hba_line(tuple_store, tupdesc, tok_line->line_num, - hbaline, tok_line->err_msg); + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + rule_number++; + + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -2698,6 +2768,64 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) MemoryContextDelete(hbacxt); } +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} + /* * SQL-accessible SRF to return all the entries in the pg_hba.conf file. */ @@ -3172,7 +3300,7 @@ hba_authname(UserAuth auth_method) } /* Number of columns in pg_hba_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5 +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -3189,7 +3317,8 @@ hba_authname(UserAuth auth_method) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, IdentLine *ident, const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -3202,6 +3331,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* mapping_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -3214,7 +3350,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -3236,6 +3372,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *ident_lines = NIL; ListCell *line; + int mapping_number = 0; MemoryContext linecxt; MemoryContext hbacxt; MemoryContext oldcxt; @@ -3270,8 +3407,13 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) identline = parse_ident_line(tok_line, DEBUG3); - fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline, - tok_line->err_msg); + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + mapping_number++; + + fill_ident_line(tuple_store, tupdesc, mapping_number, + tok_line->file_name, tok_line->line_num, identline, + tok_line->err_msg); } /* Free tokenizer memory */ diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..0b6589a7b9 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,6 +9,7 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # +# include FILE # local DATABASE USER METHOD [OPTIONS] # host DATABASE USER ADDRESS METHOD [OPTIONS] # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] @@ -18,7 +19,12 @@ # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include", it's not a mapping record but a directive to +# include records from another file, specified in the field. FILE is the file +# to include. It can be specified with a relative or absolute path, and can be +# double quoted if it contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..138359cf03 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,18 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # +# include FILE # MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include", it's not an authentication record but a +# directive to include records from another file, specified in the field. FILE +# is the file to include. It can be specified with a relative or absolute +# path, and can be double quoted if it contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 2c8f5d9c13..2292115c85 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6117,16 +6117,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o}', - proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '9556', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,text,text,text}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{line_number,map_name,sys_name,pg_usernamee,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_usernamee,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index 8d9f3821b1..45d6ce1f22 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 62cf0d8674..e6f274cb59 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname, WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); -pg_hba_file_rules| SELECT a.line_number, +pg_hba_file_rules| SELECT a.rule_number, + a.file_name, + a.line_number, a.type, a.database, a.user_name, @@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error); -pg_ident_file_mappings| SELECT a.line_number, + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); +pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, + a.line_number, a.map_name, a.sys_name, a.pg_usernamee, a.error - FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_usernamee, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_usernamee, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.33.1 --j3tmyqztg67irqsk Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v1-0003-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v3 3/4] Allow file inclusion in pg_hba and pg_ident files. @ 2022-02-21 07:45 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw) Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/catalogs.sgml | 48 +++++- doc/src/sgml/client-auth.sgml | 34 +++- src/backend/libpq/hba.c | 218 +++++++++++++++++++------ src/backend/libpq/pg_hba.conf.sample | 8 +- src/backend/libpq/pg_ident.conf.sample | 8 +- src/backend/utils/adt/hbafuncs.c | 51 ++++-- src/include/catalog/pg_proc.dat | 12 +- src/include/libpq/hba.h | 2 + src/test/regress/expected/rules.out | 12 +- 9 files changed, 314 insertions(+), 79 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index e87f0543d6..c6c5e7d08e 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -10440,12 +10440,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>rule_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this rule if the rule is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule in the given file_name </para></entry> </row> @@ -10581,6 +10600,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>mapping_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this mapping if the mapping is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this mapping + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>line_number</structfield> <type>int4</type> + </para> + <para> + Line number of this mapping in the given file_name + </para></entry> + </row> <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index 142b0affcb..e1d0e103b3 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,17 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> directive and the file to be included. The file + can be a relative of absolute path, and can be double quoted if needed. + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,6 +112,7 @@ <para> A record can have several formats: <synopsis> +include <replaceable>file</replaceable> local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> @@ -118,6 +128,15 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -835,8 +854,9 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -847,6 +867,14 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + The lines can record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> directive and the file to be included. The file + can be a relative of absolute path, and can be double quoted if needed. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index 68136f6244..1bfcaef025 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -68,6 +68,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -112,10 +118,16 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, int elevel, char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); /* @@ -355,36 +367,11 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else - { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); - } + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename, + elevel, err_msg, &inc_fullname); - inc_file = AllocateFile(inc_fullname, "r"); if (inc_file == NULL) - { - int save_errno = errno; - - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); return tokens; - } /* There is possible recursion here if the file contains @ */ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); @@ -423,12 +410,38 @@ tokenize_inc_file(List *tokens, return tokens; } +/* + * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a decicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel); + + return linecxt; +} + /* * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see struct definition * above. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -437,30 +450,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -521,29 +526,76 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + HbaToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; + char *inc_fullname; + FILE *inc_file; + + inc_filename = second->string; + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, + filename, elevel, &err_msg, + &inc_fullname); + + /* + * The included file could be open, now recursively process it. + * Errors will be reported in the general TokenizedAuthLine + * processing. + */ + if (inc_file != NULL) + { + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + else + { + /* We should got an error */ + Assert(err_msg != NULL); + } + } } + /* General processing: emit line to the TokenizedAuthLine */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; + } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -950,6 +1002,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(tok_line->file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -2298,6 +2351,67 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} /* * Parse one tokenised line from the ident config file and store the result in diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..0b6589a7b9 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,6 +9,7 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # +# include FILE # local DATABASE USER METHOD [OPTIONS] # host DATABASE USER ADDRESS METHOD [OPTIONS] # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] @@ -18,7 +19,12 @@ # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include", it's not a mapping record but a directive to +# include records from another file, specified in the field. FILE is the file +# to include. It can be specified with a relative or absolute path, and can be +# double quoted if it contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..138359cf03 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,18 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # +# include FILE # MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include", it's not an authentication record but a +# directive to include records from another file, specified in the field. FILE +# is the file to include. It can be specified with a relative or absolute +# path, and can be double quoted if it contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index 75e69383c2..ee12dc1893 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,9 +26,11 @@ static ArrayType *gethba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, + int mapping_number, const char *filename, int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); @@ -157,7 +159,7 @@ gethba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 9 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line: build one row of pg_hba_file_rules view, add it to tuplestore @@ -173,7 +175,8 @@ gethba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg) + int rule_number, const char *filename, int lineno, HbaLine *hba, + const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; bool nulls[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -192,6 +195,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* rule_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -335,7 +345,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -357,6 +367,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *hba_lines = NIL; ListCell *line; + int rule_number = 0; MemoryContext linecxt; MemoryContext hbacxt; MemoryContext oldcxt; @@ -391,8 +402,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) hbaline = parse_hba_line(tok_line, DEBUG3); - fill_hba_line(tuple_store, tupdesc, tok_line->line_num, - hbaline, tok_line->err_msg); + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + rule_number++; + + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -426,7 +441,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) } /* Number of columns in pg_hba_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5 +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -443,7 +458,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, IdentLine *ident, const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -456,6 +472,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* mapping_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -468,7 +491,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -490,6 +513,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *ident_lines = NIL; ListCell *line; + int mapping_number = 0; MemoryContext linecxt; MemoryContext identcxt; MemoryContext oldcxt; @@ -524,8 +548,13 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) identline = parse_ident_line(tok_line, DEBUG3); - fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline, - tok_line->err_msg); + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + mapping_number++; + + fill_ident_line(tuple_store, tupdesc, mapping_number, + tok_line->file_name, tok_line->line_num, identline, + tok_line->err_msg); } /* Free tokenizer memory */ diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 6ccbc9af4c..0e8a589302 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6111,16 +6111,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o}', - proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '9556', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,text,text,text}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{line_number,map_name,sys_name,pg_usernamee,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_usernamee,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index fce7db248b..551e961585 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -153,6 +154,7 @@ typedef struct IdentLine typedef struct TokenizedAuthLine { List *fields; /* List of lists of HbaTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 76a209b717..5b7a4c01ee 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname, WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); -pg_hba_file_rules| SELECT a.line_number, +pg_hba_file_rules| SELECT a.rule_number, + a.file_name, + a.line_number, a.type, a.database, a.user_name, @@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error); -pg_ident_file_mappings| SELECT a.line_number, + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); +pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, + a.line_number, a.map_name, a.sys_name, a.pg_usernamee, a.error - FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_usernamee, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_usernamee, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.33.1 --45a4to7ul53bhaf6 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v3-0004-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v6 1/2] Allow file inclusion in pg_hba and pg_ident files. @ 2022-02-21 07:45 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw) Catversion is bumped. Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/catalogs.sgml | 48 +++- doc/src/sgml/client-auth.sgml | 48 +++- src/backend/libpq/hba.c | 343 +++++++++++++++++++++---- src/backend/libpq/pg_hba.conf.sample | 10 +- src/backend/libpq/pg_ident.conf.sample | 12 +- src/backend/utils/adt/hbafuncs.c | 51 +++- src/include/catalog/pg_proc.dat | 11 +- src/include/libpq/hba.h | 2 + src/test/regress/expected/rules.out | 12 +- 9 files changed, 453 insertions(+), 84 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 7f4f79d1b5..2b5b7ef5d6 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -10496,12 +10496,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>rule_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this rule if the rule is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule in the given file_name </para></entry> </row> @@ -10636,6 +10655,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>mapping_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this mapping if the mapping is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this mapping + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>line_number</structfield> <type>int4</type> + </para> + <para> + Line number of this mapping in the given file_name + </para></entry> + </row> <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index 142b0affcb..4e1438476e 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,21 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> or <literal>include_dir</literal> directive and + the file or directory to be included. The file or directory can be a + relative of absolute path, and can be double quoted if needed. For the + <literal>include_dir</literal> form, all files not starting with a + <literal>.</literal> and ending with <literal>.conf</literal> will be + included. + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,6 +116,8 @@ <para> A record can have several formats: <synopsis> +include <replaceable>file</replaceable> +include_dir <replaceable>directory</replaceable> local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> @@ -118,6 +133,26 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_dir</literal></term> + <listitem> + <para> + This line will be replaced with the content of all the files found in + the directory, if they don't start with a <literal>.</literal> and end + with <literal>.conf</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -835,8 +870,10 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> +<replaceable>include_dir</replaceable> <replaceable>directory</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -847,6 +884,11 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + As for <filename>pg_hba.conf</filename>, the lines in this file can either + be inclusion directives or an authentication rules, and follow the same + rules. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index f8393ca8ed..5d0caa587b 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -21,6 +21,7 @@ #include <fcntl.h> #include <sys/param.h> #include <sys/socket.h> +#include <sys/stat.h> #include <netinet/in.h> #include <arpa/inet.h> #include <unistd.h> @@ -68,6 +69,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -112,10 +119,19 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, int elevel, char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); +static char *process_included_authfile(const char *inc_filename, + const char *outer_filename, int elevel, + MemoryContext linecxt, List **tok_lines); /* @@ -355,36 +371,11 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else - { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); - } + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename, + elevel, err_msg, &inc_fullname); - inc_file = AllocateFile(inc_fullname, "r"); if (inc_file == NULL) - { - int save_errno = errno; - - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); return tokens; - } /* There is possible recursion here if the file contains @ */ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); @@ -425,11 +416,36 @@ tokenize_inc_file(List *tokens, /* * tokenize_auth_file - * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a decicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel); + + return linecxt; +} + +/* + * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see the struct definition * in libpq/hba.h. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -438,30 +454,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -522,29 +530,151 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + AuthToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); + + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, filename, + elevel, linecxt, tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } + else if (strcmp(first->string, "include_dir") == 0) + { + char *dir_name; + DIR *d; + struct dirent *de; + StringInfoData err_buf; + + dir_name = second->string; + d = AllocateDir(dir_name); + if (d == NULL) + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not open directory \"%s\": %m", + dir_name))); + err_msg = psprintf("coud not open directory \"%s\": %m", + dir_name); + goto process_line; + } + + initStringInfo(&err_buf); + while ((de = ReadDir(d, dir_name)) != NULL) + { + struct stat st; + char inc_filename[MAXPGPATH]; + + /* + * Only parse files with names ending in ".conf". + * Explicitly reject files starting with ".". This + * excludes things like "." and "..", as well as typical + * hidden files, backup files, and editor debris. + */ + if (strlen(de->d_name) < 6) + continue; + if (de->d_name[0] == '.') + continue; + if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) + continue; + + join_path_components(inc_filename, dir_name, de->d_name); + canonicalize_path(inc_filename); + if (stat(inc_filename, &st) == 0) + { + /* Ignore directories. */ + if (S_ISDIR(st.st_mode)) + continue; + + /* + * err_msg is used here as a temp buffer, it will be + * overwritten at the end of the loop with the + * cumulated errors, if any. + */ + err_msg = process_included_authfile(inc_filename, + filename, elevel, linecxt, + tok_lines); + + /* Cumulate errors if any. */ + if (err_msg) + { + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfoString(&err_buf, err_msg); + } + } + else + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not stat file \"%s\": %m", + filename))); + + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfo(&err_buf, + "could not stat file \"%s\": %m", + inc_filename); + } + } + FreeDir(d); + + /* + * If there were no errors, the line is fully processed, bypass + * the general TokenizedAuthLine processing. + */ + if (err_buf.len == 0) + goto next_line; + + /* Otherwise, process the cumulated errors, if any. */ + err_msg = err_buf.data; + } } +process_line: + /* + * General processing: report the error if any and emit line to the + * TokenizedAuthLine + */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; + } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -859,7 +989,7 @@ do { \ errmsg("authentication option \"%s\" is only valid for authentication methods %s", \ optname, _(validmethods)), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \ optname, validmethods); \ return false; \ @@ -879,7 +1009,7 @@ do { \ errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname); \ return NULL; \ @@ -901,7 +1031,7 @@ do { \ ereport(elevel, \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("missing entry in file \"%s\" at end of line %d", \ - IdentFileName, line_num))); \ + tok_line->file_name, line_num))); \ *err_msg = psprintf("missing entry at end of line"); \ return NULL; \ } \ @@ -914,7 +1044,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("multiple values in ident field"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, tok_line->file_name))); \ *err_msg = psprintf("multiple values in ident field"); \ return NULL; \ } \ @@ -937,6 +1067,7 @@ HbaLine * parse_hba_line(TokenizedAuthLine *tok_line, int elevel) { int line_num = tok_line->line_num; + char *file_name = tok_line->file_name; char **err_msg = &tok_line->err_msg; char *str; struct addrinfo *gai_result; @@ -951,6 +1082,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -1677,6 +1809,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg) { int line_num = hbaline->linenumber; + char *file_name = hbaline->sourcefile; #ifdef USE_LDAP hbaline->ldapscope = LDAP_SCOPE_SUBTREE; @@ -2299,6 +2432,102 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} + +/* + * Try to open an included file, and tokenize it using the given context. + */ +static char * +process_included_authfile(const char *inc_filename, const char *outer_filename, + int elevel, MemoryContext linecxt, List **tok_lines) +{ + char *inc_fullname; + FILE *inc_file; + char *err_msg = NULL; + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, outer_filename, + elevel, &err_msg, &inc_fullname); + + if (inc_file == NULL) + { + /* open_inc_file should have reported an error. */ + Assert(err_msg != NULL); + return err_msg; + } + else + { + /* No error message should have been reported. */ + Assert(err_msg == NULL); + } + + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + return NULL; +} /* * Parse one tokenised line from the ident config file and store the result in diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..0050e24186 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,6 +9,8 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # +# include FILE +# include_dir DIRECTORY # local DATABASE USER METHOD [OPTIONS] # host DATABASE USER ADDRESS METHOD [OPTIONS] # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] @@ -18,7 +20,13 @@ # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include" or "include_dir", it's not a mapping record but a directive to +# include records from other file(s), as specified in the field. FILE is the +# file to include, and DIR is the directory containing the file(s) to include +# It can be specified with a relative or absolute path, and can be double +# quoted if it contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..9b700d9989 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,20 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # -# MAPNAME SYSTEM-USERNAME PG-USERNAME +# include FILE +# include_dir FILE +# MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include" or "include_dir", it's not an authentication +# record but a directive to include records from other file(s), specified in +# the field. FILE is the file to include and DIR is the directory containing +# the file(s) to include. It can be specified with a relative or absolute +# path, and can be double quoted if it contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index 9fe7b62c9a..4d902927e0 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,9 +26,11 @@ static ArrayType *get_hba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, + int mapping_number, const char *filename, int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); @@ -157,7 +159,7 @@ get_hba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 9 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line @@ -174,7 +176,8 @@ get_hba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg) + int rule_number, const char *filename, int lineno, HbaLine *hba, + const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; bool nulls[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -193,6 +196,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* rule_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -336,7 +346,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -359,6 +369,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *hba_lines = NIL; ListCell *line; + int rule_number = 0; MemoryContext linecxt; MemoryContext hbacxt; MemoryContext oldcxt; @@ -393,8 +404,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) hbaline = parse_hba_line(tok_line, DEBUG3); - fill_hba_line(tuple_store, tupdesc, tok_line->line_num, - hbaline, tok_line->err_msg); + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + rule_number++; + + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -430,8 +445,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) PG_RETURN_NULL(); } -/* Number of columns in pg_ident_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5 +/* Number of columns in pg_hba_file_mappings view */ +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -448,7 +463,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, IdentLine *ident, const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -461,6 +477,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* mapping_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -473,7 +496,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -495,6 +518,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *ident_lines = NIL; ListCell *line; + int mapping_number = 0; MemoryContext linecxt; MemoryContext identcxt; MemoryContext oldcxt; @@ -529,7 +553,12 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) identline = parse_ident_line(tok_line, DEBUG3); - fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline, + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + mapping_number++; + + fill_ident_line(tuple_store, tupdesc, mapping_number, + tok_line->file_name, tok_line->line_num, identline, tok_line->err_msg); } diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 01e1dd4d6d..467f6f1293 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6111,15 +6111,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o}', - proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '9556', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,text,text,text}', proargmodes => '{o,o,o,o,o}', - proargnames => '{line_number,map_name,sys_name,pg_username,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index 90036f7bcd..59f6faf9f8 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -155,6 +156,7 @@ typedef struct AuthToken typedef struct TokenizedAuthLine { List *fields; /* List of lists of AuthTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 423b9b99fb..e00185eb6d 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname, WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); -pg_hba_file_rules| SELECT a.line_number, +pg_hba_file_rules| SELECT a.rule_number, + a.file_name, + a.line_number, a.type, a.database, a.user_name, @@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error); -pg_ident_file_mappings| SELECT a.line_number, + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); +pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, + a.line_number, a.map_name, a.sys_name, a.pg_username, a.error - FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_username, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.33.1 --g2rofdrijxvb5udh Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v6-0002-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v7 1/2] Allow file inclusion in pg_hba and pg_ident files. @ 2022-02-21 07:45 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw) Catversion is bumped. Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/catalogs.sgml | 48 +++- doc/src/sgml/client-auth.sgml | 48 +++- src/backend/libpq/hba.c | 343 +++++++++++++++++++++---- src/backend/libpq/pg_hba.conf.sample | 10 +- src/backend/libpq/pg_ident.conf.sample | 12 +- src/backend/utils/adt/hbafuncs.c | 51 +++- src/include/catalog/pg_proc.dat | 11 +- src/include/libpq/hba.h | 2 + src/test/regress/expected/rules.out | 12 +- 9 files changed, 453 insertions(+), 84 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index a533a2153e..b44e62d388 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -10554,12 +10554,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>rule_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this rule if the rule is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule in the given file_name </para></entry> </row> @@ -10694,6 +10713,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>mapping_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this mapping if the mapping is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this mapping + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>line_number</structfield> <type>int4</type> + </para> + <para> + Line number of this mapping in the given file_name + </para></entry> + </row> <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index b2a459fb0d..f7d871e660 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,21 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> or <literal>include_dir</literal> directive and + the file or directory to be included. The file or directory can be a + relative of absolute path, and can be double quoted if needed. For the + <literal>include_dir</literal> form, all files not starting with a + <literal>.</literal> and ending with <literal>.conf</literal> will be + included. + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,6 +116,8 @@ <para> A record can have several formats: <synopsis> +include <replaceable>file</replaceable> +include_dir <replaceable>directory</replaceable> local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> @@ -118,6 +133,26 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_dir</literal></term> + <listitem> + <para> + This line will be replaced with the content of all the files found in + the directory, if they don't start with a <literal>.</literal> and end + with <literal>.conf</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -835,8 +870,10 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> +<replaceable>include_dir</replaceable> <replaceable>directory</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -847,6 +884,11 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + As for <filename>pg_hba.conf</filename>, the lines in this file can either + be inclusion directives or an authentication rule, and follow the same + rules. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index 327a4b42af..56b6cec9d5 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -21,6 +21,7 @@ #include <fcntl.h> #include <sys/param.h> #include <sys/socket.h> +#include <sys/stat.h> #include <netinet/in.h> #include <arpa/inet.h> #include <unistd.h> @@ -68,6 +69,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -112,10 +119,19 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, int elevel, char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); +static char *process_included_authfile(const char *inc_filename, + const char *outer_filename, int elevel, + MemoryContext linecxt, List **tok_lines); /* @@ -355,36 +371,11 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else - { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); - } + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename, + elevel, err_msg, &inc_fullname); - inc_file = AllocateFile(inc_fullname, "r"); if (inc_file == NULL) - { - int save_errno = errno; - - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); return tokens; - } /* There is possible recursion here if the file contains @ */ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); @@ -425,11 +416,36 @@ tokenize_inc_file(List *tokens, /* * tokenize_auth_file - * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a decicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel); + + return linecxt; +} + +/* + * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see the struct definition * in libpq/hba.h. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -438,30 +454,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -522,29 +530,151 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + AuthToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); + + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, filename, + elevel, linecxt, tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } + else if (strcmp(first->string, "include_dir") == 0) + { + char *dir_name; + DIR *d; + struct dirent *de; + StringInfoData err_buf; + + dir_name = second->string; + d = AllocateDir(dir_name); + if (d == NULL) + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not open directory \"%s\": %m", + dir_name))); + err_msg = psprintf("coud not open directory \"%s\": %m", + dir_name); + goto process_line; + } + + initStringInfo(&err_buf); + while ((de = ReadDir(d, dir_name)) != NULL) + { + struct stat st; + char inc_filename[MAXPGPATH]; + + /* + * Only parse files with names ending in ".conf". + * Explicitly reject files starting with ".". This + * excludes things like "." and "..", as well as typical + * hidden files, backup files, and editor debris. + */ + if (strlen(de->d_name) < 6) + continue; + if (de->d_name[0] == '.') + continue; + if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) + continue; + + join_path_components(inc_filename, dir_name, de->d_name); + canonicalize_path(inc_filename); + if (stat(inc_filename, &st) == 0) + { + /* Ignore directories. */ + if (S_ISDIR(st.st_mode)) + continue; + + /* + * err_msg is used here as a temp buffer, it will be + * overwritten at the end of the loop with the + * cumulated errors, if any. + */ + err_msg = process_included_authfile(inc_filename, + filename, elevel, linecxt, + tok_lines); + + /* Cumulate errors if any. */ + if (err_msg) + { + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfoString(&err_buf, err_msg); + } + } + else + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not stat file \"%s\": %m", + filename))); + + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfo(&err_buf, + "could not stat file \"%s\": %m", + inc_filename); + } + } + FreeDir(d); + + /* + * If there were no errors, the line is fully processed, bypass + * the general TokenizedAuthLine processing. + */ + if (err_buf.len == 0) + goto next_line; + + /* Otherwise, process the cumulated errors, if any. */ + err_msg = err_buf.data; + } } +process_line: + /* + * General processing: report the error if any and emit line to the + * TokenizedAuthLine + */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; + } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -859,7 +989,7 @@ do { \ errmsg("authentication option \"%s\" is only valid for authentication methods %s", \ optname, _(validmethods)), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \ optname, validmethods); \ return false; \ @@ -879,7 +1009,7 @@ do { \ errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname); \ return NULL; \ @@ -901,7 +1031,7 @@ do { \ ereport(elevel, \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("missing entry in file \"%s\" at end of line %d", \ - IdentFileName, line_num))); \ + tok_line->file_name, line_num))); \ *err_msg = psprintf("missing entry at end of line"); \ return NULL; \ } \ @@ -914,7 +1044,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("multiple values in ident field"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, tok_line->file_name))); \ *err_msg = psprintf("multiple values in ident field"); \ return NULL; \ } \ @@ -937,6 +1067,7 @@ HbaLine * parse_hba_line(TokenizedAuthLine *tok_line, int elevel) { int line_num = tok_line->line_num; + char *file_name = tok_line->file_name; char **err_msg = &tok_line->err_msg; char *str; struct addrinfo *gai_result; @@ -951,6 +1082,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -1675,6 +1807,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg) { int line_num = hbaline->linenumber; + char *file_name = hbaline->sourcefile; #ifdef USE_LDAP hbaline->ldapscope = LDAP_SCOPE_SUBTREE; @@ -2296,6 +2429,102 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} + +/* + * Try to open an included file, and tokenize it using the given context. + */ +static char * +process_included_authfile(const char *inc_filename, const char *outer_filename, + int elevel, MemoryContext linecxt, List **tok_lines) +{ + char *inc_fullname; + FILE *inc_file; + char *err_msg = NULL; + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, outer_filename, + elevel, &err_msg, &inc_fullname); + + if (inc_file == NULL) + { + /* open_inc_file should have reported an error. */ + Assert(err_msg != NULL); + return err_msg; + } + else + { + /* No error message should have been reported. */ + Assert(err_msg == NULL); + } + + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + return NULL; +} /* * Parse one tokenised line from the ident config file and store the result in diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..0050e24186 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,6 +9,8 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # +# include FILE +# include_dir DIRECTORY # local DATABASE USER METHOD [OPTIONS] # host DATABASE USER ADDRESS METHOD [OPTIONS] # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] @@ -18,7 +20,13 @@ # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include" or "include_dir", it's not a mapping record but a directive to +# include records from other file(s), as specified in the field. FILE is the +# file to include, and DIR is the directory containing the file(s) to include +# It can be specified with a relative or absolute path, and can be double +# quoted if it contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..9b700d9989 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,20 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # -# MAPNAME SYSTEM-USERNAME PG-USERNAME +# include FILE +# include_dir FILE +# MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include" or "include_dir", it's not an authentication +# record but a directive to include records from other file(s), specified in +# the field. FILE is the file to include and DIR is the directory containing +# the file(s) to include. It can be specified with a relative or absolute +# path, and can be double quoted if it contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index 9fe7b62c9a..4d902927e0 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,9 +26,11 @@ static ArrayType *get_hba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, + int mapping_number, const char *filename, int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); @@ -157,7 +159,7 @@ get_hba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 9 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line @@ -174,7 +176,8 @@ get_hba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg) + int rule_number, const char *filename, int lineno, HbaLine *hba, + const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; bool nulls[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -193,6 +196,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* rule_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -336,7 +346,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -359,6 +369,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *hba_lines = NIL; ListCell *line; + int rule_number = 0; MemoryContext linecxt; MemoryContext hbacxt; MemoryContext oldcxt; @@ -393,8 +404,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) hbaline = parse_hba_line(tok_line, DEBUG3); - fill_hba_line(tuple_store, tupdesc, tok_line->line_num, - hbaline, tok_line->err_msg); + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + rule_number++; + + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -430,8 +445,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) PG_RETURN_NULL(); } -/* Number of columns in pg_ident_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5 +/* Number of columns in pg_hba_file_mappings view */ +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -448,7 +463,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, IdentLine *ident, const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -461,6 +477,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* mapping_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -473,7 +496,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -495,6 +518,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *ident_lines = NIL; ListCell *line; + int mapping_number = 0; MemoryContext linecxt; MemoryContext identcxt; MemoryContext oldcxt; @@ -529,7 +553,12 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) identline = parse_ident_line(tok_line, DEBUG3); - fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline, + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + mapping_number++; + + fill_ident_line(tuple_store, tupdesc, mapping_number, + tok_line->file_name, tok_line->line_num, identline, tok_line->err_msg); } diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index babe16f00a..65bfd32753 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6128,15 +6128,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o}', - proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '6250', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,text,text,text}', proargmodes => '{o,o,o,o,o}', - proargnames => '{line_number,map_name,sys_name,pg_username,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index 90036f7bcd..59f6faf9f8 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -155,6 +156,7 @@ typedef struct AuthToken typedef struct TokenizedAuthLine { List *fields; /* List of lists of AuthTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 21effe8315..11ffe1743b 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname, WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); -pg_hba_file_rules| SELECT a.line_number, +pg_hba_file_rules| SELECT a.rule_number, + a.file_name, + a.line_number, a.type, a.database, a.user_name, @@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error); -pg_ident_file_mappings| SELECT a.line_number, + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); +pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, + a.line_number, a.map_name, a.sys_name, a.pg_username, a.error - FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_username, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.33.1 --o45jk7yo2ulu7ifa Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v7-0002-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v7 1/2] Allow file inclusion in pg_hba and pg_ident files. @ 2022-02-21 07:45 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw) Catversion is bumped. Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/catalogs.sgml | 48 +++- doc/src/sgml/client-auth.sgml | 48 +++- src/backend/libpq/hba.c | 343 +++++++++++++++++++++---- src/backend/libpq/pg_hba.conf.sample | 10 +- src/backend/libpq/pg_ident.conf.sample | 12 +- src/backend/utils/adt/hbafuncs.c | 51 +++- src/include/catalog/pg_proc.dat | 11 +- src/include/libpq/hba.h | 2 + src/test/regress/expected/rules.out | 12 +- 9 files changed, 453 insertions(+), 84 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index a533a2153e..b44e62d388 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -10554,12 +10554,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>rule_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this rule if the rule is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule in the given file_name </para></entry> </row> @@ -10694,6 +10713,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>mapping_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this mapping if the mapping is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this mapping + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>line_number</structfield> <type>int4</type> + </para> + <para> + Line number of this mapping in the given file_name + </para></entry> + </row> <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index b2a459fb0d..f7d871e660 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,21 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> or <literal>include_dir</literal> directive and + the file or directory to be included. The file or directory can be a + relative of absolute path, and can be double quoted if needed. For the + <literal>include_dir</literal> form, all files not starting with a + <literal>.</literal> and ending with <literal>.conf</literal> will be + included. + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,6 +116,8 @@ <para> A record can have several formats: <synopsis> +include <replaceable>file</replaceable> +include_dir <replaceable>directory</replaceable> local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> @@ -118,6 +133,26 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_dir</literal></term> + <listitem> + <para> + This line will be replaced with the content of all the files found in + the directory, if they don't start with a <literal>.</literal> and end + with <literal>.conf</literal>. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -835,8 +870,10 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> +<replaceable>include_dir</replaceable> <replaceable>directory</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -847,6 +884,11 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + As for <filename>pg_hba.conf</filename>, the lines in this file can either + be inclusion directives or an authentication rule, and follow the same + rules. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index 327a4b42af..56b6cec9d5 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -21,6 +21,7 @@ #include <fcntl.h> #include <sys/param.h> #include <sys/socket.h> +#include <sys/stat.h> #include <netinet/in.h> #include <arpa/inet.h> #include <unistd.h> @@ -68,6 +69,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -112,10 +119,19 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, int elevel, char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); +static char *process_included_authfile(const char *inc_filename, + const char *outer_filename, int elevel, + MemoryContext linecxt, List **tok_lines); /* @@ -355,36 +371,11 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else - { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); - } + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename, + elevel, err_msg, &inc_fullname); - inc_file = AllocateFile(inc_fullname, "r"); if (inc_file == NULL) - { - int save_errno = errno; - - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); return tokens; - } /* There is possible recursion here if the file contains @ */ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); @@ -425,11 +416,36 @@ tokenize_inc_file(List *tokens, /* * tokenize_auth_file - * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a decicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel); + + return linecxt; +} + +/* + * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see the struct definition * in libpq/hba.h. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -438,30 +454,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -522,29 +530,151 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + AuthToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); + + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, filename, + elevel, linecxt, tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } + else if (strcmp(first->string, "include_dir") == 0) + { + char *dir_name; + DIR *d; + struct dirent *de; + StringInfoData err_buf; + + dir_name = second->string; + d = AllocateDir(dir_name); + if (d == NULL) + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not open directory \"%s\": %m", + dir_name))); + err_msg = psprintf("coud not open directory \"%s\": %m", + dir_name); + goto process_line; + } + + initStringInfo(&err_buf); + while ((de = ReadDir(d, dir_name)) != NULL) + { + struct stat st; + char inc_filename[MAXPGPATH]; + + /* + * Only parse files with names ending in ".conf". + * Explicitly reject files starting with ".". This + * excludes things like "." and "..", as well as typical + * hidden files, backup files, and editor debris. + */ + if (strlen(de->d_name) < 6) + continue; + if (de->d_name[0] == '.') + continue; + if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) + continue; + + join_path_components(inc_filename, dir_name, de->d_name); + canonicalize_path(inc_filename); + if (stat(inc_filename, &st) == 0) + { + /* Ignore directories. */ + if (S_ISDIR(st.st_mode)) + continue; + + /* + * err_msg is used here as a temp buffer, it will be + * overwritten at the end of the loop with the + * cumulated errors, if any. + */ + err_msg = process_included_authfile(inc_filename, + filename, elevel, linecxt, + tok_lines); + + /* Cumulate errors if any. */ + if (err_msg) + { + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfoString(&err_buf, err_msg); + } + } + else + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not stat file \"%s\": %m", + filename))); + + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfo(&err_buf, + "could not stat file \"%s\": %m", + inc_filename); + } + } + FreeDir(d); + + /* + * If there were no errors, the line is fully processed, bypass + * the general TokenizedAuthLine processing. + */ + if (err_buf.len == 0) + goto next_line; + + /* Otherwise, process the cumulated errors, if any. */ + err_msg = err_buf.data; + } } +process_line: + /* + * General processing: report the error if any and emit line to the + * TokenizedAuthLine + */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; + } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -859,7 +989,7 @@ do { \ errmsg("authentication option \"%s\" is only valid for authentication methods %s", \ optname, _(validmethods)), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \ optname, validmethods); \ return false; \ @@ -879,7 +1009,7 @@ do { \ errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname); \ return NULL; \ @@ -901,7 +1031,7 @@ do { \ ereport(elevel, \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("missing entry in file \"%s\" at end of line %d", \ - IdentFileName, line_num))); \ + tok_line->file_name, line_num))); \ *err_msg = psprintf("missing entry at end of line"); \ return NULL; \ } \ @@ -914,7 +1044,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("multiple values in ident field"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, tok_line->file_name))); \ *err_msg = psprintf("multiple values in ident field"); \ return NULL; \ } \ @@ -937,6 +1067,7 @@ HbaLine * parse_hba_line(TokenizedAuthLine *tok_line, int elevel) { int line_num = tok_line->line_num; + char *file_name = tok_line->file_name; char **err_msg = &tok_line->err_msg; char *str; struct addrinfo *gai_result; @@ -951,6 +1082,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -1675,6 +1807,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg) { int line_num = hbaline->linenumber; + char *file_name = hbaline->sourcefile; #ifdef USE_LDAP hbaline->ldapscope = LDAP_SCOPE_SUBTREE; @@ -2296,6 +2429,102 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} + +/* + * Try to open an included file, and tokenize it using the given context. + */ +static char * +process_included_authfile(const char *inc_filename, const char *outer_filename, + int elevel, MemoryContext linecxt, List **tok_lines) +{ + char *inc_fullname; + FILE *inc_file; + char *err_msg = NULL; + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, outer_filename, + elevel, &err_msg, &inc_fullname); + + if (inc_file == NULL) + { + /* open_inc_file should have reported an error. */ + Assert(err_msg != NULL); + return err_msg; + } + else + { + /* No error message should have been reported. */ + Assert(err_msg == NULL); + } + + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + return NULL; +} /* * Parse one tokenised line from the ident config file and store the result in diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..0050e24186 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,6 +9,8 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # +# include FILE +# include_dir DIRECTORY # local DATABASE USER METHOD [OPTIONS] # host DATABASE USER ADDRESS METHOD [OPTIONS] # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] @@ -18,7 +20,13 @@ # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include" or "include_dir", it's not a mapping record but a directive to +# include records from other file(s), as specified in the field. FILE is the +# file to include, and DIR is the directory containing the file(s) to include +# It can be specified with a relative or absolute path, and can be double +# quoted if it contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..9b700d9989 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,20 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # -# MAPNAME SYSTEM-USERNAME PG-USERNAME +# include FILE +# include_dir FILE +# MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include" or "include_dir", it's not an authentication +# record but a directive to include records from other file(s), specified in +# the field. FILE is the file to include and DIR is the directory containing +# the file(s) to include. It can be specified with a relative or absolute +# path, and can be double quoted if it contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index 9fe7b62c9a..4d902927e0 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,9 +26,11 @@ static ArrayType *get_hba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, + int mapping_number, const char *filename, int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); @@ -157,7 +159,7 @@ get_hba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 9 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line @@ -174,7 +176,8 @@ get_hba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg) + int rule_number, const char *filename, int lineno, HbaLine *hba, + const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; bool nulls[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -193,6 +196,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* rule_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -336,7 +346,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -359,6 +369,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *hba_lines = NIL; ListCell *line; + int rule_number = 0; MemoryContext linecxt; MemoryContext hbacxt; MemoryContext oldcxt; @@ -393,8 +404,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) hbaline = parse_hba_line(tok_line, DEBUG3); - fill_hba_line(tuple_store, tupdesc, tok_line->line_num, - hbaline, tok_line->err_msg); + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + rule_number++; + + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -430,8 +445,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) PG_RETURN_NULL(); } -/* Number of columns in pg_ident_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5 +/* Number of columns in pg_hba_file_mappings view */ +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -448,7 +463,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, IdentLine *ident, const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -461,6 +477,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* mapping_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -473,7 +496,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -495,6 +518,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *ident_lines = NIL; ListCell *line; + int mapping_number = 0; MemoryContext linecxt; MemoryContext identcxt; MemoryContext oldcxt; @@ -529,7 +553,12 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) identline = parse_ident_line(tok_line, DEBUG3); - fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline, + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + mapping_number++; + + fill_ident_line(tuple_store, tupdesc, mapping_number, + tok_line->file_name, tok_line->line_num, identline, tok_line->err_msg); } diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index babe16f00a..65bfd32753 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6128,15 +6128,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o}', - proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '6250', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,text,text,text}', proargmodes => '{o,o,o,o,o}', - proargnames => '{line_number,map_name,sys_name,pg_username,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index 90036f7bcd..59f6faf9f8 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -155,6 +156,7 @@ typedef struct AuthToken typedef struct TokenizedAuthLine { List *fields; /* List of lists of AuthTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 21effe8315..11ffe1743b 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname, WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); -pg_hba_file_rules| SELECT a.line_number, +pg_hba_file_rules| SELECT a.rule_number, + a.file_name, + a.line_number, a.type, a.database, a.user_name, @@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error); -pg_ident_file_mappings| SELECT a.line_number, + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); +pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, + a.line_number, a.map_name, a.sys_name, a.pg_username, a.error - FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_username, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.33.1 --o45jk7yo2ulu7ifa Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v7-0002-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v4 2/3] Allow file inclusion in pg_hba and pg_ident files. @ 2022-02-21 07:45 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-02-21 07:45 UTC (permalink / raw) Catversion is bumped. Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/catalogs.sgml | 48 +++++- doc/src/sgml/client-auth.sgml | 34 +++- src/backend/libpq/hba.c | 229 +++++++++++++++++++------ src/backend/libpq/pg_hba.conf.sample | 8 +- src/backend/libpq/pg_ident.conf.sample | 8 +- src/backend/utils/adt/hbafuncs.c | 53 ++++-- src/include/catalog/pg_proc.dat | 12 +- src/include/libpq/hba.h | 2 + src/test/regress/expected/rules.out | 12 +- 9 files changed, 321 insertions(+), 85 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 75fedfa07e..bd1c9a8d21 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -10496,12 +10496,31 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>rule_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this rule if the rule is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule in the given file_name </para></entry> </row> @@ -10637,6 +10656,33 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </thead> <tbody> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>mapping_number</structfield> <type>int4</type> + </para> + <para> + Rule number, in priority order, of this mapping if the mapping is valid, + otherwise null + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + File name of this mapping + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>line_number</structfield> <type>int4</type> + </para> + <para> + Line number of this mapping in the given file_name + </para></entry> + </row> <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index 142b0affcb..e1d0e103b3 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,17 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> directive and the file to be included. The file + can be a relative of absolute path, and can be double quoted if needed. + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,6 +112,7 @@ <para> A record can have several formats: <synopsis> +include <replaceable>file</replaceable> local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> @@ -118,6 +128,15 @@ hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceabl The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -835,8 +854,9 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -847,6 +867,14 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + The lines can record can either be an inclusion directive or an authentication rule. + Inclusion records specifies files that can be included, which contains + additional records. The records will be inserted in lieu of the inclusion + records. Those records only contains two fields: the + <literal>include</literal> directive and the file to be included. The file + can be a relative of absolute path, and can be double quoted if needed. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index 556f473b41..1551b34c53 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -68,6 +68,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -112,10 +118,16 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, int elevel, char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); /* @@ -355,36 +367,11 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else - { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); - } + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, outer_filename, + elevel, err_msg, &inc_fullname); - inc_file = AllocateFile(inc_fullname, "r"); if (inc_file == NULL) - { - int save_errno = errno; - - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); return tokens; - } /* There is possible recursion here if the file contains @ */ linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); @@ -425,11 +412,36 @@ tokenize_inc_file(List *tokens, /* * tokenize_auth_file - * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a decicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, elevel); + + return linecxt; +} + +/* + * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see the struct definition * in libpq/hba.h. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -438,30 +450,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -522,29 +526,76 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + AuthToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; + char *inc_fullname; + FILE *inc_file; + + inc_filename = second->string; + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, + filename, elevel, &err_msg, + &inc_fullname); + + /* + * The included file could be open, now recursively process it. + * Errors will be reported in the general TokenizedAuthLine + * processing. + */ + if (inc_file != NULL) + { + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + else + { + /* We should got an error */ + Assert(err_msg != NULL); + } + } } + /* General processing: emit line to the TokenizedAuthLine */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; + } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -859,7 +910,7 @@ do { \ errmsg("authentication option \"%s\" is only valid for authentication methods %s", \ optname, _(validmethods)), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \ optname, validmethods); \ return false; \ @@ -879,7 +930,7 @@ do { \ errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname); \ return NULL; \ @@ -901,7 +952,7 @@ do { \ ereport(elevel, \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("missing entry in file \"%s\" at end of line %d", \ - IdentFileName, line_num))); \ + tok_line->file_name, line_num))); \ *err_msg = psprintf("missing entry at end of line"); \ return NULL; \ } \ @@ -914,7 +965,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("multiple values in ident field"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, tok_line->file_name))); \ *err_msg = psprintf("multiple values in ident field"); \ return NULL; \ } \ @@ -937,6 +988,7 @@ HbaLine * parse_hba_line(TokenizedAuthLine *tok_line, int elevel) { int line_num = tok_line->line_num; + char *file_name = tok_line->file_name; char **err_msg = &tok_line->err_msg; char *str; struct addrinfo *gai_result; @@ -951,6 +1003,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -1677,6 +1730,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg) { int line_num = hbaline->linenumber; + char *file_name = hbaline->sourcefile; #ifdef USE_LDAP hbaline->ldapscope = LDAP_SCOPE_SUBTREE; @@ -2299,6 +2353,67 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} /* * Parse one tokenised line from the ident config file and store the result in diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..0b6589a7b9 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,6 +9,7 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # +# include FILE # local DATABASE USER METHOD [OPTIONS] # host DATABASE USER ADDRESS METHOD [OPTIONS] # hostssl DATABASE USER ADDRESS METHOD [OPTIONS] @@ -18,7 +19,12 @@ # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include", it's not a mapping record but a directive to +# include records from another file, specified in the field. FILE is the file +# to include. It can be specified with a relative or absolute path, and can be +# double quoted if it contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..138359cf03 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,18 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # +# include FILE # MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include", it's not an authentication record but a +# directive to include records from another file, specified in the field. FILE +# is the file to include. It can be specified with a relative or absolute +# path, and can be double quoted if it contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index 1970b4c497..33cf5fb954 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,9 +26,11 @@ static ArrayType *get_hba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, + int mapping_number, const char *filename, int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); @@ -157,7 +159,7 @@ get_hba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 9 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line @@ -174,7 +176,8 @@ get_hba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, HbaLine *hba, const char *err_msg) + int rule_number, const char *filename, int lineno, HbaLine *hba, + const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; bool nulls[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -193,6 +196,13 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* rule_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -336,7 +346,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_HBA_FILE_RULES_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -359,6 +369,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *hba_lines = NIL; ListCell *line; + int rule_number = 0; MemoryContext linecxt; MemoryContext hbacxt; MemoryContext oldcxt; @@ -393,8 +404,12 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) hbaline = parse_hba_line(tok_line, DEBUG3); - fill_hba_line(tuple_store, tupdesc, tok_line->line_num, - hbaline, tok_line->err_msg); + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + rule_number++; + + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -430,8 +445,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) PG_RETURN_NULL(); } -/* Number of columns in pg_ident_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 5 +/* Number of columns in pg_hba_file_mappings view */ +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -448,7 +463,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int lineno, IdentLine *ident, const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -461,6 +477,13 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, memset(nulls, 0, sizeof(nulls)); index = 0; + /* mapping_number */ + if (err_msg) + nulls[index++] = true; + else + values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -473,7 +496,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[1], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 2) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -495,6 +518,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) FILE *file; List *ident_lines = NIL; ListCell *line; + int mapping_number = 0; MemoryContext linecxt; MemoryContext identcxt; MemoryContext oldcxt; @@ -529,8 +553,13 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) identline = parse_ident_line(tok_line, DEBUG3); - fill_ident_line(tuple_store, tupdesc, tok_line->line_num, identline, - tok_line->err_msg); + /* No error, set a rule number */ + if (tok_line->err_msg == NULL) + mapping_number++; + + fill_ident_line(tuple_store, tupdesc, mapping_number, + tok_line->file_name, tok_line->line_num, identline, + tok_line->err_msg); } /* Free tokenizer memory */ diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index abc9a83223..87842d5ce2 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6111,16 +6111,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o}', - proargnames => '{line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '9556', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,text,text,text,text}', - proargmodes => '{o,o,o,o,o}', - proargnames => '{line_number,map_name,sys_name,pg_username,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index 90036f7bcd..59f6faf9f8 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -155,6 +156,7 @@ typedef struct AuthToken typedef struct TokenizedAuthLine { List *fields; /* List of lists of AuthTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 5a20e5a7e1..e2c6aa8ffb 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1337,7 +1337,9 @@ pg_group| SELECT pg_authid.rolname AS groname, WHERE (pg_auth_members.roleid = pg_authid.oid)) AS grolist FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); -pg_hba_file_rules| SELECT a.line_number, +pg_hba_file_rules| SELECT a.rule_number, + a.file_name, + a.line_number, a.type, a.database, a.user_name, @@ -1346,13 +1348,15 @@ pg_hba_file_rules| SELECT a.line_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(line_number, type, database, user_name, address, netmask, auth_method, options, error); -pg_ident_file_mappings| SELECT a.line_number, + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); +pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, + a.line_number, a.map_name, a.sys_name, a.pg_username, a.error - FROM pg_ident_file_mappings() a(line_number, map_name, sys_name, pg_username, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.33.1 --4xmgvl4uv4ppkth6 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v4-0003-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v8 5/6] Allow file inclusion in pg_hba and pg_ident files. @ 2022-05-30 03:15 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-05-30 03:15 UTC (permalink / raw) pg_hba.conf file now has support for "include", "include_dir" and "include_if_exists" directives, which work similarly to the same directives in the postgresql.conf file. This fixes a possible crash if a secondary file tries to include itself as there's now a nesting depth check in the inclusion code path, same as the postgresql.conf. Many regression tests added to cover both the new directives, but also error detection for the whole pg_hba / pg_ident files. Catversion is bumped. Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/client-auth.sgml | 86 ++- doc/src/sgml/system-views.sgml | 22 +- src/backend/libpq/hba.c | 485 ++++++++++--- src/backend/libpq/pg_hba.conf.sample | 25 +- src/backend/libpq/pg_ident.conf.sample | 15 +- src/backend/utils/adt/hbafuncs.c | 43 +- src/backend/utils/misc/guc-file.l | 250 +++---- src/include/catalog/pg_proc.dat | 12 +- src/include/libpq/hba.h | 5 +- src/include/utils/guc.h | 2 + .../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++ src/test/regress/expected/rules.out | 6 +- 12 files changed, 1325 insertions(+), 283 deletions(-) create mode 100644 src/test/authentication/t/003_file_inclusion.pl diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index 433759928b..e4eacab4a5 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,23 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication + record. Inclusion directives specify files that can be included, which + contains additional records. The records will be inserted in lieu of the + inclusion records. Those records only contains two fields: the + <literal>include</literal>, <literal>include_if_exists</literal> or + <literal>include_dir</literal> directive and the file or directory to be + included. The file or directory can be a relative of absolute path, and can + be double quoted if needed. For the <literal>include_dir</literal> form, + all files not starting with a <literal>.</literal> and ending with + <literal>.conf</literal> will be included. Multiple files within an include + directory are processed in file name order (according to C locale rules, + i.e., numbers before letters, and uppercase letters before lowercase ones). + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,21 +118,57 @@ <para> A record can have several formats: <synopsis> -local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +include <replaceable>file</replaceable> +include_if_exists <replaceable>file</replaceable> +include_dir <replaceable>directory</replaceable> +local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> </synopsis> The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_if_exists</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file if the + file exists and can be read. Otherwise, a message will be logged to + indicate that the file is skipped. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_dir</literal></term> + <listitem> + <para> + This line will be replaced with the content of all the files found in + the directory, if they don't start with a <literal>.</literal> and end + with <literal>.conf</literal>, processed in file name order (according + to C locale rules, i.e., numbers before letters, and uppercase letters + before lowercase ones). + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -837,8 +888,10 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> +<replaceable>include_dir</replaceable> <replaceable>directory</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -849,6 +902,11 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + As for <filename>pg_hba.conf</filename>, the lines in this file can either + be inclusion directives or user name map records, and follow the same + rules. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index b4ee40c042..35ed69a64a 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -1003,12 +1003,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule the given <literal>file_name</literal> </para></entry> </row> @@ -1153,12 +1162,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this mapping + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_ident.conf</filename> + Line number of this mapping in the given <literal>file_name</literal> </para></entry> </row> diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index deee05c197..814bfcf30d 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -21,6 +21,7 @@ #include <fcntl.h> #include <sys/param.h> #include <sys/socket.h> +#include <sys/stat.h> #include <netinet/in.h> #include <arpa/inet.h> #include <unistd.h> @@ -68,6 +69,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -112,10 +119,22 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int depth, + int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, - const char *inc_filename, int elevel, char **err_msg); + const char *inc_filename, int depth, int elevel, + char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + bool strict, const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); +static char *process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, + int elevel, MemoryContext linecxt, + List **tok_lines); /* @@ -302,7 +321,7 @@ copy_auth_token(AuthToken *in) */ static List * next_field_expand(const char *filename, char **lineptr, - int elevel, char **err_msg) + int depth, int elevel, char **err_msg) { char buf[MAX_TOKEN]; bool trailing_comma; @@ -318,7 +337,7 @@ next_field_expand(const char *filename, char **lineptr, /* Is this referencing a file? */ if (!initial_quote && buf[0] == '@' && buf[1] != '\0') - tokens = tokenize_inc_file(tokens, filename, buf + 1, + tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1, elevel, err_msg); else tokens = lappend(tokens, make_auth_token(buf, initial_quote)); @@ -346,6 +365,7 @@ static List * tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, + int depth, int elevel, char **err_msg) { @@ -355,39 +375,30 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); + *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", *err_msg))); + return tokens; } - inc_file = AllocateFile(inc_fullname, "r"); - if (inc_file == NULL) - { - int save_errno = errno; + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true, + outer_filename, elevel, err_msg, &inc_fullname); - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); + if (inc_file == NULL) return tokens; - } /* There is possible recursion here if the file contains @ */ - linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); + linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1, + elevel); FreeFile(inc_file); pfree(inc_fullname); @@ -425,11 +436,38 @@ tokenize_inc_file(List *tokens, /* * tokenize_auth_file - * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a dedicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, + int depth, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, depth, + elevel); + + return linecxt; +} + +/* + * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see the struct definition * in libpq/hba.h. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -438,30 +476,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int depth, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -514,7 +544,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, { List *current_field; - current_field = next_field_expand(filename, &lineptr, + current_field = next_field_expand(filename, &lineptr, depth, elevel, &err_msg); /* add field to line, unless we are at EOL or comment start */ if (current_field != NIL) @@ -522,29 +552,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + AuthToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); + + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; + + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, true, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } + else if (strcmp(first->string, "include_dir") == 0) + { + char **filenames; + char *dir_name = second->string; + int num_filenames; + StringInfoData err_buf; + + filenames = GetDirConfFiles(dir_name, filename, elevel, + &num_filenames, &err_msg); + + if (!filenames) + { + /* We have the error in err_msg, simply process it */ + goto process_line; + } + + initStringInfo(&err_buf); + for (int i = 0; i < num_filenames; i++) + { + /* + * err_msg is used here as a temp buffer, it will be + * overwritten at the end of the loop with the + * cumulated errors, if any. + */ + err_msg = process_included_authfile(filenames[i], true, + filename, depth + 1, elevel, + linecxt, tok_lines); + + /* Cumulate errors if any. */ + if (err_msg) + { + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfoString(&err_buf, err_msg); + } + } + + /* + * If there were no errors, the line is fully processed, bypass + * the general TokenizedAuthLine processing. + */ + if (err_buf.len == 0) + goto next_line; + + /* Otherwise, process the cumulated errors, if any. */ + err_msg = err_buf.data; + } + else if (strcmp(first->string, "include_if_exists") == 0) + { + char *inc_filename; - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, false, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } } +process_line: + /* + * General processing: report the error if any and emit line to the + * TokenizedAuthLine + */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -859,7 +987,7 @@ do { \ errmsg("authentication option \"%s\" is only valid for authentication methods %s", \ optname, _(validmethods)), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \ optname, validmethods); \ return false; \ @@ -879,7 +1007,7 @@ do { \ errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname); \ return NULL; \ @@ -902,7 +1030,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("missing entry at end of line"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("missing entry at end of line"); \ return NULL; \ } \ @@ -915,7 +1043,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("multiple values in ident field"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("multiple values in ident field"); \ return NULL; \ } \ @@ -938,6 +1066,7 @@ HbaLine * parse_hba_line(TokenizedAuthLine *tok_line, int elevel) { int line_num = tok_line->line_num; + char *file_name = tok_line->file_name; char **err_msg = &tok_line->err_msg; char *str; struct addrinfo *gai_result; @@ -952,6 +1081,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -966,7 +1096,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for connection type"), errhint("Specify exactly one connection type per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for connection type"; return NULL; } @@ -980,7 +1110,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("local connections are not supported by this build"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "local connections are not supported by this build"; return NULL; #endif @@ -1004,7 +1134,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("hostssl record cannot match because SSL is disabled"), errhint("Set ssl = on in postgresql.conf."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostssl record cannot match because SSL is disabled"; } #else @@ -1012,7 +1142,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("hostssl record cannot match because SSL is not supported by this build"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostssl record cannot match because SSL is not supported by this build"; #endif } @@ -1024,7 +1154,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("hostgssenc record cannot match because GSSAPI is not supported by this build"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostgssenc record cannot match because GSSAPI is not supported by this build"; #endif } @@ -1045,7 +1175,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid connection type \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid connection type \"%s\"", token->string); return NULL; } @@ -1058,7 +1188,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before database specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before database specification"; return NULL; } @@ -1078,7 +1208,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before role specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before role specification"; return NULL; } @@ -1100,7 +1230,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before IP address specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before IP address specification"; return NULL; } @@ -1112,7 +1242,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for host address"), errhint("Specify one address range per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for host address"; return NULL; } @@ -1171,7 +1301,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid IP address \"%s\": %s", str, gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid IP address \"%s\": %s", str, gai_strerror(ret)); if (gai_result) @@ -1191,7 +1321,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("specifying both host name and CIDR mask is invalid: \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("specifying both host name and CIDR mask is invalid: \"%s\"", token->string); return NULL; @@ -1205,7 +1335,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid CIDR mask in address \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid CIDR mask in address \"%s\"", token->string); return NULL; @@ -1225,7 +1355,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("end-of-line before netmask specification"), errhint("Specify an address range in CIDR notation, or provide a separate netmask."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before netmask specification"; return NULL; } @@ -1236,7 +1366,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("multiple values specified for netmask"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for netmask"; return NULL; } @@ -1251,7 +1381,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid IP mask \"%s\": %s", token->string, gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid IP mask \"%s\": %s", token->string, gai_strerror(ret)); if (gai_result) @@ -1270,7 +1400,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("IP address and mask do not match"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "IP address and mask do not match"; return NULL; } @@ -1286,7 +1416,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before authentication method"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before authentication method"; return NULL; } @@ -1298,7 +1428,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for authentication type"), errhint("Specify exactly one authentication type per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for authentication type"; return NULL; } @@ -1335,7 +1465,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("MD5 authentication is not supported when \"db_user_namespace\" is enabled"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "MD5 authentication is not supported when \"db_user_namespace\" is enabled"; return NULL; } @@ -1376,7 +1506,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid authentication method \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid authentication method \"%s\"", token->string); return NULL; @@ -1389,7 +1519,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid authentication method \"%s\": not supported by this build", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid authentication method \"%s\": not supported by this build", token->string); return NULL; @@ -1411,7 +1541,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("gssapi authentication is not supported on local sockets"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "gssapi authentication is not supported on local sockets"; return NULL; } @@ -1423,7 +1553,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("peer authentication is only supported on local sockets"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "peer authentication is only supported on local sockets"; return NULL; } @@ -1441,7 +1571,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cert authentication is only supported on hostssl connections"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cert authentication is only supported on hostssl connections"; return NULL; } @@ -1491,7 +1621,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("authentication option not in name=value format: %s", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("authentication option not in name=value format: %s", token->string); return NULL; @@ -1535,7 +1665,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"; return NULL; } @@ -1546,7 +1676,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"; return NULL; } @@ -1562,7 +1692,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cannot use ldapsearchattribute together with ldapsearchfilter"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cannot use ldapsearchattribute together with ldapsearchfilter"; return NULL; } @@ -1579,7 +1709,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("list of RADIUS servers cannot be empty"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "list of RADIUS servers cannot be empty"; return NULL; } @@ -1590,7 +1720,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("list of RADIUS secrets cannot be empty"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "list of RADIUS secrets cannot be empty"; return NULL; } @@ -1609,7 +1739,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiussecrets), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS secrets (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiussecrets), list_length(parsedline->radiusservers)); @@ -1625,7 +1755,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiusports), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS ports (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiusports), list_length(parsedline->radiusservers)); @@ -1641,7 +1771,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiusidentifiers), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS identifiers (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiusidentifiers), list_length(parsedline->radiusservers)); @@ -1676,6 +1806,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg) { int line_num = hbaline->linenumber; + char *file_name = hbaline->sourcefile; #ifdef USE_LDAP hbaline->ldapscope = LDAP_SCOPE_SUBTREE; @@ -1699,7 +1830,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientcert can only be configured for \"hostssl\" rows"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientcert can only be configured for \"hostssl\" rows"; return false; } @@ -1716,7 +1847,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientcert only accepts \"verify-full\" when using \"cert\" authentication"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientcert can only be set to \"verify-full\" when using \"cert\" authentication"; return false; } @@ -1729,7 +1860,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid value for clientcert: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } } @@ -1741,7 +1872,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientname can only be configured for \"hostssl\" rows"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientname can only be configured for \"hostssl\" rows"; return false; } @@ -1760,7 +1891,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid value for clientname: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } } @@ -1846,7 +1977,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid ldapscheme value: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); hbaline->ldapscheme = pstrdup(val); } else if (strcmp(name, "ldapserver") == 0) @@ -1864,7 +1995,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid LDAP port number: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid LDAP port number: \"%s\"", val); return false; } @@ -1958,7 +2089,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS server list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -1977,7 +2108,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not translate RADIUS server name \"%s\" to address: %s", (char *) lfirst(l), gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); if (gai_result) pg_freeaddrinfo_all(hints.ai_family, gai_result); @@ -2006,7 +2137,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS port list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid RADIUS port number: \"%s\"", val); return false; } @@ -2019,7 +2150,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid RADIUS port number: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2042,7 +2173,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS secret list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2064,7 +2195,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS identifiers list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2078,7 +2209,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("unrecognized authentication option name: \"%s\"", name), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("unrecognized authentication option name: \"%s\"", name); return false; @@ -2226,7 +2357,7 @@ load_hba(void) return false; } - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ @@ -2297,6 +2428,137 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + if (strict) + { + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + } + else + { + Assert(kind == IncludedAuthFile); + ereport(LOG, + (errmsg("skipping missing authentication file \"%s\"", + *inc_fullname))); + } + + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} + +/* + * Try to open an included file, and tokenize it using the given context. + * Returns NULL if no error happens during tokenization, otherwise the error. + */ +static char * +process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, int elevel, + MemoryContext linecxt, List **tok_lines) +{ + char *inc_fullname; + FILE *inc_file; + char *err_msg = NULL; + + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) + { + char *err_msg; + + err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", err_msg))); + return err_msg; + } + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict, + outer_filename, elevel, &err_msg, &inc_fullname); + + if (inc_file == NULL) + { + if (strict) + { + /* open_inc_file should have reported an error. */ + Assert(err_msg != NULL); + return err_msg; + } + else + return NULL; + } + else + { + /* No error message should have been reported. */ + Assert(err_msg == NULL); + } + + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, depth, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + return NULL; +} /* * Parse one tokenised line from the ident config file and store the result in @@ -2315,6 +2577,7 @@ load_hba(void) IdentLine * parse_ident_line(TokenizedAuthLine *tok_line, int elevel) { + char *file_name = tok_line->file_name; int line_num = tok_line->line_num; char **err_msg = &tok_line->err_msg; ListCell *field; @@ -2375,7 +2638,7 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid regular expression \"%s\": %s", parsedline->ident_user + 1, errstr), errcontext("line %d of configuration file \"%s\"", - line_num, IdentFileName))); + line_num, file_name))); *err_msg = psprintf("invalid regular expression \"%s\": %s", parsedline->ident_user + 1, errstr); @@ -2610,7 +2873,7 @@ load_ident(void) return false; } - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..7433050112 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,16 +9,27 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # -# local DATABASE USER METHOD [OPTIONS] -# host DATABASE USER ADDRESS METHOD [OPTIONS] -# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# local DATABASE USER METHOD [OPTIONS] +# host DATABASE USER ADDRESS METHOD [OPTIONS] +# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..8e3fa29135 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,23 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # -# MAPNAME SYSTEM-USERNAME PG-USERNAME +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index c9be4bff1f..15326a01e2 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,12 +26,12 @@ static ArrayType *get_hba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int rule_number, int lineno, HbaLine *hba, - const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int mapping_number, int lineno, IdentLine *ident, - const char *err_msg); + int mapping_number, const char *filename, + int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); @@ -159,7 +159,7 @@ get_hba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 10 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line @@ -168,7 +168,8 @@ get_hba_options(HbaLine *hba) * tuple_store: where to store data * tupdesc: tuple descriptor for the view * rule_number: unique rule identifier among all valid rules - * lineno: pg_hba.conf line number (must always be valid) + * filename: name of the file containing that line + * lineno: line number in that file (must always be valid) * hba: parsed line data (can be NULL, in which case err_msg should be set) * err_msg: error message (NULL if none) * @@ -177,7 +178,7 @@ get_hba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int rule_number, int lineno, HbaLine *hba, + int rule_number, const char *filename, int lineno, HbaLine *hba, const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -202,6 +203,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -345,7 +348,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -386,7 +389,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open configuration file \"%s\": %m", HbaFileName))); - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -407,8 +410,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) rule_number++; - fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->line_num, - hbaline, tok_line->err_msg); + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -445,7 +448,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) } /* Number of columns in pg_hba_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6 +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -454,7 +457,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) * tuple_store: where to store data * tupdesc: tuple descriptor for the view * mapping_number: unique rule identifier among all valid rules - * lineno: pg_ident.conf line number (must always be valid) + * filename: name of the file containing that line + * lineno: line number in that file (must always be valid) * ident: parsed line data (can be NULL, in which case err_msg should be set) * err_msg: error message (NULL if none) * @@ -463,8 +467,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int mapping_number, int lineno, IdentLine *ident, - const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -482,6 +486,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -494,7 +500,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -534,7 +540,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open usermap file \"%s\": %m", IdentFileName))); - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -556,7 +562,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) mapping_number++; fill_ident_line(tuple_store, tupdesc, mapping_number, - tok_line->line_num, identline, tok_line->err_msg); + tok_line->file_name, tok_line->line_num, identline, + tok_line->err_msg); } /* Free tokenizer memory */ diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l index ce5633844c..b3e18e48cf 100644 --- a/src/backend/utils/misc/guc-file.l +++ b/src/backend/utils/misc/guc-file.l @@ -700,6 +700,122 @@ GUC_flex_fatal(const char *msg) return 0; /* keep compiler quiet */ } +/* + * Returns the list of config files located in a directory, in alphabetical + * order. + * + * We don't check for recursion or too-deep nesting depth here, its up to the + * caller to take care of that. + */ +char ** +GetDirConfFiles(const char *includedir, const char *calling_file, int elevel, + int *num_filenames, char **err_msg) +{ + char *directory; + DIR *d; + struct dirent *de; + char **filenames; + int size_filenames; + + /* + * Reject directory name that is all-blank (including empty), as that + * leads to confusion --- we'd read the containing directory, typically + * resulting in recursive inclusion of the same file(s). + */ + if (strspn(includedir, " \t\r\n") == strlen(includedir)) + { + ereport(elevel, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("empty configuration directory name: \"%s\"", + includedir))); + *err_msg = "empty configuration directory name"; + return NULL; + } + + directory = AbsoluteConfigLocation(includedir, calling_file); + d = AllocateDir(directory); + if (d == NULL) + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not open configuration directory \"%s\": %m", + directory))); + *err_msg = psprintf("could not open directory \"%s\"", directory); + filenames = NULL; + goto cleanup; + } + + /* + * Read the directory and put the filenames in an array, so we can sort + * them prior to caller processing the contents. + */ + size_filenames = 32; + filenames = (char **) palloc(size_filenames * sizeof(char *)); + *num_filenames = 0; + + while ((de = ReadDir(d, directory)) != NULL) + { + struct stat st; + char filename[MAXPGPATH]; + + /* + * Only parse files with names ending in ".conf". Explicitly reject + * files starting with ".". This excludes things like "." and "..", + * as well as typical hidden files, backup files, and editor debris. + */ + if (strlen(de->d_name) < 6) + continue; + if (de->d_name[0] == '.') + continue; + if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) + continue; + + join_path_components(filename, directory, de->d_name); + canonicalize_path(filename); + if (stat(filename, &st) == 0) + { + /* Ignore directories. */ + if (S_ISDIR(st.st_mode)) + continue; + + /* Add file to array, increasing its size in blocks of 32 */ + if (*num_filenames >= size_filenames) + { + size_filenames += 32; + filenames = (char **) repalloc(filenames, + size_filenames * sizeof(char *)); + } + filenames[*num_filenames] = pstrdup(filename); + (*num_filenames)++; + } + else + { + /* + * stat does not care about permissions, so the most likely reason + * a file can't be accessed now is if it was removed between the + * directory listing and now. + */ + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not stat file \"%s\": %m", + filename))); + *err_msg = psprintf("could not stat file \"%s\"", filename); + pfree(filenames); + filenames = NULL; + goto cleanup; + } + } + + if (*num_filenames > 0) + qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp); + +cleanup: + if (d) + FreeDir(d); + pfree(directory); + return filenames; +} + /* * Read and parse a single configuration file. This function recurses * to handle "include" directives. @@ -961,138 +1077,32 @@ ParseConfigDirectory(const char *includedir, ConfigVariable **head_p, ConfigVariable **tail_p) { - char *directory; - DIR *d; - struct dirent *de; + char *err_msg; char **filenames; int num_filenames; - int size_filenames; - bool status; - /* - * Reject directory name that is all-blank (including empty), as that - * leads to confusion --- we'd read the containing directory, typically - * resulting in recursive inclusion of the same file(s). - */ - if (strspn(includedir, " \t\r\n") == strlen(includedir)) - { - ereport(elevel, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("empty configuration directory name: \"%s\"", - includedir))); - record_config_file_error("empty configuration directory name", - calling_file, calling_lineno, - head_p, tail_p); - return false; - } + filenames = GetDirConfFiles(includedir, calling_file, elevel, + &num_filenames, &err_msg); - /* - * We don't check for recursion or too-deep nesting depth here; the - * subsequent calls to ParseConfigFile will take care of that. - */ - - directory = AbsoluteConfigLocation(includedir, calling_file); - d = AllocateDir(directory); - if (d == NULL) + if (!filenames) { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open configuration directory \"%s\": %m", - directory))); - record_config_file_error(psprintf("could not open directory \"%s\"", - directory), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } - - /* - * Read the directory and put the filenames in an array, so we can sort - * them prior to processing the contents. - */ - size_filenames = 32; - filenames = (char **) palloc(size_filenames * sizeof(char *)); - num_filenames = 0; - - while ((de = ReadDir(d, directory)) != NULL) - { - struct stat st; - char filename[MAXPGPATH]; - - /* - * Only parse files with names ending in ".conf". Explicitly reject - * files starting with ".". This excludes things like "." and "..", - * as well as typical hidden files, backup files, and editor debris. - */ - if (strlen(de->d_name) < 6) - continue; - if (de->d_name[0] == '.') - continue; - if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) - continue; - - join_path_components(filename, directory, de->d_name); - canonicalize_path(filename); - if (stat(filename, &st) == 0) - { - if (!S_ISDIR(st.st_mode)) - { - /* Add file to array, increasing its size in blocks of 32 */ - if (num_filenames >= size_filenames) - { - size_filenames += 32; - filenames = (char **) repalloc(filenames, - size_filenames * sizeof(char *)); - } - filenames[num_filenames] = pstrdup(filename); - num_filenames++; - } - } - else - { - /* - * stat does not care about permissions, so the most likely reason - * a file can't be accessed now is if it was removed between the - * directory listing and now. - */ - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - filename))); - record_config_file_error(psprintf("could not stat file \"%s\"", - filename), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } + record_config_file_error(err_msg, calling_file, calling_lineno, head_p, + tail_p); + return false; } - if (num_filenames > 0) + for (int i = 0; i < num_filenames; i++) { - int i; - - qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp); - for (i = 0; i < num_filenames; i++) + if (!ParseConfigFile(filenames[i], true, + calling_file, calling_lineno, + depth, elevel, + head_p, tail_p)) { - if (!ParseConfigFile(filenames[i], true, - calling_file, calling_lineno, - depth, elevel, - head_p, tail_p)) - { - status = false; - goto cleanup; - } + return false; } } - status = true; -cleanup: - if (d) - FreeDir(d); - pfree(directory); - return status; + return true; } /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index e544f9f758..d66b2443a4 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6128,16 +6128,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o}', - proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '6250', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,text,text,text}', - proargmodes => '{o,o,o,o,o,o}', - proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index 90036f7bcd..0ea100d1b8 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -155,6 +156,7 @@ typedef struct AuthToken typedef struct TokenizedAuthLine { List *fields; /* List of lists of AuthTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ @@ -174,6 +176,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel); extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel); extern bool pg_isblank(const char c); extern MemoryContext tokenize_auth_file(const char *filename, FILE *file, - List **tok_lines, int elevel); + List **tok_lines, int depth, + int elevel); #endif /* HBA_H */ diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h index e734493a48..1a3ab6306d 100644 --- a/src/include/utils/guc.h +++ b/src/include/utils/guc.h @@ -145,6 +145,8 @@ typedef struct ConfigVariable struct ConfigVariable *next; } ConfigVariable; +extern char **GetDirConfFiles(const char *includedir, const char *calling_file, + int elevel, int *num_filenames, char **err_msg); extern bool ParseConfigFile(const char *config_file, bool strict, const char *calling_file, int calling_lineno, int depth, int elevel, diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl new file mode 100644 index 0000000000..8eae72b8d4 --- /dev/null +++ b/src/test/authentication/t/003_file_inclusion.pl @@ -0,0 +1,657 @@ + +# Copyright (c) 2021-2022, PostgreSQL Global Development Group + +# Set of tests for authentication and pg_hba.conf inclusion. +# This test can only run with Unix-domain sockets. + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use Time::HiRes qw(usleep); +use IPC::Run qw(pump finish timer); +use Data::Dumper; + +if (!$use_unix_sockets) +{ + plan skip_all => + "authentication tests cannot run without Unix-domain sockets"; +} + +# stores the current line counter for each file. hba_rule and ident_rule are +# fake file names used for the global rule number for each auth view. +my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + +my $hba_file = 'subdir1/pg_hba_custom.conf'; +my $ident_file = 'subdir2/pg_ident_custom.conf'; + +# Initialize primary node +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init; +$node->start; + +my $data_dir = $node->data_dir; + +# Normalize the data directory for Windows +$data_dir =~ s/\/\.\//\//g; # reduce /./ to / +$data_dir =~ s/\/\//\//g; # reduce // to / +$data_dir =~ s/\/$//; # remove trailing / + + +# Add the given payload to the given relative HBA file of the given node. +# This function maintains the %cur_line metadata, so it has to be called in the +# expected inclusion evaluation order in order to keep it in sync. +# +# If the payload starts with "include" or "ignore", the function doesn't +# increase the general hba rule number. +# +# If an err_str is provided, it returns an arrayref containing the provided +# filename, the current line number in that file and the provided err_str. The +# err_str has to be a valid regex string. +# Otherwise it only returns the line number of the payload in the wanted file. +# This function has to be called in the expected inclusion evaluation order to +# keep the %cur_line information in sync. +sub add_hba_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'hba_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_hba_file_rules line + @tokens = split(/ /, $payload); + $tokens[1] = '{' . $tokens[1] . '}'; # database + $tokens[2] = '{' . $tokens[2] . '}'; # user_name + + # add empty address and netmask betweed user_name and auth_method + splice @tokens, 3, 0, ''; + splice @tokens, 3, 0, ''; + + # append empty options and error + push @tokens, ''; + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Add the given payload to the given relative ident file of the given node. +# Same as add_hba_line but for pg_ident files +sub add_ident_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'ident_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_ident_file_mappings line + @tokens = split(/ /, $payload); + + # append empty error + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Delete pg_hba.conf from the given node, add various entries to test the +# include infrastructure and then execute a reload to refresh it. +sub generate_valid_auth_files +{ + my $node = shift; + my $hba_expected = ''; + my $ident_expected = ''; + + # customise main auth file names + $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'"); + $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'"); + + # and make original ones invalid to be sure they're not used anywhere + $node->append_conf('pg_hba.conf', "some invalid line"); + $node->append_conf('pg_ident.conf', "some invalid line"); + + # pg_hba stuff + mkdir("$data_dir/subdir1"); + mkdir("$data_dir/hba_inc"); + mkdir("$data_dir/hba_inc_if"); + mkdir("$data_dir/hba_pos"); + + # Make sure we will still be able to connect + $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust'); + + # Add include data + add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf"); + $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject"); + + $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject"); + + add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject"); + # include is relative to current path + add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject"); + + # include_if_exists data + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none"); + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some"); + $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject"); + + # include_dir data + add_hba_line($node, "$hba_file", "include_dir ../hba_inc"); + add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included"); + $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject"); + $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject"); + + # secondary auth file + add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject'); + $node->append_conf('dbnames.conf', "db1"); + $node->append_conf('dbnames.conf', "db3"); + $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1) + . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1) + . '|local|{db1,db3}|{all}|||reject||'; + + # pg_ident stuff + mkdir("$data_dir/subdir2"); + mkdir("$data_dir/ident_inc"); + mkdir("$data_dir/ident_inc_if"); + mkdir("$data_dir/ident_pos"); + + # Add include data + add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf"); + $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar"); + + $ident_expected .= add_ident_line($node, "$ident_file", "test a b"); + + add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar"); + # include is relative to current path + add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar"); + + # include_if_exists data + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none"); + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some"); + $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar"); + + # include_dir data + add_ident_line($node, "$ident_file", "include_dir ../ident_inc"); + add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included"); + $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar"); + $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar"); + + $node->restart; + $node->connect_ok('dbname=postgres', + 'Connection ok after generating valid auth files'); + + return ($hba_expected, $ident_expected); +} + +# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal +# entries to allow authentication. +sub reset_auth_files +{ + my $node = shift; + + unlink("$data_dir/$hba_file"); + unlink("$data_dir/$ident_file"); + + %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + + return add_hba_line($node, "$hba_file", 'local all all trust'); +} + +# Generate a list of expected error regex for the given array of error +# conditions, as generated by add_hba_line/add_ident_line with an err_str. +# +# 2 regex are generated per array entry: one for the given err_str, and one for +# the expected line in the specific file. Since all lines are independant, +# there's no guarantee that a specific failure regex and the per-line regex +# will match the same error. Calling code should add at least one test with a +# single error to make sure that the line number / file name is correct. +# +# On top of that, an extra line is generated for the general failure to process +# the main auth file. +sub generate_log_err_patterns +{ + my $node = shift; + my $raw_errors = shift; + my $is_hba_err = shift; + my @errors; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + push @errors, qr/$err_str/; + + # Context messages with the file / line location aren't always emitted + if ($err_str !~ /maximum nesting depth exceeded/ and + $err_str !~ /could not open secondary authentication file/) + { + push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/ + } + } + + push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err); + + return \@errors; +} + +# Generate the expected output for the auth file view error reporting (file +# name, file line, error), for the given array of error conditions, as +# generated generated by add_hba_line/add_ident_line with an err_str. +sub generate_log_err_rows +{ + my $node = shift; + my $raw_errors = shift; + my $exp_rows = ''; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + $exp_rows .= "\n" if ($exp_rows ne ""); + + # Unescape regex patterns if any + $err_str =~ s/\\([\(\)])/$1/g; + $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str" + } + + return $exp_rows; +} + +# Reset the main auth files, append the given payload to the given config file, +# and check that the instance cannot start, raising the expected error line(s). +sub start_errors_like +{ + my $node = shift; + my $file = shift; + my $payload = shift; + my $pattern = shift; + my $should_fail = shift; + + reset_auth_files($node); + $node->append_conf($file, $payload); + + unlink($node->logfile); + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir, + '-l', $node->logfile, 'start'); + + if ($should_fail) + { + ok($ret != 0, "Cannot start postgres with faulty $file"); + } + else + { + ok($ret == 0, "postgres can start with faulty $file"); + } + + my $log_contents = slurp_file($node->logfile); + + foreach (@{$pattern}) + { + like($log_contents, + $_, + "Expected failure found in the logs"); + } + + if (not $should_fail) + { + # We can't simply call $node->stop here as the call is optimized out + # when the server isn't started with $node->start. + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', + $data_dir, 'stop', '-m', 'fast'); + ok($ret == 0, "Could stop postgres"); + } +} + +# We should be able to connect, and see an empty pg_ident.conf +is($node->psql( + 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'), + qq(0), + 'pg_ident.conf is empty'); + +############################################ +# part 1, test view reporting for valid data +############################################ +my ($exp_hba, $exp_ident) = generate_valid_auth_files($node); + +$node->connect_ok('dbname=postgres', 'Connection still ok'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_hba_file_rules'), + qq($exp_hba), + 'pg_hba_file_rules content is expected'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_ident_file_mappings'), + qq($exp_ident), + 'pg_ident_file_mappings content is expected'); + +############################################# +# part 2, test log reporting for invalid data +############################################# +reset_auth_files($node); +$node->restart('fast'); +$node->connect_ok('dbname=postgres', + 'Connection ok after resetting auth files'); + +$node->stop('fast'); + +start_errors_like($node, $hba_file, "include ../not_a_file", + [ + qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file +mkdir("$data_dir/hba_inc_fail"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token"); +start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail", + [ + qr/invalid connection type "not_a_token"/, + qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file with nested inclusion +unlink("$data_dir/hba_inc_fail/inc_dir.conf"); +my @hba_raw_errors_step1; + +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1"); + +add_hba_line($node, "hba_inc_fail/file1", "include file2"); +add_hba_line($node, "hba_inc_fail/file2", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file2", "include file3"); + +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3", + "local all all zuul", + 'invalid authentication method "zuul"'); + +start_errors_like( + $node, $hba_file, "include_dir ../hba_inc_fail", + generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @hba_raw_errors_step2; + +# include_if_exists, with various problems +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local", + "end-of-line before database specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local,host", + "multiple values specified for connection type"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all", + "end-of-line before role specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all all", + "end-of-line before authentication method"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "host all all test/42", + 'specifying both host name and CIDR mask is invalid: "test/42"'); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + 'local @dbnames_fails.conf all reject', + "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory"); + +add_hba_line($node, "hba_if_exists.conf", "include recurse.conf"); +push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf", + "include recurse.conf", + 'could not open configuration file "recurse.conf": maximum nesting depth exceeded'); + +# Generate the regex for the expected errors in the logs. There's no guarantee +# that the generated "line X of file..." will be emitted for the expected line, +# but previous tests already ensured that the correct line number / file name +# was emitted, so ensuring that there's an error in all expected lines is +# enough here. +my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2, + 1); + +# Not an error, but it should raise a message in the logs. Manually add an +# extra log message to detect +add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none"); +push @{$expected_errors}, + qr/skipping missing authentication file "$data_dir\/if_exists_none"/; + +start_errors_like( + $node, $hba_file, "include_if_exists ../hba_if_exists.conf", + $expected_errors, 1); + +# Mostly the same, but for ident files +reset_auth_files($node); + +my @ident_raw_errors_step1; + +# include_dir, single included file with nested inclusion +mkdir("$data_dir/ident_inc_fail"); +add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1"); + +add_ident_line($node, "ident_inc_fail/file1", "include file2"); +add_ident_line($node, "ident_inc_fail/file2", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file2", "include file3"); + +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3", + "failmap /(fail postgres", + 'invalid regular expression "\(fail": parentheses \(\) not balanced'); + +start_errors_like( + $node, $ident_file, "include_dir ../ident_inc_fail", + generate_log_err_patterns($node, \@ident_raw_errors_step1, 0), + 0); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @ident_raw_errors_step2; + +# include_if_exists, with various problems +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map", + "missing entry at end of line"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2", + "multiple values in ident field"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", + 'map @osnames_fails.conf postgres', + "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory"); + +add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf", + 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded'); + +start_errors_like( + $node, $ident_file, "include_if_exists ../ident_if_exists.conf", + # There's no guarantee that the generated "line X of file..." will be + # emitted for the expected line, but previous tests already ensured that + # the correct line number / file name was emitted, so ensuring that there's + # an error in all expected lines is enough here. + generate_log_err_patterns($node, \@ident_raw_errors_step2, 0), + 0); + +##################################################### +# part 3, test reporting of various error scenario +# NOTE: this will be bypassed -DEXEC_BACKEND or win32 +##################################################### +reset_auth_files($node); + +$node->start; +$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset'); + +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_hba_file_rules'); + +add_ident_line($node, $ident_file, ''); +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_ident_file_mappings'); + +# The instance could be restarted and no error is detected. Now check if the +# build is compatible with the view error reporting (EXEC_BACKEND / win32 will +# fail when trying to connect as they always rely on the current auth files +# content) +my @hba_raw_errors; + +push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +my ($stdout, $stderr); +my $cmdret = $node->psql('postgres', 'SELECT 1', + stdout => \$stdout, stderr => \$stderr); + +if ($cmdret != 0) +{ + # Connection failed. Bail out, but make sure to raise a failure if it + # didn't fail for the expected hba file modification. + like($stderr, + qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/, + "Connection failed due to loading an invalid hba file"); + + done_testing(); + diag("Build not compatible with auth file view error reporting, bail out.\n"); + exit; +} + +# Combine errors generated at step 2, in the same order. +$node->append_conf($hba_file, "include_dir ../hba_inc_fail"); +push @hba_raw_errors, @hba_raw_errors_step1; + +$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf"); +push @hba_raw_errors, @hba_raw_errors_step2; + +my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules' + . ' WHERE error IS NOT NULL ORDER BY rule_number'), + qq($hba_expected), + 'Detected all error in hba file'); + +# and do the same for pg_ident +my @ident_raw_errors; + +push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +$node->append_conf($ident_file, "include_dir ../ident_inc_fail"); +push @ident_raw_errors, @ident_raw_errors_step1; + +$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf"); +push @ident_raw_errors, @ident_raw_errors_step2; + +my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings' + . ' WHERE error IS NOT NULL ORDER BY mapping_number'), + qq($ident_expected), + 'Detected all error in ident file'); + +done_testing(); diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 79408710e0..5ed2fe3704 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname, FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); pg_hba_file_rules| SELECT a.rule_number, + a.file_name, a.line_number, a.type, a.database, @@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error); + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, a.line_number, a.map_name, a.sys_name, a.pg_username, a.error - FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.37.0 --lqayot32rzm7n4sf Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v8-0006-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v10 2/3] Allow file inclusion in pg_hba and pg_ident files. @ 2022-05-30 03:15 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-05-30 03:15 UTC (permalink / raw) pg_hba.conf file now has support for "include", "include_dir" and "include_if_exists" directives, which work similarly to the same directives in the postgresql.conf file. This fixes a possible crash if a secondary file tries to include itself as there's now a nesting depth check in the inclusion code path, same as the postgresql.conf. Many regression tests added to cover both the new directives, but also error detection for the whole pg_hba / pg_ident files. Catversion is bumped. Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/client-auth.sgml | 86 ++- doc/src/sgml/system-views.sgml | 22 +- src/backend/libpq/hba.c | 483 ++++++++++--- src/backend/libpq/pg_hba.conf.sample | 25 +- src/backend/libpq/pg_ident.conf.sample | 15 +- src/backend/utils/adt/hbafuncs.c | 43 +- src/backend/utils/misc/guc-file.l | 229 +++--- src/include/catalog/pg_proc.dat | 12 +- src/include/libpq/hba.h | 5 +- src/include/utils/guc.h | 2 + .../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++ src/test/regress/expected/rules.out | 6 +- 12 files changed, 1312 insertions(+), 273 deletions(-) create mode 100644 src/test/authentication/t/003_file_inclusion.pl diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index c6f1b70fd3..42ceb6f3e6 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,23 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication + record. Inclusion directives specify files that can be included, which + contains additional records. The records will be inserted in lieu of the + inclusion records. Those records only contains two fields: the + <literal>include</literal>, <literal>include_if_exists</literal> or + <literal>include_dir</literal> directive and the file or directory to be + included. The file or directory can be a relative of absolute path, and can + be double quoted if needed. For the <literal>include_dir</literal> form, + all files not starting with a <literal>.</literal> and ending with + <literal>.conf</literal> will be included. Multiple files within an include + directory are processed in file name order (according to C locale rules, + i.e., numbers before letters, and uppercase letters before lowercase ones). + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,21 +118,57 @@ <para> A record can have several formats: <synopsis> -local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +include <replaceable>file</replaceable> +include_if_exists <replaceable>file</replaceable> +include_dir <replaceable>directory</replaceable> +local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> </synopsis> The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_if_exists</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file if the + file exists and can be read. Otherwise, a message will be logged to + indicate that the file is skipped. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_dir</literal></term> + <listitem> + <para> + This line will be replaced with the content of all the files found in + the directory, if they don't start with a <literal>.</literal> and end + with <literal>.conf</literal>, processed in file name order (according + to C locale rules, i.e., numbers before letters, and uppercase letters + before lowercase ones). + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -835,8 +886,10 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> +<replaceable>include_dir</replaceable> <replaceable>directory</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -847,6 +900,11 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + As for <filename>pg_hba.conf</filename>, the lines in this file can either + be inclusion directives or user name map records, and follow the same + rules. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 1d619427c1..4e63dec74c 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -1003,12 +1003,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule the given <literal>file_name</literal> </para></entry> </row> @@ -1153,12 +1162,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this mapping + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_ident.conf</filename> + Line number of this mapping in the given <literal>file_name</literal> </para></entry> </row> diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index 4637426d62..071bf1ff95 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -22,6 +22,7 @@ #include <sys/param.h> #include <sys/socket.h> #include <netdb.h> +#include <sys/stat.h> #include <netinet/in.h> #include <arpa/inet.h> #include <unistd.h> @@ -69,6 +70,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -113,10 +120,22 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int depth, + int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, - const char *inc_filename, int elevel, char **err_msg); + const char *inc_filename, int depth, int elevel, + char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + bool strict, const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); +static char *process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, + int elevel, MemoryContext linecxt, + List **tok_lines); /* @@ -303,7 +322,7 @@ copy_auth_token(AuthToken *in) */ static List * next_field_expand(const char *filename, char **lineptr, - int elevel, char **err_msg) + int depth, int elevel, char **err_msg) { char buf[MAX_TOKEN]; bool trailing_comma; @@ -319,7 +338,7 @@ next_field_expand(const char *filename, char **lineptr, /* Is this referencing a file? */ if (!initial_quote && buf[0] == '@' && buf[1] != '\0') - tokens = tokenize_inc_file(tokens, filename, buf + 1, + tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1, elevel, err_msg); else tokens = lappend(tokens, make_auth_token(buf, initial_quote)); @@ -347,6 +366,7 @@ static List * tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, + int depth, int elevel, char **err_msg) { @@ -356,39 +376,30 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); + *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", *err_msg))); + return tokens; } - inc_file = AllocateFile(inc_fullname, "r"); - if (inc_file == NULL) - { - int save_errno = errno; + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true, + outer_filename, elevel, err_msg, &inc_fullname); - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); + if (inc_file == NULL) return tokens; - } /* There is possible recursion here if the file contains @ */ - linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); + linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1, + elevel); FreeFile(inc_file); pfree(inc_fullname); @@ -426,11 +437,38 @@ tokenize_inc_file(List *tokens, /* * tokenize_auth_file - * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a dedicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, + int depth, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, depth, + elevel); + + return linecxt; +} + +/* + * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see the struct definition * in libpq/hba.h. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -439,30 +477,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int depth, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -515,7 +545,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, { List *current_field; - current_field = next_field_expand(filename, &lineptr, + current_field = next_field_expand(filename, &lineptr, depth, elevel, &err_msg); /* add field to line, unless we are at EOL or comment start */ if (current_field != NIL) @@ -523,29 +553,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + AuthToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); + + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; + + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, true, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } + else if (strcmp(first->string, "include_dir") == 0) + { + char **filenames; + char *dir_name = second->string; + int num_filenames; + StringInfoData err_buf; + + filenames = GetDirConfFiles(dir_name, filename, elevel, + &num_filenames, &err_msg); + + if (!filenames) + { + /* We have the error in err_msg, simply process it */ + goto process_line; + } + + initStringInfo(&err_buf); + for (int i = 0; i < num_filenames; i++) + { + /* + * err_msg is used here as a temp buffer, it will be + * overwritten at the end of the loop with the + * cumulated errors, if any. + */ + err_msg = process_included_authfile(filenames[i], true, + filename, depth + 1, elevel, + linecxt, tok_lines); + + /* Cumulate errors if any. */ + if (err_msg) + { + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfoString(&err_buf, err_msg); + } + } + + /* + * If there were no errors, the line is fully processed, bypass + * the general TokenizedAuthLine processing. + */ + if (err_buf.len == 0) + goto next_line; + + /* Otherwise, process the cumulated errors, if any. */ + err_msg = err_buf.data; + } + else if (strcmp(first->string, "include_if_exists") == 0) + { + char *inc_filename; - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, false, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } } +process_line: + /* + * General processing: report the error if any and emit line to the + * TokenizedAuthLine + */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -855,7 +983,7 @@ do { \ errmsg("authentication option \"%s\" is only valid for authentication methods %s", \ optname, _(validmethods)), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \ optname, validmethods); \ return false; \ @@ -875,7 +1003,7 @@ do { \ errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname); \ return NULL; \ @@ -898,7 +1026,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("missing entry at end of line"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, file_name))); \ *err_msg = pstrdup("missing entry at end of line"); \ return NULL; \ } \ @@ -911,7 +1039,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("multiple values in ident field"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, file_name))); \ *err_msg = pstrdup("multiple values in ident field"); \ return NULL; \ } \ @@ -934,6 +1062,7 @@ HbaLine * parse_hba_line(TokenizedAuthLine *tok_line, int elevel) { int line_num = tok_line->line_num; + char *file_name = tok_line->file_name; char **err_msg = &tok_line->err_msg; char *str; struct addrinfo *gai_result; @@ -948,6 +1077,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -962,7 +1092,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for connection type"), errhint("Specify exactly one connection type per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for connection type"; return NULL; } @@ -990,7 +1120,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("hostssl record cannot match because SSL is disabled"), errhint("Set ssl = on in postgresql.conf."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostssl record cannot match because SSL is disabled"; } #else @@ -998,7 +1128,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("hostssl record cannot match because SSL is not supported by this build"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostssl record cannot match because SSL is not supported by this build"; #endif } @@ -1010,7 +1140,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("hostgssenc record cannot match because GSSAPI is not supported by this build"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostgssenc record cannot match because GSSAPI is not supported by this build"; #endif } @@ -1031,7 +1161,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid connection type \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid connection type \"%s\"", token->string); return NULL; } @@ -1044,7 +1174,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before database specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before database specification"; return NULL; } @@ -1064,7 +1194,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before role specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before role specification"; return NULL; } @@ -1086,7 +1216,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before IP address specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before IP address specification"; return NULL; } @@ -1098,7 +1228,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for host address"), errhint("Specify one address range per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for host address"; return NULL; } @@ -1157,7 +1287,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid IP address \"%s\": %s", str, gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid IP address \"%s\": %s", str, gai_strerror(ret)); if (gai_result) @@ -1177,7 +1307,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("specifying both host name and CIDR mask is invalid: \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("specifying both host name and CIDR mask is invalid: \"%s\"", token->string); return NULL; @@ -1191,7 +1321,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid CIDR mask in address \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid CIDR mask in address \"%s\"", token->string); return NULL; @@ -1211,7 +1341,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("end-of-line before netmask specification"), errhint("Specify an address range in CIDR notation, or provide a separate netmask."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before netmask specification"; return NULL; } @@ -1222,7 +1352,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("multiple values specified for netmask"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for netmask"; return NULL; } @@ -1237,7 +1367,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid IP mask \"%s\": %s", token->string, gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid IP mask \"%s\": %s", token->string, gai_strerror(ret)); if (gai_result) @@ -1256,7 +1386,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("IP address and mask do not match"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "IP address and mask do not match"; return NULL; } @@ -1272,7 +1402,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before authentication method"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before authentication method"; return NULL; } @@ -1284,7 +1414,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for authentication type"), errhint("Specify exactly one authentication type per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for authentication type"; return NULL; } @@ -1321,7 +1451,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("MD5 authentication is not supported when \"db_user_namespace\" is enabled"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "MD5 authentication is not supported when \"db_user_namespace\" is enabled"; return NULL; } @@ -1362,7 +1492,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid authentication method \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid authentication method \"%s\"", token->string); return NULL; @@ -1375,7 +1505,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid authentication method \"%s\": not supported by this build", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid authentication method \"%s\": not supported by this build", token->string); return NULL; @@ -1397,7 +1527,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("gssapi authentication is not supported on local sockets"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "gssapi authentication is not supported on local sockets"; return NULL; } @@ -1409,7 +1539,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("peer authentication is only supported on local sockets"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "peer authentication is only supported on local sockets"; return NULL; } @@ -1427,7 +1557,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cert authentication is only supported on hostssl connections"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cert authentication is only supported on hostssl connections"; return NULL; } @@ -1477,7 +1607,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("authentication option not in name=value format: %s", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("authentication option not in name=value format: %s", token->string); return NULL; @@ -1521,7 +1651,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"; return NULL; } @@ -1532,7 +1662,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"; return NULL; } @@ -1548,7 +1678,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cannot use ldapsearchattribute together with ldapsearchfilter"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cannot use ldapsearchattribute together with ldapsearchfilter"; return NULL; } @@ -1565,7 +1695,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("list of RADIUS servers cannot be empty"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "list of RADIUS servers cannot be empty"; return NULL; } @@ -1576,7 +1706,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("list of RADIUS secrets cannot be empty"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "list of RADIUS secrets cannot be empty"; return NULL; } @@ -1595,7 +1725,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiussecrets), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS secrets (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiussecrets), list_length(parsedline->radiusservers)); @@ -1611,7 +1741,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiusports), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS ports (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiusports), list_length(parsedline->radiusservers)); @@ -1627,7 +1757,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiusidentifiers), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS identifiers (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiusidentifiers), list_length(parsedline->radiusservers)); @@ -1662,6 +1792,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg) { int line_num = hbaline->linenumber; + char *file_name = hbaline->sourcefile; #ifdef USE_LDAP hbaline->ldapscope = LDAP_SCOPE_SUBTREE; @@ -1685,7 +1816,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientcert can only be configured for \"hostssl\" rows"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientcert can only be configured for \"hostssl\" rows"; return false; } @@ -1702,7 +1833,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientcert only accepts \"verify-full\" when using \"cert\" authentication"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientcert can only be set to \"verify-full\" when using \"cert\" authentication"; return false; } @@ -1715,7 +1846,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid value for clientcert: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } } @@ -1727,7 +1858,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientname can only be configured for \"hostssl\" rows"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientname can only be configured for \"hostssl\" rows"; return false; } @@ -1746,7 +1877,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid value for clientname: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } } @@ -1832,7 +1963,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid ldapscheme value: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); hbaline->ldapscheme = pstrdup(val); } else if (strcmp(name, "ldapserver") == 0) @@ -1850,7 +1981,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid LDAP port number: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid LDAP port number: \"%s\"", val); return false; } @@ -1944,7 +2075,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS server list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -1963,7 +2094,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not translate RADIUS server name \"%s\" to address: %s", (char *) lfirst(l), gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); if (gai_result) pg_freeaddrinfo_all(hints.ai_family, gai_result); @@ -1992,7 +2123,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS port list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid RADIUS port number: \"%s\"", val); return false; } @@ -2005,7 +2136,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid RADIUS port number: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2028,7 +2159,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS secret list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2050,7 +2181,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS identifiers list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2064,7 +2195,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("unrecognized authentication option name: \"%s\"", name), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("unrecognized authentication option name: \"%s\"", name); return false; @@ -2212,7 +2343,7 @@ load_hba(void) return false; } - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ @@ -2283,6 +2414,137 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + if (strict) + { + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + } + else + { + Assert(kind == IncludedAuthFile); + ereport(LOG, + (errmsg("skipping missing authentication file \"%s\"", + *inc_fullname))); + } + + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} + +/* + * Try to open an included file, and tokenize it using the given context. + * Returns NULL if no error happens during tokenization, otherwise the error. + */ +static char * +process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, int elevel, + MemoryContext linecxt, List **tok_lines) +{ + char *inc_fullname; + FILE *inc_file; + char *err_msg = NULL; + + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) + { + char *err_msg; + + err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", err_msg))); + return err_msg; + } + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict, + outer_filename, elevel, &err_msg, &inc_fullname); + + if (inc_file == NULL) + { + if (strict) + { + /* open_inc_file should have reported an error. */ + Assert(err_msg != NULL); + return err_msg; + } + else + return NULL; + } + else + { + /* No error message should have been reported. */ + Assert(err_msg == NULL); + } + + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, depth, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + return NULL; +} /* * Parse one tokenised line from the ident config file and store the result in @@ -2301,6 +2563,7 @@ load_hba(void) IdentLine * parse_ident_line(TokenizedAuthLine *tok_line, int elevel) { + char *file_name = tok_line->file_name; int line_num = tok_line->line_num; char **err_msg = &tok_line->err_msg; ListCell *field; @@ -2361,7 +2624,7 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid regular expression \"%s\": %s", parsedline->ident_user + 1, errstr), errcontext("line %d of configuration file \"%s\"", - line_num, IdentFileName))); + line_num, file_name))); *err_msg = psprintf("invalid regular expression \"%s\": %s", parsedline->ident_user + 1, errstr); @@ -2596,7 +2859,7 @@ load_ident(void) return false; } - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..7433050112 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,16 +9,27 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # -# local DATABASE USER METHOD [OPTIONS] -# host DATABASE USER ADDRESS METHOD [OPTIONS] -# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# local DATABASE USER METHOD [OPTIONS] +# host DATABASE USER ADDRESS METHOD [OPTIONS] +# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..8e3fa29135 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,23 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # -# MAPNAME SYSTEM-USERNAME PG-USERNAME +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index c9be4bff1f..15326a01e2 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,12 +26,12 @@ static ArrayType *get_hba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int rule_number, int lineno, HbaLine *hba, - const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int mapping_number, int lineno, IdentLine *ident, - const char *err_msg); + int mapping_number, const char *filename, + int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); @@ -159,7 +159,7 @@ get_hba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 10 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line @@ -168,7 +168,8 @@ get_hba_options(HbaLine *hba) * tuple_store: where to store data * tupdesc: tuple descriptor for the view * rule_number: unique rule identifier among all valid rules - * lineno: pg_hba.conf line number (must always be valid) + * filename: name of the file containing that line + * lineno: line number in that file (must always be valid) * hba: parsed line data (can be NULL, in which case err_msg should be set) * err_msg: error message (NULL if none) * @@ -177,7 +178,7 @@ get_hba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int rule_number, int lineno, HbaLine *hba, + int rule_number, const char *filename, int lineno, HbaLine *hba, const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -202,6 +203,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -345,7 +348,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -386,7 +389,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open configuration file \"%s\": %m", HbaFileName))); - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -407,8 +410,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) rule_number++; - fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->line_num, - hbaline, tok_line->err_msg); + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -445,7 +448,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) } /* Number of columns in pg_hba_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6 +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -454,7 +457,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) * tuple_store: where to store data * tupdesc: tuple descriptor for the view * mapping_number: unique rule identifier among all valid rules - * lineno: pg_ident.conf line number (must always be valid) + * filename: name of the file containing that line + * lineno: line number in that file (must always be valid) * ident: parsed line data (can be NULL, in which case err_msg should be set) * err_msg: error message (NULL if none) * @@ -463,8 +467,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int mapping_number, int lineno, IdentLine *ident, - const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -482,6 +486,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -494,7 +500,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -534,7 +540,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open usermap file \"%s\": %m", IdentFileName))); - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -556,7 +562,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) mapping_number++; fill_ident_line(tuple_store, tupdesc, mapping_number, - tok_line->line_num, identline, tok_line->err_msg); + tok_line->file_name, tok_line->line_num, identline, + tok_line->err_msg); } /* Free tokenizer memory */ diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l index 4f43430db4..ecfb4a2a28 100644 --- a/src/backend/utils/misc/guc-file.l +++ b/src/backend/utils/misc/guc-file.l @@ -345,6 +345,110 @@ GUC_flex_fatal(const char *msg) return 0; /* keep compiler quiet */ } +/* + * Returns the list of config files located in a directory, in alphabetical + * order. + * + * We don't check for recursion or too-deep nesting depth here, its up to the + * caller to take care of that. + */ +char ** +GetDirConfFiles(const char *includedir, const char *calling_file, int elevel, + int *num_filenames, char **err_msg) +{ + char *directory; + DIR *d; + struct dirent *de; + char **filenames; + int size_filenames; + + /* + * Reject directory name that is all-blank (including empty), as that + * leads to confusion --- we'd read the containing directory, typically + * resulting in recursive inclusion of the same file(s). + */ + if (strspn(includedir, " \t\r\n") == strlen(includedir)) + { + ereport(elevel, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("empty configuration directory name: \"%s\"", + includedir))); + *err_msg = "empty configuration directory name"; + return NULL; + } + + directory = AbsoluteConfigLocation(includedir, calling_file); + d = AllocateDir(directory); + if (d == NULL) + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not open configuration directory \"%s\": %m", + directory))); + *err_msg = psprintf("could not open directory \"%s\"", directory); + filenames = NULL; + goto cleanup; + } + + /* + * Read the directory and put the filenames in an array, so we can sort + * them prior to caller processing the contents. + */ + size_filenames = 32; + filenames = (char **) palloc(size_filenames * sizeof(char *)); + *num_filenames = 0; + + while ((de = ReadDir(d, directory)) != NULL) + { + PGFileType de_type; + char filename[MAXPGPATH]; + + /* + * Only parse files with names ending in ".conf". Explicitly reject + * files starting with ".". This excludes things like "." and "..", + * as well as typical hidden files, backup files, and editor debris. + */ + if (strlen(de->d_name) < 6) + continue; + if (de->d_name[0] == '.') + continue; + if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) + continue; + + join_path_components(filename, directory, de->d_name); + canonicalize_path(filename); + de_type = get_dirent_type(filename, de, true, elevel); + if (de_type == PGFILETYPE_ERROR) + { + *err_msg = psprintf("could not stat file \"%s\"", filename); + pfree(filenames); + filenames = NULL; + goto cleanup; + } + else if (de_type != PGFILETYPE_DIR) + { + /* Add file to array, increasing its size in blocks of 32 */ + if (*num_filenames >= size_filenames) + { + size_filenames += 32; + filenames = (char **) repalloc(filenames, + size_filenames * sizeof(char *)); + } + filenames[*num_filenames] = pstrdup(filename); + (*num_filenames)++; + } + } + + if (*num_filenames > 0) + qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp); + +cleanup: + if (d) + FreeDir(d); + pfree(directory); + return filenames; +} + /* * Read and parse a single configuration file. This function recurses * to handle "include" directives. @@ -606,127 +710,30 @@ ParseConfigDirectory(const char *includedir, ConfigVariable **head_p, ConfigVariable **tail_p) { - char *directory; - DIR *d; - struct dirent *de; + char *err_msg; char **filenames; int num_filenames; - int size_filenames; - bool status; - - /* - * Reject directory name that is all-blank (including empty), as that - * leads to confusion --- we'd read the containing directory, typically - * resulting in recursive inclusion of the same file(s). - */ - if (strspn(includedir, " \t\r\n") == strlen(includedir)) - { - ereport(elevel, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("empty configuration directory name: \"%s\"", - includedir))); - record_config_file_error("empty configuration directory name", - calling_file, calling_lineno, - head_p, tail_p); - return false; - } - - /* - * We don't check for recursion or too-deep nesting depth here; the - * subsequent calls to ParseConfigFile will take care of that. - */ - - directory = AbsoluteConfigLocation(includedir, calling_file); - d = AllocateDir(directory); - if (d == NULL) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open configuration directory \"%s\": %m", - directory))); - record_config_file_error(psprintf("could not open directory \"%s\"", - directory), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } - /* - * Read the directory and put the filenames in an array, so we can sort - * them prior to processing the contents. - */ - size_filenames = 32; - filenames = (char **) palloc(size_filenames * sizeof(char *)); - num_filenames = 0; + filenames = GetDirConfFiles(includedir, calling_file, elevel, + &num_filenames, &err_msg); - while ((de = ReadDir(d, directory)) != NULL) + if (!filenames) { - PGFileType de_type; - char filename[MAXPGPATH]; - - /* - * Only parse files with names ending in ".conf". Explicitly reject - * files starting with ".". This excludes things like "." and "..", - * as well as typical hidden files, backup files, and editor debris. - */ - if (strlen(de->d_name) < 6) - continue; - if (de->d_name[0] == '.') - continue; - if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) - continue; - - join_path_components(filename, directory, de->d_name); - canonicalize_path(filename); - de_type = get_dirent_type(filename, de, true, elevel); - if (de_type == PGFILETYPE_ERROR) - { - record_config_file_error(psprintf("could not stat file \"%s\"", - filename), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } - else if (de_type != PGFILETYPE_DIR) - { - /* Add file to array, increasing its size in blocks of 32 */ - if (num_filenames >= size_filenames) - { - size_filenames += 32; - filenames = (char **) repalloc(filenames, - size_filenames * sizeof(char *)); - } - filenames[num_filenames] = pstrdup(filename); - num_filenames++; - } + record_config_file_error(err_msg, calling_file, calling_lineno, head_p, + tail_p); + return false; } - if (num_filenames > 0) + for (int i = 0; i < num_filenames; i++) { - int i; - - qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp); - for (i = 0; i < num_filenames; i++) - { - if (!ParseConfigFile(filenames[i], true, - calling_file, calling_lineno, - depth, elevel, - head_p, tail_p)) - { - status = false; - goto cleanup; - } - } + if (!ParseConfigFile(filenames[i], true, + calling_file, calling_lineno, + depth, elevel, + head_p, tail_p)) + return false; } - status = true; -cleanup: - if (d) - FreeDir(d); - pfree(directory); - return status; + return true; } /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 060bafaced..e6a54cc3d6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6128,16 +6128,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o}', - proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '6250', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,text,text,text}', - proargmodes => '{o,o,o,o,o,o}', - proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index 90036f7bcd..0ea100d1b8 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -155,6 +156,7 @@ typedef struct AuthToken typedef struct TokenizedAuthLine { List *fields; /* List of lists of AuthTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ @@ -174,6 +176,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel); extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel); extern bool pg_isblank(const char c); extern MemoryContext tokenize_auth_file(const char *filename, FILE *file, - List **tok_lines, int elevel); + List **tok_lines, int depth, + int elevel); #endif /* HBA_H */ diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h index 45ae1b537f..cb3bbcf949 100644 --- a/src/include/utils/guc.h +++ b/src/include/utils/guc.h @@ -145,6 +145,8 @@ typedef struct ConfigVariable struct ConfigVariable *next; } ConfigVariable; +extern char **GetDirConfFiles(const char *includedir, const char *calling_file, + int elevel, int *num_filenames, char **err_msg); extern bool ParseConfigFile(const char *config_file, bool strict, const char *calling_file, int calling_lineno, int depth, int elevel, diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl new file mode 100644 index 0000000000..8eae72b8d4 --- /dev/null +++ b/src/test/authentication/t/003_file_inclusion.pl @@ -0,0 +1,657 @@ + +# Copyright (c) 2021-2022, PostgreSQL Global Development Group + +# Set of tests for authentication and pg_hba.conf inclusion. +# This test can only run with Unix-domain sockets. + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use Time::HiRes qw(usleep); +use IPC::Run qw(pump finish timer); +use Data::Dumper; + +if (!$use_unix_sockets) +{ + plan skip_all => + "authentication tests cannot run without Unix-domain sockets"; +} + +# stores the current line counter for each file. hba_rule and ident_rule are +# fake file names used for the global rule number for each auth view. +my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + +my $hba_file = 'subdir1/pg_hba_custom.conf'; +my $ident_file = 'subdir2/pg_ident_custom.conf'; + +# Initialize primary node +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init; +$node->start; + +my $data_dir = $node->data_dir; + +# Normalize the data directory for Windows +$data_dir =~ s/\/\.\//\//g; # reduce /./ to / +$data_dir =~ s/\/\//\//g; # reduce // to / +$data_dir =~ s/\/$//; # remove trailing / + + +# Add the given payload to the given relative HBA file of the given node. +# This function maintains the %cur_line metadata, so it has to be called in the +# expected inclusion evaluation order in order to keep it in sync. +# +# If the payload starts with "include" or "ignore", the function doesn't +# increase the general hba rule number. +# +# If an err_str is provided, it returns an arrayref containing the provided +# filename, the current line number in that file and the provided err_str. The +# err_str has to be a valid regex string. +# Otherwise it only returns the line number of the payload in the wanted file. +# This function has to be called in the expected inclusion evaluation order to +# keep the %cur_line information in sync. +sub add_hba_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'hba_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_hba_file_rules line + @tokens = split(/ /, $payload); + $tokens[1] = '{' . $tokens[1] . '}'; # database + $tokens[2] = '{' . $tokens[2] . '}'; # user_name + + # add empty address and netmask betweed user_name and auth_method + splice @tokens, 3, 0, ''; + splice @tokens, 3, 0, ''; + + # append empty options and error + push @tokens, ''; + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Add the given payload to the given relative ident file of the given node. +# Same as add_hba_line but for pg_ident files +sub add_ident_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'ident_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_ident_file_mappings line + @tokens = split(/ /, $payload); + + # append empty error + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Delete pg_hba.conf from the given node, add various entries to test the +# include infrastructure and then execute a reload to refresh it. +sub generate_valid_auth_files +{ + my $node = shift; + my $hba_expected = ''; + my $ident_expected = ''; + + # customise main auth file names + $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'"); + $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'"); + + # and make original ones invalid to be sure they're not used anywhere + $node->append_conf('pg_hba.conf', "some invalid line"); + $node->append_conf('pg_ident.conf', "some invalid line"); + + # pg_hba stuff + mkdir("$data_dir/subdir1"); + mkdir("$data_dir/hba_inc"); + mkdir("$data_dir/hba_inc_if"); + mkdir("$data_dir/hba_pos"); + + # Make sure we will still be able to connect + $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust'); + + # Add include data + add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf"); + $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject"); + + $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject"); + + add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject"); + # include is relative to current path + add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject"); + + # include_if_exists data + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none"); + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some"); + $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject"); + + # include_dir data + add_hba_line($node, "$hba_file", "include_dir ../hba_inc"); + add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included"); + $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject"); + $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject"); + + # secondary auth file + add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject'); + $node->append_conf('dbnames.conf', "db1"); + $node->append_conf('dbnames.conf', "db3"); + $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1) + . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1) + . '|local|{db1,db3}|{all}|||reject||'; + + # pg_ident stuff + mkdir("$data_dir/subdir2"); + mkdir("$data_dir/ident_inc"); + mkdir("$data_dir/ident_inc_if"); + mkdir("$data_dir/ident_pos"); + + # Add include data + add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf"); + $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar"); + + $ident_expected .= add_ident_line($node, "$ident_file", "test a b"); + + add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar"); + # include is relative to current path + add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar"); + + # include_if_exists data + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none"); + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some"); + $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar"); + + # include_dir data + add_ident_line($node, "$ident_file", "include_dir ../ident_inc"); + add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included"); + $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar"); + $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar"); + + $node->restart; + $node->connect_ok('dbname=postgres', + 'Connection ok after generating valid auth files'); + + return ($hba_expected, $ident_expected); +} + +# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal +# entries to allow authentication. +sub reset_auth_files +{ + my $node = shift; + + unlink("$data_dir/$hba_file"); + unlink("$data_dir/$ident_file"); + + %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + + return add_hba_line($node, "$hba_file", 'local all all trust'); +} + +# Generate a list of expected error regex for the given array of error +# conditions, as generated by add_hba_line/add_ident_line with an err_str. +# +# 2 regex are generated per array entry: one for the given err_str, and one for +# the expected line in the specific file. Since all lines are independant, +# there's no guarantee that a specific failure regex and the per-line regex +# will match the same error. Calling code should add at least one test with a +# single error to make sure that the line number / file name is correct. +# +# On top of that, an extra line is generated for the general failure to process +# the main auth file. +sub generate_log_err_patterns +{ + my $node = shift; + my $raw_errors = shift; + my $is_hba_err = shift; + my @errors; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + push @errors, qr/$err_str/; + + # Context messages with the file / line location aren't always emitted + if ($err_str !~ /maximum nesting depth exceeded/ and + $err_str !~ /could not open secondary authentication file/) + { + push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/ + } + } + + push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err); + + return \@errors; +} + +# Generate the expected output for the auth file view error reporting (file +# name, file line, error), for the given array of error conditions, as +# generated generated by add_hba_line/add_ident_line with an err_str. +sub generate_log_err_rows +{ + my $node = shift; + my $raw_errors = shift; + my $exp_rows = ''; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + $exp_rows .= "\n" if ($exp_rows ne ""); + + # Unescape regex patterns if any + $err_str =~ s/\\([\(\)])/$1/g; + $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str" + } + + return $exp_rows; +} + +# Reset the main auth files, append the given payload to the given config file, +# and check that the instance cannot start, raising the expected error line(s). +sub start_errors_like +{ + my $node = shift; + my $file = shift; + my $payload = shift; + my $pattern = shift; + my $should_fail = shift; + + reset_auth_files($node); + $node->append_conf($file, $payload); + + unlink($node->logfile); + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir, + '-l', $node->logfile, 'start'); + + if ($should_fail) + { + ok($ret != 0, "Cannot start postgres with faulty $file"); + } + else + { + ok($ret == 0, "postgres can start with faulty $file"); + } + + my $log_contents = slurp_file($node->logfile); + + foreach (@{$pattern}) + { + like($log_contents, + $_, + "Expected failure found in the logs"); + } + + if (not $should_fail) + { + # We can't simply call $node->stop here as the call is optimized out + # when the server isn't started with $node->start. + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', + $data_dir, 'stop', '-m', 'fast'); + ok($ret == 0, "Could stop postgres"); + } +} + +# We should be able to connect, and see an empty pg_ident.conf +is($node->psql( + 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'), + qq(0), + 'pg_ident.conf is empty'); + +############################################ +# part 1, test view reporting for valid data +############################################ +my ($exp_hba, $exp_ident) = generate_valid_auth_files($node); + +$node->connect_ok('dbname=postgres', 'Connection still ok'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_hba_file_rules'), + qq($exp_hba), + 'pg_hba_file_rules content is expected'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_ident_file_mappings'), + qq($exp_ident), + 'pg_ident_file_mappings content is expected'); + +############################################# +# part 2, test log reporting for invalid data +############################################# +reset_auth_files($node); +$node->restart('fast'); +$node->connect_ok('dbname=postgres', + 'Connection ok after resetting auth files'); + +$node->stop('fast'); + +start_errors_like($node, $hba_file, "include ../not_a_file", + [ + qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file +mkdir("$data_dir/hba_inc_fail"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token"); +start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail", + [ + qr/invalid connection type "not_a_token"/, + qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file with nested inclusion +unlink("$data_dir/hba_inc_fail/inc_dir.conf"); +my @hba_raw_errors_step1; + +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1"); + +add_hba_line($node, "hba_inc_fail/file1", "include file2"); +add_hba_line($node, "hba_inc_fail/file2", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file2", "include file3"); + +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3", + "local all all zuul", + 'invalid authentication method "zuul"'); + +start_errors_like( + $node, $hba_file, "include_dir ../hba_inc_fail", + generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @hba_raw_errors_step2; + +# include_if_exists, with various problems +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local", + "end-of-line before database specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local,host", + "multiple values specified for connection type"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all", + "end-of-line before role specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all all", + "end-of-line before authentication method"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "host all all test/42", + 'specifying both host name and CIDR mask is invalid: "test/42"'); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + 'local @dbnames_fails.conf all reject', + "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory"); + +add_hba_line($node, "hba_if_exists.conf", "include recurse.conf"); +push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf", + "include recurse.conf", + 'could not open configuration file "recurse.conf": maximum nesting depth exceeded'); + +# Generate the regex for the expected errors in the logs. There's no guarantee +# that the generated "line X of file..." will be emitted for the expected line, +# but previous tests already ensured that the correct line number / file name +# was emitted, so ensuring that there's an error in all expected lines is +# enough here. +my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2, + 1); + +# Not an error, but it should raise a message in the logs. Manually add an +# extra log message to detect +add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none"); +push @{$expected_errors}, + qr/skipping missing authentication file "$data_dir\/if_exists_none"/; + +start_errors_like( + $node, $hba_file, "include_if_exists ../hba_if_exists.conf", + $expected_errors, 1); + +# Mostly the same, but for ident files +reset_auth_files($node); + +my @ident_raw_errors_step1; + +# include_dir, single included file with nested inclusion +mkdir("$data_dir/ident_inc_fail"); +add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1"); + +add_ident_line($node, "ident_inc_fail/file1", "include file2"); +add_ident_line($node, "ident_inc_fail/file2", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file2", "include file3"); + +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3", + "failmap /(fail postgres", + 'invalid regular expression "\(fail": parentheses \(\) not balanced'); + +start_errors_like( + $node, $ident_file, "include_dir ../ident_inc_fail", + generate_log_err_patterns($node, \@ident_raw_errors_step1, 0), + 0); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @ident_raw_errors_step2; + +# include_if_exists, with various problems +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map", + "missing entry at end of line"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2", + "multiple values in ident field"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", + 'map @osnames_fails.conf postgres', + "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory"); + +add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf", + 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded'); + +start_errors_like( + $node, $ident_file, "include_if_exists ../ident_if_exists.conf", + # There's no guarantee that the generated "line X of file..." will be + # emitted for the expected line, but previous tests already ensured that + # the correct line number / file name was emitted, so ensuring that there's + # an error in all expected lines is enough here. + generate_log_err_patterns($node, \@ident_raw_errors_step2, 0), + 0); + +##################################################### +# part 3, test reporting of various error scenario +# NOTE: this will be bypassed -DEXEC_BACKEND or win32 +##################################################### +reset_auth_files($node); + +$node->start; +$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset'); + +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_hba_file_rules'); + +add_ident_line($node, $ident_file, ''); +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_ident_file_mappings'); + +# The instance could be restarted and no error is detected. Now check if the +# build is compatible with the view error reporting (EXEC_BACKEND / win32 will +# fail when trying to connect as they always rely on the current auth files +# content) +my @hba_raw_errors; + +push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +my ($stdout, $stderr); +my $cmdret = $node->psql('postgres', 'SELECT 1', + stdout => \$stdout, stderr => \$stderr); + +if ($cmdret != 0) +{ + # Connection failed. Bail out, but make sure to raise a failure if it + # didn't fail for the expected hba file modification. + like($stderr, + qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/, + "Connection failed due to loading an invalid hba file"); + + done_testing(); + diag("Build not compatible with auth file view error reporting, bail out.\n"); + exit; +} + +# Combine errors generated at step 2, in the same order. +$node->append_conf($hba_file, "include_dir ../hba_inc_fail"); +push @hba_raw_errors, @hba_raw_errors_step1; + +$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf"); +push @hba_raw_errors, @hba_raw_errors_step2; + +my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules' + . ' WHERE error IS NOT NULL ORDER BY rule_number'), + qq($hba_expected), + 'Detected all error in hba file'); + +# and do the same for pg_ident +my @ident_raw_errors; + +push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +$node->append_conf($ident_file, "include_dir ../ident_inc_fail"); +push @ident_raw_errors, @ident_raw_errors_step1; + +$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf"); +push @ident_raw_errors, @ident_raw_errors_step2; + +my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings' + . ' WHERE error IS NOT NULL ORDER BY mapping_number'), + qq($ident_expected), + 'Detected all error in ident file'); + +done_testing(); diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 79408710e0..5ed2fe3704 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname, FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); pg_hba_file_rules| SELECT a.rule_number, + a.file_name, a.line_number, a.type, a.database, @@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error); + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, a.line_number, a.map_name, a.sys_name, a.pg_username, a.error - FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.37.0 --fy3hxyvyv2spu3wq Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v10-0003-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v11 2/3] Allow file inclusion in pg_hba and pg_ident files. @ 2022-05-30 03:15 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-05-30 03:15 UTC (permalink / raw) pg_hba.conf file now has support for "include", "include_dir" and "include_if_exists" directives, which work similarly to the same directives in the postgresql.conf file. This fixes a possible crash if a secondary file tries to include itself as there's now a nesting depth check in the inclusion code path, same as the postgresql.conf. Many regression tests added to cover both the new directives, but also error detection for the whole pg_hba / pg_ident files. Catversion is bumped. Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/client-auth.sgml | 86 ++- doc/src/sgml/system-views.sgml | 22 +- src/backend/libpq/hba.c | 481 ++++++++++--- src/backend/libpq/pg_hba.conf.sample | 25 +- src/backend/libpq/pg_ident.conf.sample | 15 +- src/backend/utils/adt/hbafuncs.c | 43 +- src/backend/utils/misc/guc-file.l | 229 +++--- src/include/catalog/pg_proc.dat | 12 +- src/include/libpq/hba.h | 5 +- src/include/utils/guc.h | 2 + .../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++ src/test/regress/expected/rules.out | 6 +- 12 files changed, 1310 insertions(+), 273 deletions(-) create mode 100644 src/test/authentication/t/003_file_inclusion.pl diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index c6f1b70fd3..42ceb6f3e6 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,23 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication + record. Inclusion directives specify files that can be included, which + contains additional records. The records will be inserted in lieu of the + inclusion records. Those records only contains two fields: the + <literal>include</literal>, <literal>include_if_exists</literal> or + <literal>include_dir</literal> directive and the file or directory to be + included. The file or directory can be a relative of absolute path, and can + be double quoted if needed. For the <literal>include_dir</literal> form, + all files not starting with a <literal>.</literal> and ending with + <literal>.conf</literal> will be included. Multiple files within an include + directory are processed in file name order (according to C locale rules, + i.e., numbers before letters, and uppercase letters before lowercase ones). + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,21 +118,57 @@ <para> A record can have several formats: <synopsis> -local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +include <replaceable>file</replaceable> +include_if_exists <replaceable>file</replaceable> +include_dir <replaceable>directory</replaceable> +local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> </synopsis> The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_if_exists</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file if the + file exists and can be read. Otherwise, a message will be logged to + indicate that the file is skipped. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_dir</literal></term> + <listitem> + <para> + This line will be replaced with the content of all the files found in + the directory, if they don't start with a <literal>.</literal> and end + with <literal>.conf</literal>, processed in file name order (according + to C locale rules, i.e., numbers before letters, and uppercase letters + before lowercase ones). + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -835,8 +886,10 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> +<replaceable>include_dir</replaceable> <replaceable>directory</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -847,6 +900,11 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + As for <filename>pg_hba.conf</filename>, the lines in this file can either + be inclusion directives or user name map records, and follow the same + rules. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 4723f712a7..7d1cec8b7f 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -1003,12 +1003,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule the given <literal>file_name</literal> </para></entry> </row> @@ -1153,12 +1162,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this mapping + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_ident.conf</filename> + Line number of this mapping in the given <literal>file_name</literal> </para></entry> </row> diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index 4637426d62..4e4a45b793 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -22,6 +22,7 @@ #include <sys/param.h> #include <sys/socket.h> #include <netdb.h> +#include <sys/stat.h> #include <netinet/in.h> #include <arpa/inet.h> #include <unistd.h> @@ -69,6 +70,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -113,10 +120,22 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int depth, + int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, - const char *inc_filename, int elevel, char **err_msg); + const char *inc_filename, int depth, int elevel, + char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + bool strict, const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); +static char *process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, + int elevel, MemoryContext linecxt, + List **tok_lines); /* @@ -303,7 +322,7 @@ copy_auth_token(AuthToken *in) */ static List * next_field_expand(const char *filename, char **lineptr, - int elevel, char **err_msg) + int depth, int elevel, char **err_msg) { char buf[MAX_TOKEN]; bool trailing_comma; @@ -319,7 +338,7 @@ next_field_expand(const char *filename, char **lineptr, /* Is this referencing a file? */ if (!initial_quote && buf[0] == '@' && buf[1] != '\0') - tokens = tokenize_inc_file(tokens, filename, buf + 1, + tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1, elevel, err_msg); else tokens = lappend(tokens, make_auth_token(buf, initial_quote)); @@ -347,6 +366,7 @@ static List * tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, + int depth, int elevel, char **err_msg) { @@ -356,39 +376,30 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); + *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", *err_msg))); + return tokens; } - inc_file = AllocateFile(inc_fullname, "r"); - if (inc_file == NULL) - { - int save_errno = errno; + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true, + outer_filename, elevel, err_msg, &inc_fullname); - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); + if (inc_file == NULL) return tokens; - } /* There is possible recursion here if the file contains @ */ - linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); + linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1, + elevel); FreeFile(inc_file); pfree(inc_fullname); @@ -426,11 +437,38 @@ tokenize_inc_file(List *tokens, /* * tokenize_auth_file - * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a dedicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, + int depth, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, depth, + elevel); + + return linecxt; +} + +/* + * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see the struct definition * in libpq/hba.h. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -439,30 +477,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int depth, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -515,7 +545,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, { List *current_field; - current_field = next_field_expand(filename, &lineptr, + current_field = next_field_expand(filename, &lineptr, depth, elevel, &err_msg); /* add field to line, unless we are at EOL or comment start */ if (current_field != NIL) @@ -523,29 +553,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + AuthToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); + + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; + + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, true, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } + else if (strcmp(first->string, "include_dir") == 0) + { + char **filenames; + char *dir_name = second->string; + int num_filenames; + StringInfoData err_buf; + + filenames = GetDirConfFiles(dir_name, filename, elevel, + &num_filenames, &err_msg); + + if (!filenames) + { + /* We have the error in err_msg, simply process it */ + goto process_line; + } + + initStringInfo(&err_buf); + for (int i = 0; i < num_filenames; i++) + { + /* + * err_msg is used here as a temp buffer, it will be + * overwritten at the end of the loop with the + * cumulated errors, if any. + */ + err_msg = process_included_authfile(filenames[i], true, + filename, depth + 1, elevel, + linecxt, tok_lines); + + /* Cumulate errors if any. */ + if (err_msg) + { + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfoString(&err_buf, err_msg); + } + } + + /* + * If there were no errors, the line is fully processed, bypass + * the general TokenizedAuthLine processing. + */ + if (err_buf.len == 0) + goto next_line; + + /* Otherwise, process the cumulated errors, if any. */ + err_msg = err_buf.data; + } + else if (strcmp(first->string, "include_if_exists") == 0) + { + char *inc_filename; - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, false, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } } +process_line: + /* + * General processing: report the error if any and emit line to the + * TokenizedAuthLine + */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -855,7 +983,7 @@ do { \ errmsg("authentication option \"%s\" is only valid for authentication methods %s", \ optname, _(validmethods)), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \ optname, validmethods); \ return false; \ @@ -875,7 +1003,7 @@ do { \ errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname); \ return NULL; \ @@ -898,7 +1026,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("missing entry at end of line"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, file_name))); \ *err_msg = pstrdup("missing entry at end of line"); \ return NULL; \ } \ @@ -911,7 +1039,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("multiple values in ident field"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, file_name))); \ *err_msg = pstrdup("multiple values in ident field"); \ return NULL; \ } \ @@ -934,6 +1062,7 @@ HbaLine * parse_hba_line(TokenizedAuthLine *tok_line, int elevel) { int line_num = tok_line->line_num; + char *file_name = tok_line->file_name; char **err_msg = &tok_line->err_msg; char *str; struct addrinfo *gai_result; @@ -948,6 +1077,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -962,7 +1092,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for connection type"), errhint("Specify exactly one connection type per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for connection type"; return NULL; } @@ -990,7 +1120,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("hostssl record cannot match because SSL is disabled"), errhint("Set ssl = on in postgresql.conf."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostssl record cannot match because SSL is disabled"; } #else @@ -998,7 +1128,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("hostssl record cannot match because SSL is not supported by this build"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostssl record cannot match because SSL is not supported by this build"; #endif } @@ -1010,7 +1140,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("hostgssenc record cannot match because GSSAPI is not supported by this build"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostgssenc record cannot match because GSSAPI is not supported by this build"; #endif } @@ -1031,7 +1161,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid connection type \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid connection type \"%s\"", token->string); return NULL; } @@ -1044,7 +1174,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before database specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before database specification"; return NULL; } @@ -1064,7 +1194,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before role specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before role specification"; return NULL; } @@ -1086,7 +1216,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before IP address specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before IP address specification"; return NULL; } @@ -1098,7 +1228,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for host address"), errhint("Specify one address range per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for host address"; return NULL; } @@ -1157,7 +1287,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid IP address \"%s\": %s", str, gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid IP address \"%s\": %s", str, gai_strerror(ret)); if (gai_result) @@ -1177,7 +1307,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("specifying both host name and CIDR mask is invalid: \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("specifying both host name and CIDR mask is invalid: \"%s\"", token->string); return NULL; @@ -1191,7 +1321,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid CIDR mask in address \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid CIDR mask in address \"%s\"", token->string); return NULL; @@ -1211,7 +1341,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("end-of-line before netmask specification"), errhint("Specify an address range in CIDR notation, or provide a separate netmask."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before netmask specification"; return NULL; } @@ -1222,7 +1352,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("multiple values specified for netmask"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for netmask"; return NULL; } @@ -1237,7 +1367,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid IP mask \"%s\": %s", token->string, gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid IP mask \"%s\": %s", token->string, gai_strerror(ret)); if (gai_result) @@ -1256,7 +1386,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("IP address and mask do not match"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "IP address and mask do not match"; return NULL; } @@ -1272,7 +1402,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before authentication method"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before authentication method"; return NULL; } @@ -1284,7 +1414,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for authentication type"), errhint("Specify exactly one authentication type per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for authentication type"; return NULL; } @@ -1321,7 +1451,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("MD5 authentication is not supported when \"db_user_namespace\" is enabled"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "MD5 authentication is not supported when \"db_user_namespace\" is enabled"; return NULL; } @@ -1362,7 +1492,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid authentication method \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid authentication method \"%s\"", token->string); return NULL; @@ -1375,7 +1505,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid authentication method \"%s\": not supported by this build", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid authentication method \"%s\": not supported by this build", token->string); return NULL; @@ -1397,7 +1527,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("gssapi authentication is not supported on local sockets"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "gssapi authentication is not supported on local sockets"; return NULL; } @@ -1409,7 +1539,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("peer authentication is only supported on local sockets"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "peer authentication is only supported on local sockets"; return NULL; } @@ -1427,7 +1557,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cert authentication is only supported on hostssl connections"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cert authentication is only supported on hostssl connections"; return NULL; } @@ -1477,7 +1607,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("authentication option not in name=value format: %s", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("authentication option not in name=value format: %s", token->string); return NULL; @@ -1521,7 +1651,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"; return NULL; } @@ -1532,7 +1662,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"; return NULL; } @@ -1548,7 +1678,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cannot use ldapsearchattribute together with ldapsearchfilter"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cannot use ldapsearchattribute together with ldapsearchfilter"; return NULL; } @@ -1565,7 +1695,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("list of RADIUS servers cannot be empty"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "list of RADIUS servers cannot be empty"; return NULL; } @@ -1576,7 +1706,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("list of RADIUS secrets cannot be empty"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "list of RADIUS secrets cannot be empty"; return NULL; } @@ -1595,7 +1725,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiussecrets), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS secrets (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiussecrets), list_length(parsedline->radiusservers)); @@ -1611,7 +1741,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiusports), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS ports (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiusports), list_length(parsedline->radiusservers)); @@ -1627,7 +1757,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiusidentifiers), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS identifiers (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiusidentifiers), list_length(parsedline->radiusservers)); @@ -1662,6 +1792,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg) { int line_num = hbaline->linenumber; + char *file_name = hbaline->sourcefile; #ifdef USE_LDAP hbaline->ldapscope = LDAP_SCOPE_SUBTREE; @@ -1685,7 +1816,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientcert can only be configured for \"hostssl\" rows"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientcert can only be configured for \"hostssl\" rows"; return false; } @@ -1702,7 +1833,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientcert only accepts \"verify-full\" when using \"cert\" authentication"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientcert can only be set to \"verify-full\" when using \"cert\" authentication"; return false; } @@ -1715,7 +1846,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid value for clientcert: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } } @@ -1727,7 +1858,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientname can only be configured for \"hostssl\" rows"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientname can only be configured for \"hostssl\" rows"; return false; } @@ -1746,7 +1877,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid value for clientname: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } } @@ -1832,7 +1963,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid ldapscheme value: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); hbaline->ldapscheme = pstrdup(val); } else if (strcmp(name, "ldapserver") == 0) @@ -1850,7 +1981,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid LDAP port number: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid LDAP port number: \"%s\"", val); return false; } @@ -1944,7 +2075,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS server list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -1963,7 +2094,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not translate RADIUS server name \"%s\" to address: %s", (char *) lfirst(l), gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); if (gai_result) pg_freeaddrinfo_all(hints.ai_family, gai_result); @@ -1992,7 +2123,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS port list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid RADIUS port number: \"%s\"", val); return false; } @@ -2005,7 +2136,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid RADIUS port number: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2028,7 +2159,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS secret list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2050,7 +2181,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS identifiers list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2064,7 +2195,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("unrecognized authentication option name: \"%s\"", name), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("unrecognized authentication option name: \"%s\"", name); return false; @@ -2212,7 +2343,7 @@ load_hba(void) return false; } - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ @@ -2283,6 +2414,135 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + if (strict) + { + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + } + else + { + Assert(kind == IncludedAuthFile); + ereport(LOG, + (errmsg("skipping missing authentication file \"%s\"", + *inc_fullname))); + } + + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} + +/* + * Try to open an included file, and tokenize it using the given context. + * Returns NULL if no error happens during tokenization, otherwise the error. + */ +static char * +process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, int elevel, + MemoryContext linecxt, List **tok_lines) +{ + char *inc_fullname; + FILE *inc_file; + char *err_msg = NULL; + + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) + { + err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", err_msg))); + return err_msg; + } + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict, + outer_filename, elevel, &err_msg, &inc_fullname); + + if (inc_file == NULL) + { + if (strict) + { + /* open_inc_file should have reported an error. */ + Assert(err_msg != NULL); + return err_msg; + } + else + return NULL; + } + else + { + /* No error message should have been reported. */ + Assert(err_msg == NULL); + } + + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, depth, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + return NULL; +} /* * Parse one tokenised line from the ident config file and store the result in @@ -2301,6 +2561,7 @@ load_hba(void) IdentLine * parse_ident_line(TokenizedAuthLine *tok_line, int elevel) { + char *file_name = tok_line->file_name; int line_num = tok_line->line_num; char **err_msg = &tok_line->err_msg; ListCell *field; @@ -2361,7 +2622,7 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid regular expression \"%s\": %s", parsedline->ident_user + 1, errstr), errcontext("line %d of configuration file \"%s\"", - line_num, IdentFileName))); + line_num, file_name))); *err_msg = psprintf("invalid regular expression \"%s\": %s", parsedline->ident_user + 1, errstr); @@ -2596,7 +2857,7 @@ load_ident(void) return false; } - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..7433050112 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,16 +9,27 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # -# local DATABASE USER METHOD [OPTIONS] -# host DATABASE USER ADDRESS METHOD [OPTIONS] -# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# local DATABASE USER METHOD [OPTIONS] +# host DATABASE USER ADDRESS METHOD [OPTIONS] +# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..8e3fa29135 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,23 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # -# MAPNAME SYSTEM-USERNAME PG-USERNAME +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index c9be4bff1f..15326a01e2 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,12 +26,12 @@ static ArrayType *get_hba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int rule_number, int lineno, HbaLine *hba, - const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int mapping_number, int lineno, IdentLine *ident, - const char *err_msg); + int mapping_number, const char *filename, + int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); @@ -159,7 +159,7 @@ get_hba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 10 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line @@ -168,7 +168,8 @@ get_hba_options(HbaLine *hba) * tuple_store: where to store data * tupdesc: tuple descriptor for the view * rule_number: unique rule identifier among all valid rules - * lineno: pg_hba.conf line number (must always be valid) + * filename: name of the file containing that line + * lineno: line number in that file (must always be valid) * hba: parsed line data (can be NULL, in which case err_msg should be set) * err_msg: error message (NULL if none) * @@ -177,7 +178,7 @@ get_hba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int rule_number, int lineno, HbaLine *hba, + int rule_number, const char *filename, int lineno, HbaLine *hba, const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -202,6 +203,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -345,7 +348,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -386,7 +389,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open configuration file \"%s\": %m", HbaFileName))); - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -407,8 +410,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) rule_number++; - fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->line_num, - hbaline, tok_line->err_msg); + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -445,7 +448,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) } /* Number of columns in pg_hba_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6 +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -454,7 +457,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) * tuple_store: where to store data * tupdesc: tuple descriptor for the view * mapping_number: unique rule identifier among all valid rules - * lineno: pg_ident.conf line number (must always be valid) + * filename: name of the file containing that line + * lineno: line number in that file (must always be valid) * ident: parsed line data (can be NULL, in which case err_msg should be set) * err_msg: error message (NULL if none) * @@ -463,8 +467,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int mapping_number, int lineno, IdentLine *ident, - const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -482,6 +486,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -494,7 +500,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -534,7 +540,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open usermap file \"%s\": %m", IdentFileName))); - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -556,7 +562,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) mapping_number++; fill_ident_line(tuple_store, tupdesc, mapping_number, - tok_line->line_num, identline, tok_line->err_msg); + tok_line->file_name, tok_line->line_num, identline, + tok_line->err_msg); } /* Free tokenizer memory */ diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l index 721628c0cf..86b6cc1c8a 100644 --- a/src/backend/utils/misc/guc-file.l +++ b/src/backend/utils/misc/guc-file.l @@ -345,6 +345,110 @@ GUC_flex_fatal(const char *msg) return 0; /* keep compiler quiet */ } +/* + * Returns the list of config files located in a directory, in alphabetical + * order. + * + * We don't check for recursion or too-deep nesting depth here, its up to the + * caller to take care of that. + */ +char ** +GetDirConfFiles(const char *includedir, const char *calling_file, int elevel, + int *num_filenames, char **err_msg) +{ + char *directory; + DIR *d; + struct dirent *de; + char **filenames; + int size_filenames; + + /* + * Reject directory name that is all-blank (including empty), as that + * leads to confusion --- we'd read the containing directory, typically + * resulting in recursive inclusion of the same file(s). + */ + if (strspn(includedir, " \t\r\n") == strlen(includedir)) + { + ereport(elevel, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("empty configuration directory name: \"%s\"", + includedir))); + *err_msg = "empty configuration directory name"; + return NULL; + } + + directory = AbsoluteConfigLocation(includedir, calling_file); + d = AllocateDir(directory); + if (d == NULL) + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not open configuration directory \"%s\": %m", + directory))); + *err_msg = psprintf("could not open directory \"%s\"", directory); + filenames = NULL; + goto cleanup; + } + + /* + * Read the directory and put the filenames in an array, so we can sort + * them prior to caller processing the contents. + */ + size_filenames = 32; + filenames = (char **) palloc(size_filenames * sizeof(char *)); + *num_filenames = 0; + + while ((de = ReadDir(d, directory)) != NULL) + { + PGFileType de_type; + char filename[MAXPGPATH]; + + /* + * Only parse files with names ending in ".conf". Explicitly reject + * files starting with ".". This excludes things like "." and "..", + * as well as typical hidden files, backup files, and editor debris. + */ + if (strlen(de->d_name) < 6) + continue; + if (de->d_name[0] == '.') + continue; + if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) + continue; + + join_path_components(filename, directory, de->d_name); + canonicalize_path(filename); + de_type = get_dirent_type(filename, de, true, elevel); + if (de_type == PGFILETYPE_ERROR) + { + *err_msg = psprintf("could not stat file \"%s\"", filename); + pfree(filenames); + filenames = NULL; + goto cleanup; + } + else if (de_type != PGFILETYPE_DIR) + { + /* Add file to array, increasing its size in blocks of 32 */ + if (*num_filenames >= size_filenames) + { + size_filenames += 32; + filenames = (char **) repalloc(filenames, + size_filenames * sizeof(char *)); + } + filenames[*num_filenames] = pstrdup(filename); + (*num_filenames)++; + } + } + + if (*num_filenames > 0) + qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp); + +cleanup: + if (d) + FreeDir(d); + pfree(directory); + return filenames; +} + /* * Read and parse a single configuration file. This function recurses * to handle "include" directives. @@ -606,127 +710,30 @@ ParseConfigDirectory(const char *includedir, ConfigVariable **head_p, ConfigVariable **tail_p) { - char *directory; - DIR *d; - struct dirent *de; + char *err_msg; char **filenames; int num_filenames; - int size_filenames; - bool status; - - /* - * Reject directory name that is all-blank (including empty), as that - * leads to confusion --- we'd read the containing directory, typically - * resulting in recursive inclusion of the same file(s). - */ - if (strspn(includedir, " \t\r\n") == strlen(includedir)) - { - ereport(elevel, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("empty configuration directory name: \"%s\"", - includedir))); - record_config_file_error("empty configuration directory name", - calling_file, calling_lineno, - head_p, tail_p); - return false; - } - - /* - * We don't check for recursion or too-deep nesting depth here; the - * subsequent calls to ParseConfigFile will take care of that. - */ - - directory = AbsoluteConfigLocation(includedir, calling_file); - d = AllocateDir(directory); - if (d == NULL) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open configuration directory \"%s\": %m", - directory))); - record_config_file_error(psprintf("could not open directory \"%s\"", - directory), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } - /* - * Read the directory and put the filenames in an array, so we can sort - * them prior to processing the contents. - */ - size_filenames = 32; - filenames = (char **) palloc(size_filenames * sizeof(char *)); - num_filenames = 0; + filenames = GetDirConfFiles(includedir, calling_file, elevel, + &num_filenames, &err_msg); - while ((de = ReadDir(d, directory)) != NULL) + if (!filenames) { - PGFileType de_type; - char filename[MAXPGPATH]; - - /* - * Only parse files with names ending in ".conf". Explicitly reject - * files starting with ".". This excludes things like "." and "..", - * as well as typical hidden files, backup files, and editor debris. - */ - if (strlen(de->d_name) < 6) - continue; - if (de->d_name[0] == '.') - continue; - if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) - continue; - - join_path_components(filename, directory, de->d_name); - canonicalize_path(filename); - de_type = get_dirent_type(filename, de, true, elevel); - if (de_type == PGFILETYPE_ERROR) - { - record_config_file_error(psprintf("could not stat file \"%s\"", - filename), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } - else if (de_type != PGFILETYPE_DIR) - { - /* Add file to array, increasing its size in blocks of 32 */ - if (num_filenames >= size_filenames) - { - size_filenames += 32; - filenames = (char **) repalloc(filenames, - size_filenames * sizeof(char *)); - } - filenames[num_filenames] = pstrdup(filename); - num_filenames++; - } + record_config_file_error(err_msg, calling_file, calling_lineno, head_p, + tail_p); + return false; } - if (num_filenames > 0) + for (int i = 0; i < num_filenames; i++) { - int i; - - qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp); - for (i = 0; i < num_filenames; i++) - { - if (!ParseConfigFile(filenames[i], true, - calling_file, calling_lineno, - depth, elevel, - head_p, tail_p)) - { - status = false; - goto cleanup; - } - } + if (!ParseConfigFile(filenames[i], true, + calling_file, calling_lineno, + depth, elevel, + head_p, tail_p)) + return false; } - status = true; -cleanup: - if (d) - FreeDir(d); - pfree(directory); - return status; + return true; } /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index e10d5124ab..807fe3890b 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6131,16 +6131,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o}', - proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '6250', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,text,text,text}', - proargmodes => '{o,o,o,o,o,o}', - proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index d06da81806..b45b5cff38 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -155,6 +156,7 @@ typedef struct AuthToken typedef struct TokenizedAuthLine { List *fields; /* List of lists of AuthTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ @@ -174,6 +176,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel); extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel); extern bool pg_isblank(const char c); extern MemoryContext tokenize_auth_file(const char *filename, FILE *file, - List **tok_lines, int elevel); + List **tok_lines, int depth, + int elevel); #endif /* HBA_H */ diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h index 1788361974..b465dc0356 100644 --- a/src/include/utils/guc.h +++ b/src/include/utils/guc.h @@ -144,6 +144,8 @@ typedef struct ConfigVariable struct ConfigVariable *next; } ConfigVariable; +extern char **GetDirConfFiles(const char *includedir, const char *calling_file, + int elevel, int *num_filenames, char **err_msg); extern bool ParseConfigFile(const char *config_file, bool strict, const char *calling_file, int calling_lineno, int depth, int elevel, diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl new file mode 100644 index 0000000000..8eae72b8d4 --- /dev/null +++ b/src/test/authentication/t/003_file_inclusion.pl @@ -0,0 +1,657 @@ + +# Copyright (c) 2021-2022, PostgreSQL Global Development Group + +# Set of tests for authentication and pg_hba.conf inclusion. +# This test can only run with Unix-domain sockets. + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use Time::HiRes qw(usleep); +use IPC::Run qw(pump finish timer); +use Data::Dumper; + +if (!$use_unix_sockets) +{ + plan skip_all => + "authentication tests cannot run without Unix-domain sockets"; +} + +# stores the current line counter for each file. hba_rule and ident_rule are +# fake file names used for the global rule number for each auth view. +my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + +my $hba_file = 'subdir1/pg_hba_custom.conf'; +my $ident_file = 'subdir2/pg_ident_custom.conf'; + +# Initialize primary node +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init; +$node->start; + +my $data_dir = $node->data_dir; + +# Normalize the data directory for Windows +$data_dir =~ s/\/\.\//\//g; # reduce /./ to / +$data_dir =~ s/\/\//\//g; # reduce // to / +$data_dir =~ s/\/$//; # remove trailing / + + +# Add the given payload to the given relative HBA file of the given node. +# This function maintains the %cur_line metadata, so it has to be called in the +# expected inclusion evaluation order in order to keep it in sync. +# +# If the payload starts with "include" or "ignore", the function doesn't +# increase the general hba rule number. +# +# If an err_str is provided, it returns an arrayref containing the provided +# filename, the current line number in that file and the provided err_str. The +# err_str has to be a valid regex string. +# Otherwise it only returns the line number of the payload in the wanted file. +# This function has to be called in the expected inclusion evaluation order to +# keep the %cur_line information in sync. +sub add_hba_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'hba_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_hba_file_rules line + @tokens = split(/ /, $payload); + $tokens[1] = '{' . $tokens[1] . '}'; # database + $tokens[2] = '{' . $tokens[2] . '}'; # user_name + + # add empty address and netmask betweed user_name and auth_method + splice @tokens, 3, 0, ''; + splice @tokens, 3, 0, ''; + + # append empty options and error + push @tokens, ''; + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Add the given payload to the given relative ident file of the given node. +# Same as add_hba_line but for pg_ident files +sub add_ident_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'ident_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_ident_file_mappings line + @tokens = split(/ /, $payload); + + # append empty error + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Delete pg_hba.conf from the given node, add various entries to test the +# include infrastructure and then execute a reload to refresh it. +sub generate_valid_auth_files +{ + my $node = shift; + my $hba_expected = ''; + my $ident_expected = ''; + + # customise main auth file names + $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'"); + $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'"); + + # and make original ones invalid to be sure they're not used anywhere + $node->append_conf('pg_hba.conf', "some invalid line"); + $node->append_conf('pg_ident.conf', "some invalid line"); + + # pg_hba stuff + mkdir("$data_dir/subdir1"); + mkdir("$data_dir/hba_inc"); + mkdir("$data_dir/hba_inc_if"); + mkdir("$data_dir/hba_pos"); + + # Make sure we will still be able to connect + $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust'); + + # Add include data + add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf"); + $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject"); + + $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject"); + + add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject"); + # include is relative to current path + add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject"); + + # include_if_exists data + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none"); + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some"); + $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject"); + + # include_dir data + add_hba_line($node, "$hba_file", "include_dir ../hba_inc"); + add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included"); + $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject"); + $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject"); + + # secondary auth file + add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject'); + $node->append_conf('dbnames.conf', "db1"); + $node->append_conf('dbnames.conf', "db3"); + $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1) + . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1) + . '|local|{db1,db3}|{all}|||reject||'; + + # pg_ident stuff + mkdir("$data_dir/subdir2"); + mkdir("$data_dir/ident_inc"); + mkdir("$data_dir/ident_inc_if"); + mkdir("$data_dir/ident_pos"); + + # Add include data + add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf"); + $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar"); + + $ident_expected .= add_ident_line($node, "$ident_file", "test a b"); + + add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar"); + # include is relative to current path + add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar"); + + # include_if_exists data + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none"); + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some"); + $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar"); + + # include_dir data + add_ident_line($node, "$ident_file", "include_dir ../ident_inc"); + add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included"); + $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar"); + $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar"); + + $node->restart; + $node->connect_ok('dbname=postgres', + 'Connection ok after generating valid auth files'); + + return ($hba_expected, $ident_expected); +} + +# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal +# entries to allow authentication. +sub reset_auth_files +{ + my $node = shift; + + unlink("$data_dir/$hba_file"); + unlink("$data_dir/$ident_file"); + + %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + + return add_hba_line($node, "$hba_file", 'local all all trust'); +} + +# Generate a list of expected error regex for the given array of error +# conditions, as generated by add_hba_line/add_ident_line with an err_str. +# +# 2 regex are generated per array entry: one for the given err_str, and one for +# the expected line in the specific file. Since all lines are independant, +# there's no guarantee that a specific failure regex and the per-line regex +# will match the same error. Calling code should add at least one test with a +# single error to make sure that the line number / file name is correct. +# +# On top of that, an extra line is generated for the general failure to process +# the main auth file. +sub generate_log_err_patterns +{ + my $node = shift; + my $raw_errors = shift; + my $is_hba_err = shift; + my @errors; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + push @errors, qr/$err_str/; + + # Context messages with the file / line location aren't always emitted + if ($err_str !~ /maximum nesting depth exceeded/ and + $err_str !~ /could not open secondary authentication file/) + { + push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/ + } + } + + push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err); + + return \@errors; +} + +# Generate the expected output for the auth file view error reporting (file +# name, file line, error), for the given array of error conditions, as +# generated generated by add_hba_line/add_ident_line with an err_str. +sub generate_log_err_rows +{ + my $node = shift; + my $raw_errors = shift; + my $exp_rows = ''; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + $exp_rows .= "\n" if ($exp_rows ne ""); + + # Unescape regex patterns if any + $err_str =~ s/\\([\(\)])/$1/g; + $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str" + } + + return $exp_rows; +} + +# Reset the main auth files, append the given payload to the given config file, +# and check that the instance cannot start, raising the expected error line(s). +sub start_errors_like +{ + my $node = shift; + my $file = shift; + my $payload = shift; + my $pattern = shift; + my $should_fail = shift; + + reset_auth_files($node); + $node->append_conf($file, $payload); + + unlink($node->logfile); + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir, + '-l', $node->logfile, 'start'); + + if ($should_fail) + { + ok($ret != 0, "Cannot start postgres with faulty $file"); + } + else + { + ok($ret == 0, "postgres can start with faulty $file"); + } + + my $log_contents = slurp_file($node->logfile); + + foreach (@{$pattern}) + { + like($log_contents, + $_, + "Expected failure found in the logs"); + } + + if (not $should_fail) + { + # We can't simply call $node->stop here as the call is optimized out + # when the server isn't started with $node->start. + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', + $data_dir, 'stop', '-m', 'fast'); + ok($ret == 0, "Could stop postgres"); + } +} + +# We should be able to connect, and see an empty pg_ident.conf +is($node->psql( + 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'), + qq(0), + 'pg_ident.conf is empty'); + +############################################ +# part 1, test view reporting for valid data +############################################ +my ($exp_hba, $exp_ident) = generate_valid_auth_files($node); + +$node->connect_ok('dbname=postgres', 'Connection still ok'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_hba_file_rules'), + qq($exp_hba), + 'pg_hba_file_rules content is expected'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_ident_file_mappings'), + qq($exp_ident), + 'pg_ident_file_mappings content is expected'); + +############################################# +# part 2, test log reporting for invalid data +############################################# +reset_auth_files($node); +$node->restart('fast'); +$node->connect_ok('dbname=postgres', + 'Connection ok after resetting auth files'); + +$node->stop('fast'); + +start_errors_like($node, $hba_file, "include ../not_a_file", + [ + qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file +mkdir("$data_dir/hba_inc_fail"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token"); +start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail", + [ + qr/invalid connection type "not_a_token"/, + qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file with nested inclusion +unlink("$data_dir/hba_inc_fail/inc_dir.conf"); +my @hba_raw_errors_step1; + +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1"); + +add_hba_line($node, "hba_inc_fail/file1", "include file2"); +add_hba_line($node, "hba_inc_fail/file2", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file2", "include file3"); + +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3", + "local all all zuul", + 'invalid authentication method "zuul"'); + +start_errors_like( + $node, $hba_file, "include_dir ../hba_inc_fail", + generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @hba_raw_errors_step2; + +# include_if_exists, with various problems +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local", + "end-of-line before database specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local,host", + "multiple values specified for connection type"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all", + "end-of-line before role specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all all", + "end-of-line before authentication method"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "host all all test/42", + 'specifying both host name and CIDR mask is invalid: "test/42"'); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + 'local @dbnames_fails.conf all reject', + "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory"); + +add_hba_line($node, "hba_if_exists.conf", "include recurse.conf"); +push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf", + "include recurse.conf", + 'could not open configuration file "recurse.conf": maximum nesting depth exceeded'); + +# Generate the regex for the expected errors in the logs. There's no guarantee +# that the generated "line X of file..." will be emitted for the expected line, +# but previous tests already ensured that the correct line number / file name +# was emitted, so ensuring that there's an error in all expected lines is +# enough here. +my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2, + 1); + +# Not an error, but it should raise a message in the logs. Manually add an +# extra log message to detect +add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none"); +push @{$expected_errors}, + qr/skipping missing authentication file "$data_dir\/if_exists_none"/; + +start_errors_like( + $node, $hba_file, "include_if_exists ../hba_if_exists.conf", + $expected_errors, 1); + +# Mostly the same, but for ident files +reset_auth_files($node); + +my @ident_raw_errors_step1; + +# include_dir, single included file with nested inclusion +mkdir("$data_dir/ident_inc_fail"); +add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1"); + +add_ident_line($node, "ident_inc_fail/file1", "include file2"); +add_ident_line($node, "ident_inc_fail/file2", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file2", "include file3"); + +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3", + "failmap /(fail postgres", + 'invalid regular expression "\(fail": parentheses \(\) not balanced'); + +start_errors_like( + $node, $ident_file, "include_dir ../ident_inc_fail", + generate_log_err_patterns($node, \@ident_raw_errors_step1, 0), + 0); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @ident_raw_errors_step2; + +# include_if_exists, with various problems +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map", + "missing entry at end of line"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2", + "multiple values in ident field"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", + 'map @osnames_fails.conf postgres', + "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory"); + +add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf", + 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded'); + +start_errors_like( + $node, $ident_file, "include_if_exists ../ident_if_exists.conf", + # There's no guarantee that the generated "line X of file..." will be + # emitted for the expected line, but previous tests already ensured that + # the correct line number / file name was emitted, so ensuring that there's + # an error in all expected lines is enough here. + generate_log_err_patterns($node, \@ident_raw_errors_step2, 0), + 0); + +##################################################### +# part 3, test reporting of various error scenario +# NOTE: this will be bypassed -DEXEC_BACKEND or win32 +##################################################### +reset_auth_files($node); + +$node->start; +$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset'); + +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_hba_file_rules'); + +add_ident_line($node, $ident_file, ''); +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_ident_file_mappings'); + +# The instance could be restarted and no error is detected. Now check if the +# build is compatible with the view error reporting (EXEC_BACKEND / win32 will +# fail when trying to connect as they always rely on the current auth files +# content) +my @hba_raw_errors; + +push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +my ($stdout, $stderr); +my $cmdret = $node->psql('postgres', 'SELECT 1', + stdout => \$stdout, stderr => \$stderr); + +if ($cmdret != 0) +{ + # Connection failed. Bail out, but make sure to raise a failure if it + # didn't fail for the expected hba file modification. + like($stderr, + qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/, + "Connection failed due to loading an invalid hba file"); + + done_testing(); + diag("Build not compatible with auth file view error reporting, bail out.\n"); + exit; +} + +# Combine errors generated at step 2, in the same order. +$node->append_conf($hba_file, "include_dir ../hba_inc_fail"); +push @hba_raw_errors, @hba_raw_errors_step1; + +$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf"); +push @hba_raw_errors, @hba_raw_errors_step2; + +my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules' + . ' WHERE error IS NOT NULL ORDER BY rule_number'), + qq($hba_expected), + 'Detected all error in hba file'); + +# and do the same for pg_ident +my @ident_raw_errors; + +push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +$node->append_conf($ident_file, "include_dir ../ident_inc_fail"); +push @ident_raw_errors, @ident_raw_errors_step1; + +$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf"); +push @ident_raw_errors, @ident_raw_errors_step2; + +my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings' + . ' WHERE error IS NOT NULL ORDER BY mapping_number'), + qq($ident_expected), + 'Detected all error in ident file'); + +done_testing(); diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 6e29a6acc8..ef17bf1e8a 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname, FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); pg_hba_file_rules| SELECT a.rule_number, + a.file_name, a.line_number, a.type, a.database, @@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error); + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, a.line_number, a.map_name, a.sys_name, a.pg_username, a.error - FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.37.0 --z5mbrmnmd43fw5pt Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v11-0003-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v9 2/3] Allow file inclusion in pg_hba and pg_ident files. @ 2022-05-30 03:15 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-05-30 03:15 UTC (permalink / raw) pg_hba.conf file now has support for "include", "include_dir" and "include_if_exists" directives, which work similarly to the same directives in the postgresql.conf file. This fixes a possible crash if a secondary file tries to include itself as there's now a nesting depth check in the inclusion code path, same as the postgresql.conf. Many regression tests added to cover both the new directives, but also error detection for the whole pg_hba / pg_ident files. Catversion is bumped. Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/client-auth.sgml | 86 ++- doc/src/sgml/system-views.sgml | 22 +- src/backend/libpq/hba.c | 483 ++++++++++--- src/backend/libpq/pg_hba.conf.sample | 25 +- src/backend/libpq/pg_ident.conf.sample | 15 +- src/backend/utils/adt/hbafuncs.c | 43 +- src/backend/utils/misc/guc-file.l | 250 +++---- src/include/catalog/pg_proc.dat | 12 +- src/include/libpq/hba.h | 5 +- src/include/utils/guc.h | 2 + .../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++ src/test/regress/expected/rules.out | 6 +- 12 files changed, 1324 insertions(+), 282 deletions(-) create mode 100644 src/test/authentication/t/003_file_inclusion.pl diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index 433759928b..e4eacab4a5 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,23 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication + record. Inclusion directives specify files that can be included, which + contains additional records. The records will be inserted in lieu of the + inclusion records. Those records only contains two fields: the + <literal>include</literal>, <literal>include_if_exists</literal> or + <literal>include_dir</literal> directive and the file or directory to be + included. The file or directory can be a relative of absolute path, and can + be double quoted if needed. For the <literal>include_dir</literal> form, + all files not starting with a <literal>.</literal> and ending with + <literal>.conf</literal> will be included. Multiple files within an include + directory are processed in file name order (according to C locale rules, + i.e., numbers before letters, and uppercase letters before lowercase ones). + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,21 +118,57 @@ <para> A record can have several formats: <synopsis> -local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +include <replaceable>file</replaceable> +include_if_exists <replaceable>file</replaceable> +include_dir <replaceable>directory</replaceable> +local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> </synopsis> The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_if_exists</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file if the + file exists and can be read. Otherwise, a message will be logged to + indicate that the file is skipped. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_dir</literal></term> + <listitem> + <para> + This line will be replaced with the content of all the files found in + the directory, if they don't start with a <literal>.</literal> and end + with <literal>.conf</literal>, processed in file name order (according + to C locale rules, i.e., numbers before letters, and uppercase letters + before lowercase ones). + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -837,8 +888,10 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> +<replaceable>include_dir</replaceable> <replaceable>directory</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -849,6 +902,11 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + As for <filename>pg_hba.conf</filename>, the lines in this file can either + be inclusion directives or user name map records, and follow the same + rules. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 1d619427c1..4e63dec74c 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -1003,12 +1003,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule the given <literal>file_name</literal> </para></entry> </row> @@ -1153,12 +1162,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this mapping + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_ident.conf</filename> + Line number of this mapping in the given <literal>file_name</literal> </para></entry> </row> diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index 857b9e5eb2..49a8c56f41 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -22,6 +22,7 @@ #include <sys/param.h> #include <sys/socket.h> #include <netdb.h> +#include <sys/stat.h> #include <netinet/in.h> #include <arpa/inet.h> #include <unistd.h> @@ -69,6 +70,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -113,10 +120,22 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int depth, + int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, - const char *inc_filename, int elevel, char **err_msg); + const char *inc_filename, int depth, int elevel, + char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + bool strict, const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); +static char *process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, + int elevel, MemoryContext linecxt, + List **tok_lines); /* @@ -303,7 +322,7 @@ copy_auth_token(AuthToken *in) */ static List * next_field_expand(const char *filename, char **lineptr, - int elevel, char **err_msg) + int depth, int elevel, char **err_msg) { char buf[MAX_TOKEN]; bool trailing_comma; @@ -319,7 +338,7 @@ next_field_expand(const char *filename, char **lineptr, /* Is this referencing a file? */ if (!initial_quote && buf[0] == '@' && buf[1] != '\0') - tokens = tokenize_inc_file(tokens, filename, buf + 1, + tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1, elevel, err_msg); else tokens = lappend(tokens, make_auth_token(buf, initial_quote)); @@ -347,6 +366,7 @@ static List * tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, + int depth, int elevel, char **err_msg) { @@ -356,39 +376,30 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); + *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", *err_msg))); + return tokens; } - inc_file = AllocateFile(inc_fullname, "r"); - if (inc_file == NULL) - { - int save_errno = errno; + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true, + outer_filename, elevel, err_msg, &inc_fullname); - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); + if (inc_file == NULL) return tokens; - } /* There is possible recursion here if the file contains @ */ - linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); + linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1, + elevel); FreeFile(inc_file); pfree(inc_fullname); @@ -426,11 +437,38 @@ tokenize_inc_file(List *tokens, /* * tokenize_auth_file - * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a dedicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, + int depth, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, depth, + elevel); + + return linecxt; +} + +/* + * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see the struct definition * in libpq/hba.h. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -439,30 +477,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int depth, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -515,7 +545,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, { List *current_field; - current_field = next_field_expand(filename, &lineptr, + current_field = next_field_expand(filename, &lineptr, depth, elevel, &err_msg); /* add field to line, unless we are at EOL or comment start */ if (current_field != NIL) @@ -523,29 +553,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + AuthToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); + + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; + + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, true, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } + else if (strcmp(first->string, "include_dir") == 0) + { + char **filenames; + char *dir_name = second->string; + int num_filenames; + StringInfoData err_buf; + + filenames = GetDirConfFiles(dir_name, filename, elevel, + &num_filenames, &err_msg); + + if (!filenames) + { + /* We have the error in err_msg, simply process it */ + goto process_line; + } + + initStringInfo(&err_buf); + for (int i = 0; i < num_filenames; i++) + { + /* + * err_msg is used here as a temp buffer, it will be + * overwritten at the end of the loop with the + * cumulated errors, if any. + */ + err_msg = process_included_authfile(filenames[i], true, + filename, depth + 1, elevel, + linecxt, tok_lines); + + /* Cumulate errors if any. */ + if (err_msg) + { + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfoString(&err_buf, err_msg); + } + } + + /* + * If there were no errors, the line is fully processed, bypass + * the general TokenizedAuthLine processing. + */ + if (err_buf.len == 0) + goto next_line; + + /* Otherwise, process the cumulated errors, if any. */ + err_msg = err_buf.data; + } + else if (strcmp(first->string, "include_if_exists") == 0) + { + char *inc_filename; - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, false, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } } +process_line: + /* + * General processing: report the error if any and emit line to the + * TokenizedAuthLine + */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -860,7 +988,7 @@ do { \ errmsg("authentication option \"%s\" is only valid for authentication methods %s", \ optname, _(validmethods)), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \ optname, validmethods); \ return false; \ @@ -880,7 +1008,7 @@ do { \ errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname); \ return NULL; \ @@ -903,7 +1031,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("missing entry at end of line"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("missing entry at end of line"); \ return NULL; \ } \ @@ -916,7 +1044,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("multiple values in ident field"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("multiple values in ident field"); \ return NULL; \ } \ @@ -939,6 +1067,7 @@ HbaLine * parse_hba_line(TokenizedAuthLine *tok_line, int elevel) { int line_num = tok_line->line_num; + char *file_name = tok_line->file_name; char **err_msg = &tok_line->err_msg; char *str; struct addrinfo *gai_result; @@ -953,6 +1082,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -967,7 +1097,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for connection type"), errhint("Specify exactly one connection type per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for connection type"; return NULL; } @@ -995,7 +1125,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("hostssl record cannot match because SSL is disabled"), errhint("Set ssl = on in postgresql.conf."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostssl record cannot match because SSL is disabled"; } #else @@ -1003,7 +1133,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("hostssl record cannot match because SSL is not supported by this build"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostssl record cannot match because SSL is not supported by this build"; #endif } @@ -1015,7 +1145,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("hostgssenc record cannot match because GSSAPI is not supported by this build"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostgssenc record cannot match because GSSAPI is not supported by this build"; #endif } @@ -1036,7 +1166,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid connection type \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid connection type \"%s\"", token->string); return NULL; } @@ -1049,7 +1179,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before database specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before database specification"; return NULL; } @@ -1069,7 +1199,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before role specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before role specification"; return NULL; } @@ -1091,7 +1221,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before IP address specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before IP address specification"; return NULL; } @@ -1103,7 +1233,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for host address"), errhint("Specify one address range per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for host address"; return NULL; } @@ -1162,7 +1292,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid IP address \"%s\": %s", str, gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid IP address \"%s\": %s", str, gai_strerror(ret)); if (gai_result) @@ -1182,7 +1312,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("specifying both host name and CIDR mask is invalid: \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("specifying both host name and CIDR mask is invalid: \"%s\"", token->string); return NULL; @@ -1196,7 +1326,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid CIDR mask in address \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid CIDR mask in address \"%s\"", token->string); return NULL; @@ -1216,7 +1346,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("end-of-line before netmask specification"), errhint("Specify an address range in CIDR notation, or provide a separate netmask."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before netmask specification"; return NULL; } @@ -1227,7 +1357,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("multiple values specified for netmask"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for netmask"; return NULL; } @@ -1242,7 +1372,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid IP mask \"%s\": %s", token->string, gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid IP mask \"%s\": %s", token->string, gai_strerror(ret)); if (gai_result) @@ -1261,7 +1391,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("IP address and mask do not match"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "IP address and mask do not match"; return NULL; } @@ -1277,7 +1407,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before authentication method"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before authentication method"; return NULL; } @@ -1289,7 +1419,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for authentication type"), errhint("Specify exactly one authentication type per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for authentication type"; return NULL; } @@ -1326,7 +1456,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("MD5 authentication is not supported when \"db_user_namespace\" is enabled"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "MD5 authentication is not supported when \"db_user_namespace\" is enabled"; return NULL; } @@ -1367,7 +1497,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid authentication method \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid authentication method \"%s\"", token->string); return NULL; @@ -1380,7 +1510,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid authentication method \"%s\": not supported by this build", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid authentication method \"%s\": not supported by this build", token->string); return NULL; @@ -1402,7 +1532,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("gssapi authentication is not supported on local sockets"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "gssapi authentication is not supported on local sockets"; return NULL; } @@ -1414,7 +1544,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("peer authentication is only supported on local sockets"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "peer authentication is only supported on local sockets"; return NULL; } @@ -1432,7 +1562,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cert authentication is only supported on hostssl connections"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cert authentication is only supported on hostssl connections"; return NULL; } @@ -1482,7 +1612,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("authentication option not in name=value format: %s", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("authentication option not in name=value format: %s", token->string); return NULL; @@ -1526,7 +1656,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"; return NULL; } @@ -1537,7 +1667,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"; return NULL; } @@ -1553,7 +1683,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cannot use ldapsearchattribute together with ldapsearchfilter"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cannot use ldapsearchattribute together with ldapsearchfilter"; return NULL; } @@ -1570,7 +1700,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("list of RADIUS servers cannot be empty"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "list of RADIUS servers cannot be empty"; return NULL; } @@ -1581,7 +1711,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("list of RADIUS secrets cannot be empty"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "list of RADIUS secrets cannot be empty"; return NULL; } @@ -1600,7 +1730,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiussecrets), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS secrets (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiussecrets), list_length(parsedline->radiusservers)); @@ -1616,7 +1746,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiusports), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS ports (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiusports), list_length(parsedline->radiusservers)); @@ -1632,7 +1762,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiusidentifiers), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS identifiers (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiusidentifiers), list_length(parsedline->radiusservers)); @@ -1667,6 +1797,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg) { int line_num = hbaline->linenumber; + char *file_name = hbaline->sourcefile; #ifdef USE_LDAP hbaline->ldapscope = LDAP_SCOPE_SUBTREE; @@ -1690,7 +1821,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientcert can only be configured for \"hostssl\" rows"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientcert can only be configured for \"hostssl\" rows"; return false; } @@ -1707,7 +1838,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientcert only accepts \"verify-full\" when using \"cert\" authentication"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientcert can only be set to \"verify-full\" when using \"cert\" authentication"; return false; } @@ -1720,7 +1851,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid value for clientcert: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } } @@ -1732,7 +1863,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientname can only be configured for \"hostssl\" rows"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientname can only be configured for \"hostssl\" rows"; return false; } @@ -1751,7 +1882,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid value for clientname: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } } @@ -1837,7 +1968,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid ldapscheme value: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); hbaline->ldapscheme = pstrdup(val); } else if (strcmp(name, "ldapserver") == 0) @@ -1855,7 +1986,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid LDAP port number: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid LDAP port number: \"%s\"", val); return false; } @@ -1949,7 +2080,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS server list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -1968,7 +2099,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not translate RADIUS server name \"%s\" to address: %s", (char *) lfirst(l), gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); if (gai_result) pg_freeaddrinfo_all(hints.ai_family, gai_result); @@ -1997,7 +2128,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS port list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid RADIUS port number: \"%s\"", val); return false; } @@ -2010,7 +2141,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid RADIUS port number: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2033,7 +2164,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS secret list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2055,7 +2186,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS identifiers list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2069,7 +2200,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("unrecognized authentication option name: \"%s\"", name), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("unrecognized authentication option name: \"%s\"", name); return false; @@ -2217,7 +2348,7 @@ load_hba(void) return false; } - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ @@ -2288,6 +2419,137 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + if (strict) + { + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + } + else + { + Assert(kind == IncludedAuthFile); + ereport(LOG, + (errmsg("skipping missing authentication file \"%s\"", + *inc_fullname))); + } + + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} + +/* + * Try to open an included file, and tokenize it using the given context. + * Returns NULL if no error happens during tokenization, otherwise the error. + */ +static char * +process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, int elevel, + MemoryContext linecxt, List **tok_lines) +{ + char *inc_fullname; + FILE *inc_file; + char *err_msg = NULL; + + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) + { + char *err_msg; + + err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", err_msg))); + return err_msg; + } + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict, + outer_filename, elevel, &err_msg, &inc_fullname); + + if (inc_file == NULL) + { + if (strict) + { + /* open_inc_file should have reported an error. */ + Assert(err_msg != NULL); + return err_msg; + } + else + return NULL; + } + else + { + /* No error message should have been reported. */ + Assert(err_msg == NULL); + } + + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, depth, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + return NULL; +} /* * Parse one tokenised line from the ident config file and store the result in @@ -2306,6 +2568,7 @@ load_hba(void) IdentLine * parse_ident_line(TokenizedAuthLine *tok_line, int elevel) { + char *file_name = tok_line->file_name; int line_num = tok_line->line_num; char **err_msg = &tok_line->err_msg; ListCell *field; @@ -2366,7 +2629,7 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid regular expression \"%s\": %s", parsedline->ident_user + 1, errstr), errcontext("line %d of configuration file \"%s\"", - line_num, IdentFileName))); + line_num, file_name))); *err_msg = psprintf("invalid regular expression \"%s\": %s", parsedline->ident_user + 1, errstr); @@ -2601,7 +2864,7 @@ load_ident(void) return false; } - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..7433050112 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,16 +9,27 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # -# local DATABASE USER METHOD [OPTIONS] -# host DATABASE USER ADDRESS METHOD [OPTIONS] -# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# local DATABASE USER METHOD [OPTIONS] +# host DATABASE USER ADDRESS METHOD [OPTIONS] +# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..8e3fa29135 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,23 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # -# MAPNAME SYSTEM-USERNAME PG-USERNAME +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index c9be4bff1f..15326a01e2 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,12 +26,12 @@ static ArrayType *get_hba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int rule_number, int lineno, HbaLine *hba, - const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int mapping_number, int lineno, IdentLine *ident, - const char *err_msg); + int mapping_number, const char *filename, + int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); @@ -159,7 +159,7 @@ get_hba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 10 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line @@ -168,7 +168,8 @@ get_hba_options(HbaLine *hba) * tuple_store: where to store data * tupdesc: tuple descriptor for the view * rule_number: unique rule identifier among all valid rules - * lineno: pg_hba.conf line number (must always be valid) + * filename: name of the file containing that line + * lineno: line number in that file (must always be valid) * hba: parsed line data (can be NULL, in which case err_msg should be set) * err_msg: error message (NULL if none) * @@ -177,7 +178,7 @@ get_hba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int rule_number, int lineno, HbaLine *hba, + int rule_number, const char *filename, int lineno, HbaLine *hba, const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -202,6 +203,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -345,7 +348,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -386,7 +389,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open configuration file \"%s\": %m", HbaFileName))); - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -407,8 +410,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) rule_number++; - fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->line_num, - hbaline, tok_line->err_msg); + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -445,7 +448,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) } /* Number of columns in pg_hba_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6 +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -454,7 +457,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) * tuple_store: where to store data * tupdesc: tuple descriptor for the view * mapping_number: unique rule identifier among all valid rules - * lineno: pg_ident.conf line number (must always be valid) + * filename: name of the file containing that line + * lineno: line number in that file (must always be valid) * ident: parsed line data (can be NULL, in which case err_msg should be set) * err_msg: error message (NULL if none) * @@ -463,8 +467,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int mapping_number, int lineno, IdentLine *ident, - const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -482,6 +486,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -494,7 +500,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -534,7 +540,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open usermap file \"%s\": %m", IdentFileName))); - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -556,7 +562,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) mapping_number++; fill_ident_line(tuple_store, tupdesc, mapping_number, - tok_line->line_num, identline, tok_line->err_msg); + tok_line->file_name, tok_line->line_num, identline, + tok_line->err_msg); } /* Free tokenizer memory */ diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l index ce5633844c..b3e18e48cf 100644 --- a/src/backend/utils/misc/guc-file.l +++ b/src/backend/utils/misc/guc-file.l @@ -700,6 +700,122 @@ GUC_flex_fatal(const char *msg) return 0; /* keep compiler quiet */ } +/* + * Returns the list of config files located in a directory, in alphabetical + * order. + * + * We don't check for recursion or too-deep nesting depth here, its up to the + * caller to take care of that. + */ +char ** +GetDirConfFiles(const char *includedir, const char *calling_file, int elevel, + int *num_filenames, char **err_msg) +{ + char *directory; + DIR *d; + struct dirent *de; + char **filenames; + int size_filenames; + + /* + * Reject directory name that is all-blank (including empty), as that + * leads to confusion --- we'd read the containing directory, typically + * resulting in recursive inclusion of the same file(s). + */ + if (strspn(includedir, " \t\r\n") == strlen(includedir)) + { + ereport(elevel, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("empty configuration directory name: \"%s\"", + includedir))); + *err_msg = "empty configuration directory name"; + return NULL; + } + + directory = AbsoluteConfigLocation(includedir, calling_file); + d = AllocateDir(directory); + if (d == NULL) + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not open configuration directory \"%s\": %m", + directory))); + *err_msg = psprintf("could not open directory \"%s\"", directory); + filenames = NULL; + goto cleanup; + } + + /* + * Read the directory and put the filenames in an array, so we can sort + * them prior to caller processing the contents. + */ + size_filenames = 32; + filenames = (char **) palloc(size_filenames * sizeof(char *)); + *num_filenames = 0; + + while ((de = ReadDir(d, directory)) != NULL) + { + struct stat st; + char filename[MAXPGPATH]; + + /* + * Only parse files with names ending in ".conf". Explicitly reject + * files starting with ".". This excludes things like "." and "..", + * as well as typical hidden files, backup files, and editor debris. + */ + if (strlen(de->d_name) < 6) + continue; + if (de->d_name[0] == '.') + continue; + if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) + continue; + + join_path_components(filename, directory, de->d_name); + canonicalize_path(filename); + if (stat(filename, &st) == 0) + { + /* Ignore directories. */ + if (S_ISDIR(st.st_mode)) + continue; + + /* Add file to array, increasing its size in blocks of 32 */ + if (*num_filenames >= size_filenames) + { + size_filenames += 32; + filenames = (char **) repalloc(filenames, + size_filenames * sizeof(char *)); + } + filenames[*num_filenames] = pstrdup(filename); + (*num_filenames)++; + } + else + { + /* + * stat does not care about permissions, so the most likely reason + * a file can't be accessed now is if it was removed between the + * directory listing and now. + */ + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not stat file \"%s\": %m", + filename))); + *err_msg = psprintf("could not stat file \"%s\"", filename); + pfree(filenames); + filenames = NULL; + goto cleanup; + } + } + + if (*num_filenames > 0) + qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp); + +cleanup: + if (d) + FreeDir(d); + pfree(directory); + return filenames; +} + /* * Read and parse a single configuration file. This function recurses * to handle "include" directives. @@ -961,138 +1077,32 @@ ParseConfigDirectory(const char *includedir, ConfigVariable **head_p, ConfigVariable **tail_p) { - char *directory; - DIR *d; - struct dirent *de; + char *err_msg; char **filenames; int num_filenames; - int size_filenames; - bool status; - /* - * Reject directory name that is all-blank (including empty), as that - * leads to confusion --- we'd read the containing directory, typically - * resulting in recursive inclusion of the same file(s). - */ - if (strspn(includedir, " \t\r\n") == strlen(includedir)) - { - ereport(elevel, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("empty configuration directory name: \"%s\"", - includedir))); - record_config_file_error("empty configuration directory name", - calling_file, calling_lineno, - head_p, tail_p); - return false; - } + filenames = GetDirConfFiles(includedir, calling_file, elevel, + &num_filenames, &err_msg); - /* - * We don't check for recursion or too-deep nesting depth here; the - * subsequent calls to ParseConfigFile will take care of that. - */ - - directory = AbsoluteConfigLocation(includedir, calling_file); - d = AllocateDir(directory); - if (d == NULL) + if (!filenames) { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open configuration directory \"%s\": %m", - directory))); - record_config_file_error(psprintf("could not open directory \"%s\"", - directory), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } - - /* - * Read the directory and put the filenames in an array, so we can sort - * them prior to processing the contents. - */ - size_filenames = 32; - filenames = (char **) palloc(size_filenames * sizeof(char *)); - num_filenames = 0; - - while ((de = ReadDir(d, directory)) != NULL) - { - struct stat st; - char filename[MAXPGPATH]; - - /* - * Only parse files with names ending in ".conf". Explicitly reject - * files starting with ".". This excludes things like "." and "..", - * as well as typical hidden files, backup files, and editor debris. - */ - if (strlen(de->d_name) < 6) - continue; - if (de->d_name[0] == '.') - continue; - if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) - continue; - - join_path_components(filename, directory, de->d_name); - canonicalize_path(filename); - if (stat(filename, &st) == 0) - { - if (!S_ISDIR(st.st_mode)) - { - /* Add file to array, increasing its size in blocks of 32 */ - if (num_filenames >= size_filenames) - { - size_filenames += 32; - filenames = (char **) repalloc(filenames, - size_filenames * sizeof(char *)); - } - filenames[num_filenames] = pstrdup(filename); - num_filenames++; - } - } - else - { - /* - * stat does not care about permissions, so the most likely reason - * a file can't be accessed now is if it was removed between the - * directory listing and now. - */ - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not stat file \"%s\": %m", - filename))); - record_config_file_error(psprintf("could not stat file \"%s\"", - filename), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } + record_config_file_error(err_msg, calling_file, calling_lineno, head_p, + tail_p); + return false; } - if (num_filenames > 0) + for (int i = 0; i < num_filenames; i++) { - int i; - - qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp); - for (i = 0; i < num_filenames; i++) + if (!ParseConfigFile(filenames[i], true, + calling_file, calling_lineno, + depth, elevel, + head_p, tail_p)) { - if (!ParseConfigFile(filenames[i], true, - calling_file, calling_lineno, - depth, elevel, - head_p, tail_p)) - { - status = false; - goto cleanup; - } + return false; } } - status = true; -cleanup: - if (d) - FreeDir(d); - pfree(directory); - return status; + return true; } /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index b284212bd3..4470011977 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6128,16 +6128,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o}', - proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '6250', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,text,text,text}', - proargmodes => '{o,o,o,o,o,o}', - proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index 90036f7bcd..0ea100d1b8 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -155,6 +156,7 @@ typedef struct AuthToken typedef struct TokenizedAuthLine { List *fields; /* List of lists of AuthTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ @@ -174,6 +176,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel); extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel); extern bool pg_isblank(const char c); extern MemoryContext tokenize_auth_file(const char *filename, FILE *file, - List **tok_lines, int elevel); + List **tok_lines, int depth, + int elevel); #endif /* HBA_H */ diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h index e734493a48..1a3ab6306d 100644 --- a/src/include/utils/guc.h +++ b/src/include/utils/guc.h @@ -145,6 +145,8 @@ typedef struct ConfigVariable struct ConfigVariable *next; } ConfigVariable; +extern char **GetDirConfFiles(const char *includedir, const char *calling_file, + int elevel, int *num_filenames, char **err_msg); extern bool ParseConfigFile(const char *config_file, bool strict, const char *calling_file, int calling_lineno, int depth, int elevel, diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl new file mode 100644 index 0000000000..8eae72b8d4 --- /dev/null +++ b/src/test/authentication/t/003_file_inclusion.pl @@ -0,0 +1,657 @@ + +# Copyright (c) 2021-2022, PostgreSQL Global Development Group + +# Set of tests for authentication and pg_hba.conf inclusion. +# This test can only run with Unix-domain sockets. + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use Time::HiRes qw(usleep); +use IPC::Run qw(pump finish timer); +use Data::Dumper; + +if (!$use_unix_sockets) +{ + plan skip_all => + "authentication tests cannot run without Unix-domain sockets"; +} + +# stores the current line counter for each file. hba_rule and ident_rule are +# fake file names used for the global rule number for each auth view. +my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + +my $hba_file = 'subdir1/pg_hba_custom.conf'; +my $ident_file = 'subdir2/pg_ident_custom.conf'; + +# Initialize primary node +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init; +$node->start; + +my $data_dir = $node->data_dir; + +# Normalize the data directory for Windows +$data_dir =~ s/\/\.\//\//g; # reduce /./ to / +$data_dir =~ s/\/\//\//g; # reduce // to / +$data_dir =~ s/\/$//; # remove trailing / + + +# Add the given payload to the given relative HBA file of the given node. +# This function maintains the %cur_line metadata, so it has to be called in the +# expected inclusion evaluation order in order to keep it in sync. +# +# If the payload starts with "include" or "ignore", the function doesn't +# increase the general hba rule number. +# +# If an err_str is provided, it returns an arrayref containing the provided +# filename, the current line number in that file and the provided err_str. The +# err_str has to be a valid regex string. +# Otherwise it only returns the line number of the payload in the wanted file. +# This function has to be called in the expected inclusion evaluation order to +# keep the %cur_line information in sync. +sub add_hba_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'hba_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_hba_file_rules line + @tokens = split(/ /, $payload); + $tokens[1] = '{' . $tokens[1] . '}'; # database + $tokens[2] = '{' . $tokens[2] . '}'; # user_name + + # add empty address and netmask betweed user_name and auth_method + splice @tokens, 3, 0, ''; + splice @tokens, 3, 0, ''; + + # append empty options and error + push @tokens, ''; + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Add the given payload to the given relative ident file of the given node. +# Same as add_hba_line but for pg_ident files +sub add_ident_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'ident_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_ident_file_mappings line + @tokens = split(/ /, $payload); + + # append empty error + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Delete pg_hba.conf from the given node, add various entries to test the +# include infrastructure and then execute a reload to refresh it. +sub generate_valid_auth_files +{ + my $node = shift; + my $hba_expected = ''; + my $ident_expected = ''; + + # customise main auth file names + $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'"); + $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'"); + + # and make original ones invalid to be sure they're not used anywhere + $node->append_conf('pg_hba.conf', "some invalid line"); + $node->append_conf('pg_ident.conf', "some invalid line"); + + # pg_hba stuff + mkdir("$data_dir/subdir1"); + mkdir("$data_dir/hba_inc"); + mkdir("$data_dir/hba_inc_if"); + mkdir("$data_dir/hba_pos"); + + # Make sure we will still be able to connect + $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust'); + + # Add include data + add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf"); + $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject"); + + $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject"); + + add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject"); + # include is relative to current path + add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject"); + + # include_if_exists data + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none"); + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some"); + $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject"); + + # include_dir data + add_hba_line($node, "$hba_file", "include_dir ../hba_inc"); + add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included"); + $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject"); + $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject"); + + # secondary auth file + add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject'); + $node->append_conf('dbnames.conf', "db1"); + $node->append_conf('dbnames.conf', "db3"); + $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1) + . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1) + . '|local|{db1,db3}|{all}|||reject||'; + + # pg_ident stuff + mkdir("$data_dir/subdir2"); + mkdir("$data_dir/ident_inc"); + mkdir("$data_dir/ident_inc_if"); + mkdir("$data_dir/ident_pos"); + + # Add include data + add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf"); + $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar"); + + $ident_expected .= add_ident_line($node, "$ident_file", "test a b"); + + add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar"); + # include is relative to current path + add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar"); + + # include_if_exists data + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none"); + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some"); + $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar"); + + # include_dir data + add_ident_line($node, "$ident_file", "include_dir ../ident_inc"); + add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included"); + $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar"); + $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar"); + + $node->restart; + $node->connect_ok('dbname=postgres', + 'Connection ok after generating valid auth files'); + + return ($hba_expected, $ident_expected); +} + +# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal +# entries to allow authentication. +sub reset_auth_files +{ + my $node = shift; + + unlink("$data_dir/$hba_file"); + unlink("$data_dir/$ident_file"); + + %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + + return add_hba_line($node, "$hba_file", 'local all all trust'); +} + +# Generate a list of expected error regex for the given array of error +# conditions, as generated by add_hba_line/add_ident_line with an err_str. +# +# 2 regex are generated per array entry: one for the given err_str, and one for +# the expected line in the specific file. Since all lines are independant, +# there's no guarantee that a specific failure regex and the per-line regex +# will match the same error. Calling code should add at least one test with a +# single error to make sure that the line number / file name is correct. +# +# On top of that, an extra line is generated for the general failure to process +# the main auth file. +sub generate_log_err_patterns +{ + my $node = shift; + my $raw_errors = shift; + my $is_hba_err = shift; + my @errors; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + push @errors, qr/$err_str/; + + # Context messages with the file / line location aren't always emitted + if ($err_str !~ /maximum nesting depth exceeded/ and + $err_str !~ /could not open secondary authentication file/) + { + push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/ + } + } + + push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err); + + return \@errors; +} + +# Generate the expected output for the auth file view error reporting (file +# name, file line, error), for the given array of error conditions, as +# generated generated by add_hba_line/add_ident_line with an err_str. +sub generate_log_err_rows +{ + my $node = shift; + my $raw_errors = shift; + my $exp_rows = ''; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + $exp_rows .= "\n" if ($exp_rows ne ""); + + # Unescape regex patterns if any + $err_str =~ s/\\([\(\)])/$1/g; + $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str" + } + + return $exp_rows; +} + +# Reset the main auth files, append the given payload to the given config file, +# and check that the instance cannot start, raising the expected error line(s). +sub start_errors_like +{ + my $node = shift; + my $file = shift; + my $payload = shift; + my $pattern = shift; + my $should_fail = shift; + + reset_auth_files($node); + $node->append_conf($file, $payload); + + unlink($node->logfile); + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir, + '-l', $node->logfile, 'start'); + + if ($should_fail) + { + ok($ret != 0, "Cannot start postgres with faulty $file"); + } + else + { + ok($ret == 0, "postgres can start with faulty $file"); + } + + my $log_contents = slurp_file($node->logfile); + + foreach (@{$pattern}) + { + like($log_contents, + $_, + "Expected failure found in the logs"); + } + + if (not $should_fail) + { + # We can't simply call $node->stop here as the call is optimized out + # when the server isn't started with $node->start. + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', + $data_dir, 'stop', '-m', 'fast'); + ok($ret == 0, "Could stop postgres"); + } +} + +# We should be able to connect, and see an empty pg_ident.conf +is($node->psql( + 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'), + qq(0), + 'pg_ident.conf is empty'); + +############################################ +# part 1, test view reporting for valid data +############################################ +my ($exp_hba, $exp_ident) = generate_valid_auth_files($node); + +$node->connect_ok('dbname=postgres', 'Connection still ok'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_hba_file_rules'), + qq($exp_hba), + 'pg_hba_file_rules content is expected'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_ident_file_mappings'), + qq($exp_ident), + 'pg_ident_file_mappings content is expected'); + +############################################# +# part 2, test log reporting for invalid data +############################################# +reset_auth_files($node); +$node->restart('fast'); +$node->connect_ok('dbname=postgres', + 'Connection ok after resetting auth files'); + +$node->stop('fast'); + +start_errors_like($node, $hba_file, "include ../not_a_file", + [ + qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file +mkdir("$data_dir/hba_inc_fail"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token"); +start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail", + [ + qr/invalid connection type "not_a_token"/, + qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file with nested inclusion +unlink("$data_dir/hba_inc_fail/inc_dir.conf"); +my @hba_raw_errors_step1; + +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1"); + +add_hba_line($node, "hba_inc_fail/file1", "include file2"); +add_hba_line($node, "hba_inc_fail/file2", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file2", "include file3"); + +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3", + "local all all zuul", + 'invalid authentication method "zuul"'); + +start_errors_like( + $node, $hba_file, "include_dir ../hba_inc_fail", + generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @hba_raw_errors_step2; + +# include_if_exists, with various problems +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local", + "end-of-line before database specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local,host", + "multiple values specified for connection type"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all", + "end-of-line before role specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all all", + "end-of-line before authentication method"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "host all all test/42", + 'specifying both host name and CIDR mask is invalid: "test/42"'); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + 'local @dbnames_fails.conf all reject', + "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory"); + +add_hba_line($node, "hba_if_exists.conf", "include recurse.conf"); +push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf", + "include recurse.conf", + 'could not open configuration file "recurse.conf": maximum nesting depth exceeded'); + +# Generate the regex for the expected errors in the logs. There's no guarantee +# that the generated "line X of file..." will be emitted for the expected line, +# but previous tests already ensured that the correct line number / file name +# was emitted, so ensuring that there's an error in all expected lines is +# enough here. +my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2, + 1); + +# Not an error, but it should raise a message in the logs. Manually add an +# extra log message to detect +add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none"); +push @{$expected_errors}, + qr/skipping missing authentication file "$data_dir\/if_exists_none"/; + +start_errors_like( + $node, $hba_file, "include_if_exists ../hba_if_exists.conf", + $expected_errors, 1); + +# Mostly the same, but for ident files +reset_auth_files($node); + +my @ident_raw_errors_step1; + +# include_dir, single included file with nested inclusion +mkdir("$data_dir/ident_inc_fail"); +add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1"); + +add_ident_line($node, "ident_inc_fail/file1", "include file2"); +add_ident_line($node, "ident_inc_fail/file2", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file2", "include file3"); + +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3", + "failmap /(fail postgres", + 'invalid regular expression "\(fail": parentheses \(\) not balanced'); + +start_errors_like( + $node, $ident_file, "include_dir ../ident_inc_fail", + generate_log_err_patterns($node, \@ident_raw_errors_step1, 0), + 0); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @ident_raw_errors_step2; + +# include_if_exists, with various problems +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map", + "missing entry at end of line"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2", + "multiple values in ident field"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", + 'map @osnames_fails.conf postgres', + "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory"); + +add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf", + 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded'); + +start_errors_like( + $node, $ident_file, "include_if_exists ../ident_if_exists.conf", + # There's no guarantee that the generated "line X of file..." will be + # emitted for the expected line, but previous tests already ensured that + # the correct line number / file name was emitted, so ensuring that there's + # an error in all expected lines is enough here. + generate_log_err_patterns($node, \@ident_raw_errors_step2, 0), + 0); + +##################################################### +# part 3, test reporting of various error scenario +# NOTE: this will be bypassed -DEXEC_BACKEND or win32 +##################################################### +reset_auth_files($node); + +$node->start; +$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset'); + +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_hba_file_rules'); + +add_ident_line($node, $ident_file, ''); +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_ident_file_mappings'); + +# The instance could be restarted and no error is detected. Now check if the +# build is compatible with the view error reporting (EXEC_BACKEND / win32 will +# fail when trying to connect as they always rely on the current auth files +# content) +my @hba_raw_errors; + +push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +my ($stdout, $stderr); +my $cmdret = $node->psql('postgres', 'SELECT 1', + stdout => \$stdout, stderr => \$stderr); + +if ($cmdret != 0) +{ + # Connection failed. Bail out, but make sure to raise a failure if it + # didn't fail for the expected hba file modification. + like($stderr, + qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/, + "Connection failed due to loading an invalid hba file"); + + done_testing(); + diag("Build not compatible with auth file view error reporting, bail out.\n"); + exit; +} + +# Combine errors generated at step 2, in the same order. +$node->append_conf($hba_file, "include_dir ../hba_inc_fail"); +push @hba_raw_errors, @hba_raw_errors_step1; + +$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf"); +push @hba_raw_errors, @hba_raw_errors_step2; + +my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules' + . ' WHERE error IS NOT NULL ORDER BY rule_number'), + qq($hba_expected), + 'Detected all error in hba file'); + +# and do the same for pg_ident +my @ident_raw_errors; + +push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +$node->append_conf($ident_file, "include_dir ../ident_inc_fail"); +push @ident_raw_errors, @ident_raw_errors_step1; + +$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf"); +push @ident_raw_errors, @ident_raw_errors_step2; + +my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings' + . ' WHERE error IS NOT NULL ORDER BY mapping_number'), + qq($ident_expected), + 'Detected all error in ident file'); + +done_testing(); diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 79408710e0..5ed2fe3704 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname, FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); pg_hba_file_rules| SELECT a.rule_number, + a.file_name, a.line_number, a.type, a.database, @@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error); + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, a.line_number, a.map_name, a.sys_name, a.pg_username, a.error - FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.37.0 --u2elhxoozhwco244 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v9-0003-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v10 2/3] Allow file inclusion in pg_hba and pg_ident files. @ 2022-05-30 03:15 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-05-30 03:15 UTC (permalink / raw) pg_hba.conf file now has support for "include", "include_dir" and "include_if_exists" directives, which work similarly to the same directives in the postgresql.conf file. This fixes a possible crash if a secondary file tries to include itself as there's now a nesting depth check in the inclusion code path, same as the postgresql.conf. Many regression tests added to cover both the new directives, but also error detection for the whole pg_hba / pg_ident files. Catversion is bumped. Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/client-auth.sgml | 86 ++- doc/src/sgml/system-views.sgml | 22 +- src/backend/libpq/hba.c | 483 ++++++++++--- src/backend/libpq/pg_hba.conf.sample | 25 +- src/backend/libpq/pg_ident.conf.sample | 15 +- src/backend/utils/adt/hbafuncs.c | 43 +- src/backend/utils/misc/guc-file.l | 229 +++--- src/include/catalog/pg_proc.dat | 12 +- src/include/libpq/hba.h | 5 +- src/include/utils/guc.h | 2 + .../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++ src/test/regress/expected/rules.out | 6 +- 12 files changed, 1312 insertions(+), 273 deletions(-) create mode 100644 src/test/authentication/t/003_file_inclusion.pl diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index c6f1b70fd3..42ceb6f3e6 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,23 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication + record. Inclusion directives specify files that can be included, which + contains additional records. The records will be inserted in lieu of the + inclusion records. Those records only contains two fields: the + <literal>include</literal>, <literal>include_if_exists</literal> or + <literal>include_dir</literal> directive and the file or directory to be + included. The file or directory can be a relative of absolute path, and can + be double quoted if needed. For the <literal>include_dir</literal> form, + all files not starting with a <literal>.</literal> and ending with + <literal>.conf</literal> will be included. Multiple files within an include + directory are processed in file name order (according to C locale rules, + i.e., numbers before letters, and uppercase letters before lowercase ones). + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,21 +118,57 @@ <para> A record can have several formats: <synopsis> -local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +include <replaceable>file</replaceable> +include_if_exists <replaceable>file</replaceable> +include_dir <replaceable>directory</replaceable> +local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> </synopsis> The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_if_exists</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file if the + file exists and can be read. Otherwise, a message will be logged to + indicate that the file is skipped. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_dir</literal></term> + <listitem> + <para> + This line will be replaced with the content of all the files found in + the directory, if they don't start with a <literal>.</literal> and end + with <literal>.conf</literal>, processed in file name order (according + to C locale rules, i.e., numbers before letters, and uppercase letters + before lowercase ones). + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -835,8 +886,10 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> +<replaceable>include_dir</replaceable> <replaceable>directory</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -847,6 +900,11 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + As for <filename>pg_hba.conf</filename>, the lines in this file can either + be inclusion directives or user name map records, and follow the same + rules. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 1d619427c1..4e63dec74c 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -1003,12 +1003,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule the given <literal>file_name</literal> </para></entry> </row> @@ -1153,12 +1162,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this mapping + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_ident.conf</filename> + Line number of this mapping in the given <literal>file_name</literal> </para></entry> </row> diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index 4637426d62..071bf1ff95 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -22,6 +22,7 @@ #include <sys/param.h> #include <sys/socket.h> #include <netdb.h> +#include <sys/stat.h> #include <netinet/in.h> #include <arpa/inet.h> #include <unistd.h> @@ -69,6 +70,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -113,10 +120,22 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int depth, + int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, - const char *inc_filename, int elevel, char **err_msg); + const char *inc_filename, int depth, int elevel, + char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + bool strict, const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); +static char *process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, + int elevel, MemoryContext linecxt, + List **tok_lines); /* @@ -303,7 +322,7 @@ copy_auth_token(AuthToken *in) */ static List * next_field_expand(const char *filename, char **lineptr, - int elevel, char **err_msg) + int depth, int elevel, char **err_msg) { char buf[MAX_TOKEN]; bool trailing_comma; @@ -319,7 +338,7 @@ next_field_expand(const char *filename, char **lineptr, /* Is this referencing a file? */ if (!initial_quote && buf[0] == '@' && buf[1] != '\0') - tokens = tokenize_inc_file(tokens, filename, buf + 1, + tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1, elevel, err_msg); else tokens = lappend(tokens, make_auth_token(buf, initial_quote)); @@ -347,6 +366,7 @@ static List * tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, + int depth, int elevel, char **err_msg) { @@ -356,39 +376,30 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); + *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", *err_msg))); + return tokens; } - inc_file = AllocateFile(inc_fullname, "r"); - if (inc_file == NULL) - { - int save_errno = errno; + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true, + outer_filename, elevel, err_msg, &inc_fullname); - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); + if (inc_file == NULL) return tokens; - } /* There is possible recursion here if the file contains @ */ - linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); + linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1, + elevel); FreeFile(inc_file); pfree(inc_fullname); @@ -426,11 +437,38 @@ tokenize_inc_file(List *tokens, /* * tokenize_auth_file - * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a dedicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, + int depth, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, depth, + elevel); + + return linecxt; +} + +/* + * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see the struct definition * in libpq/hba.h. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -439,30 +477,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int depth, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -515,7 +545,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, { List *current_field; - current_field = next_field_expand(filename, &lineptr, + current_field = next_field_expand(filename, &lineptr, depth, elevel, &err_msg); /* add field to line, unless we are at EOL or comment start */ if (current_field != NIL) @@ -523,29 +553,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + AuthToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); + + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; + + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, true, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } + else if (strcmp(first->string, "include_dir") == 0) + { + char **filenames; + char *dir_name = second->string; + int num_filenames; + StringInfoData err_buf; + + filenames = GetDirConfFiles(dir_name, filename, elevel, + &num_filenames, &err_msg); + + if (!filenames) + { + /* We have the error in err_msg, simply process it */ + goto process_line; + } + + initStringInfo(&err_buf); + for (int i = 0; i < num_filenames; i++) + { + /* + * err_msg is used here as a temp buffer, it will be + * overwritten at the end of the loop with the + * cumulated errors, if any. + */ + err_msg = process_included_authfile(filenames[i], true, + filename, depth + 1, elevel, + linecxt, tok_lines); + + /* Cumulate errors if any. */ + if (err_msg) + { + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfoString(&err_buf, err_msg); + } + } + + /* + * If there were no errors, the line is fully processed, bypass + * the general TokenizedAuthLine processing. + */ + if (err_buf.len == 0) + goto next_line; + + /* Otherwise, process the cumulated errors, if any. */ + err_msg = err_buf.data; + } + else if (strcmp(first->string, "include_if_exists") == 0) + { + char *inc_filename; - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, false, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } } +process_line: + /* + * General processing: report the error if any and emit line to the + * TokenizedAuthLine + */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -855,7 +983,7 @@ do { \ errmsg("authentication option \"%s\" is only valid for authentication methods %s", \ optname, _(validmethods)), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \ optname, validmethods); \ return false; \ @@ -875,7 +1003,7 @@ do { \ errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname); \ return NULL; \ @@ -898,7 +1026,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("missing entry at end of line"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, file_name))); \ *err_msg = pstrdup("missing entry at end of line"); \ return NULL; \ } \ @@ -911,7 +1039,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("multiple values in ident field"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, file_name))); \ *err_msg = pstrdup("multiple values in ident field"); \ return NULL; \ } \ @@ -934,6 +1062,7 @@ HbaLine * parse_hba_line(TokenizedAuthLine *tok_line, int elevel) { int line_num = tok_line->line_num; + char *file_name = tok_line->file_name; char **err_msg = &tok_line->err_msg; char *str; struct addrinfo *gai_result; @@ -948,6 +1077,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -962,7 +1092,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for connection type"), errhint("Specify exactly one connection type per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for connection type"; return NULL; } @@ -990,7 +1120,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("hostssl record cannot match because SSL is disabled"), errhint("Set ssl = on in postgresql.conf."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostssl record cannot match because SSL is disabled"; } #else @@ -998,7 +1128,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("hostssl record cannot match because SSL is not supported by this build"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostssl record cannot match because SSL is not supported by this build"; #endif } @@ -1010,7 +1140,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("hostgssenc record cannot match because GSSAPI is not supported by this build"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostgssenc record cannot match because GSSAPI is not supported by this build"; #endif } @@ -1031,7 +1161,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid connection type \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid connection type \"%s\"", token->string); return NULL; } @@ -1044,7 +1174,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before database specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before database specification"; return NULL; } @@ -1064,7 +1194,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before role specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before role specification"; return NULL; } @@ -1086,7 +1216,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before IP address specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before IP address specification"; return NULL; } @@ -1098,7 +1228,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for host address"), errhint("Specify one address range per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for host address"; return NULL; } @@ -1157,7 +1287,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid IP address \"%s\": %s", str, gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid IP address \"%s\": %s", str, gai_strerror(ret)); if (gai_result) @@ -1177,7 +1307,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("specifying both host name and CIDR mask is invalid: \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("specifying both host name and CIDR mask is invalid: \"%s\"", token->string); return NULL; @@ -1191,7 +1321,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid CIDR mask in address \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid CIDR mask in address \"%s\"", token->string); return NULL; @@ -1211,7 +1341,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("end-of-line before netmask specification"), errhint("Specify an address range in CIDR notation, or provide a separate netmask."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before netmask specification"; return NULL; } @@ -1222,7 +1352,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("multiple values specified for netmask"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for netmask"; return NULL; } @@ -1237,7 +1367,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid IP mask \"%s\": %s", token->string, gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid IP mask \"%s\": %s", token->string, gai_strerror(ret)); if (gai_result) @@ -1256,7 +1386,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("IP address and mask do not match"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "IP address and mask do not match"; return NULL; } @@ -1272,7 +1402,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before authentication method"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before authentication method"; return NULL; } @@ -1284,7 +1414,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for authentication type"), errhint("Specify exactly one authentication type per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for authentication type"; return NULL; } @@ -1321,7 +1451,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("MD5 authentication is not supported when \"db_user_namespace\" is enabled"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "MD5 authentication is not supported when \"db_user_namespace\" is enabled"; return NULL; } @@ -1362,7 +1492,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid authentication method \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid authentication method \"%s\"", token->string); return NULL; @@ -1375,7 +1505,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid authentication method \"%s\": not supported by this build", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid authentication method \"%s\": not supported by this build", token->string); return NULL; @@ -1397,7 +1527,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("gssapi authentication is not supported on local sockets"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "gssapi authentication is not supported on local sockets"; return NULL; } @@ -1409,7 +1539,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("peer authentication is only supported on local sockets"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "peer authentication is only supported on local sockets"; return NULL; } @@ -1427,7 +1557,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cert authentication is only supported on hostssl connections"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cert authentication is only supported on hostssl connections"; return NULL; } @@ -1477,7 +1607,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("authentication option not in name=value format: %s", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("authentication option not in name=value format: %s", token->string); return NULL; @@ -1521,7 +1651,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"; return NULL; } @@ -1532,7 +1662,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"; return NULL; } @@ -1548,7 +1678,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cannot use ldapsearchattribute together with ldapsearchfilter"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cannot use ldapsearchattribute together with ldapsearchfilter"; return NULL; } @@ -1565,7 +1695,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("list of RADIUS servers cannot be empty"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "list of RADIUS servers cannot be empty"; return NULL; } @@ -1576,7 +1706,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("list of RADIUS secrets cannot be empty"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "list of RADIUS secrets cannot be empty"; return NULL; } @@ -1595,7 +1725,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiussecrets), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS secrets (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiussecrets), list_length(parsedline->radiusservers)); @@ -1611,7 +1741,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiusports), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS ports (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiusports), list_length(parsedline->radiusservers)); @@ -1627,7 +1757,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiusidentifiers), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS identifiers (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiusidentifiers), list_length(parsedline->radiusservers)); @@ -1662,6 +1792,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg) { int line_num = hbaline->linenumber; + char *file_name = hbaline->sourcefile; #ifdef USE_LDAP hbaline->ldapscope = LDAP_SCOPE_SUBTREE; @@ -1685,7 +1816,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientcert can only be configured for \"hostssl\" rows"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientcert can only be configured for \"hostssl\" rows"; return false; } @@ -1702,7 +1833,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientcert only accepts \"verify-full\" when using \"cert\" authentication"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientcert can only be set to \"verify-full\" when using \"cert\" authentication"; return false; } @@ -1715,7 +1846,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid value for clientcert: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } } @@ -1727,7 +1858,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientname can only be configured for \"hostssl\" rows"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientname can only be configured for \"hostssl\" rows"; return false; } @@ -1746,7 +1877,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid value for clientname: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } } @@ -1832,7 +1963,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid ldapscheme value: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); hbaline->ldapscheme = pstrdup(val); } else if (strcmp(name, "ldapserver") == 0) @@ -1850,7 +1981,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid LDAP port number: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid LDAP port number: \"%s\"", val); return false; } @@ -1944,7 +2075,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS server list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -1963,7 +2094,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not translate RADIUS server name \"%s\" to address: %s", (char *) lfirst(l), gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); if (gai_result) pg_freeaddrinfo_all(hints.ai_family, gai_result); @@ -1992,7 +2123,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS port list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid RADIUS port number: \"%s\"", val); return false; } @@ -2005,7 +2136,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid RADIUS port number: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2028,7 +2159,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS secret list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2050,7 +2181,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS identifiers list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2064,7 +2195,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("unrecognized authentication option name: \"%s\"", name), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("unrecognized authentication option name: \"%s\"", name); return false; @@ -2212,7 +2343,7 @@ load_hba(void) return false; } - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ @@ -2283,6 +2414,137 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + if (strict) + { + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + } + else + { + Assert(kind == IncludedAuthFile); + ereport(LOG, + (errmsg("skipping missing authentication file \"%s\"", + *inc_fullname))); + } + + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} + +/* + * Try to open an included file, and tokenize it using the given context. + * Returns NULL if no error happens during tokenization, otherwise the error. + */ +static char * +process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, int elevel, + MemoryContext linecxt, List **tok_lines) +{ + char *inc_fullname; + FILE *inc_file; + char *err_msg = NULL; + + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) + { + char *err_msg; + + err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", err_msg))); + return err_msg; + } + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict, + outer_filename, elevel, &err_msg, &inc_fullname); + + if (inc_file == NULL) + { + if (strict) + { + /* open_inc_file should have reported an error. */ + Assert(err_msg != NULL); + return err_msg; + } + else + return NULL; + } + else + { + /* No error message should have been reported. */ + Assert(err_msg == NULL); + } + + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, depth, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + return NULL; +} /* * Parse one tokenised line from the ident config file and store the result in @@ -2301,6 +2563,7 @@ load_hba(void) IdentLine * parse_ident_line(TokenizedAuthLine *tok_line, int elevel) { + char *file_name = tok_line->file_name; int line_num = tok_line->line_num; char **err_msg = &tok_line->err_msg; ListCell *field; @@ -2361,7 +2624,7 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid regular expression \"%s\": %s", parsedline->ident_user + 1, errstr), errcontext("line %d of configuration file \"%s\"", - line_num, IdentFileName))); + line_num, file_name))); *err_msg = psprintf("invalid regular expression \"%s\": %s", parsedline->ident_user + 1, errstr); @@ -2596,7 +2859,7 @@ load_ident(void) return false; } - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..7433050112 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,16 +9,27 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # -# local DATABASE USER METHOD [OPTIONS] -# host DATABASE USER ADDRESS METHOD [OPTIONS] -# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# local DATABASE USER METHOD [OPTIONS] +# host DATABASE USER ADDRESS METHOD [OPTIONS] +# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..8e3fa29135 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,23 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # -# MAPNAME SYSTEM-USERNAME PG-USERNAME +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index c9be4bff1f..15326a01e2 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,12 +26,12 @@ static ArrayType *get_hba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int rule_number, int lineno, HbaLine *hba, - const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int mapping_number, int lineno, IdentLine *ident, - const char *err_msg); + int mapping_number, const char *filename, + int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); @@ -159,7 +159,7 @@ get_hba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 10 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line @@ -168,7 +168,8 @@ get_hba_options(HbaLine *hba) * tuple_store: where to store data * tupdesc: tuple descriptor for the view * rule_number: unique rule identifier among all valid rules - * lineno: pg_hba.conf line number (must always be valid) + * filename: name of the file containing that line + * lineno: line number in that file (must always be valid) * hba: parsed line data (can be NULL, in which case err_msg should be set) * err_msg: error message (NULL if none) * @@ -177,7 +178,7 @@ get_hba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int rule_number, int lineno, HbaLine *hba, + int rule_number, const char *filename, int lineno, HbaLine *hba, const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -202,6 +203,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -345,7 +348,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -386,7 +389,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open configuration file \"%s\": %m", HbaFileName))); - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -407,8 +410,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) rule_number++; - fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->line_num, - hbaline, tok_line->err_msg); + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -445,7 +448,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) } /* Number of columns in pg_hba_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6 +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -454,7 +457,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) * tuple_store: where to store data * tupdesc: tuple descriptor for the view * mapping_number: unique rule identifier among all valid rules - * lineno: pg_ident.conf line number (must always be valid) + * filename: name of the file containing that line + * lineno: line number in that file (must always be valid) * ident: parsed line data (can be NULL, in which case err_msg should be set) * err_msg: error message (NULL if none) * @@ -463,8 +467,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int mapping_number, int lineno, IdentLine *ident, - const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -482,6 +486,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -494,7 +500,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -534,7 +540,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open usermap file \"%s\": %m", IdentFileName))); - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -556,7 +562,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) mapping_number++; fill_ident_line(tuple_store, tupdesc, mapping_number, - tok_line->line_num, identline, tok_line->err_msg); + tok_line->file_name, tok_line->line_num, identline, + tok_line->err_msg); } /* Free tokenizer memory */ diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l index 4f43430db4..ecfb4a2a28 100644 --- a/src/backend/utils/misc/guc-file.l +++ b/src/backend/utils/misc/guc-file.l @@ -345,6 +345,110 @@ GUC_flex_fatal(const char *msg) return 0; /* keep compiler quiet */ } +/* + * Returns the list of config files located in a directory, in alphabetical + * order. + * + * We don't check for recursion or too-deep nesting depth here, its up to the + * caller to take care of that. + */ +char ** +GetDirConfFiles(const char *includedir, const char *calling_file, int elevel, + int *num_filenames, char **err_msg) +{ + char *directory; + DIR *d; + struct dirent *de; + char **filenames; + int size_filenames; + + /* + * Reject directory name that is all-blank (including empty), as that + * leads to confusion --- we'd read the containing directory, typically + * resulting in recursive inclusion of the same file(s). + */ + if (strspn(includedir, " \t\r\n") == strlen(includedir)) + { + ereport(elevel, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("empty configuration directory name: \"%s\"", + includedir))); + *err_msg = "empty configuration directory name"; + return NULL; + } + + directory = AbsoluteConfigLocation(includedir, calling_file); + d = AllocateDir(directory); + if (d == NULL) + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not open configuration directory \"%s\": %m", + directory))); + *err_msg = psprintf("could not open directory \"%s\"", directory); + filenames = NULL; + goto cleanup; + } + + /* + * Read the directory and put the filenames in an array, so we can sort + * them prior to caller processing the contents. + */ + size_filenames = 32; + filenames = (char **) palloc(size_filenames * sizeof(char *)); + *num_filenames = 0; + + while ((de = ReadDir(d, directory)) != NULL) + { + PGFileType de_type; + char filename[MAXPGPATH]; + + /* + * Only parse files with names ending in ".conf". Explicitly reject + * files starting with ".". This excludes things like "." and "..", + * as well as typical hidden files, backup files, and editor debris. + */ + if (strlen(de->d_name) < 6) + continue; + if (de->d_name[0] == '.') + continue; + if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) + continue; + + join_path_components(filename, directory, de->d_name); + canonicalize_path(filename); + de_type = get_dirent_type(filename, de, true, elevel); + if (de_type == PGFILETYPE_ERROR) + { + *err_msg = psprintf("could not stat file \"%s\"", filename); + pfree(filenames); + filenames = NULL; + goto cleanup; + } + else if (de_type != PGFILETYPE_DIR) + { + /* Add file to array, increasing its size in blocks of 32 */ + if (*num_filenames >= size_filenames) + { + size_filenames += 32; + filenames = (char **) repalloc(filenames, + size_filenames * sizeof(char *)); + } + filenames[*num_filenames] = pstrdup(filename); + (*num_filenames)++; + } + } + + if (*num_filenames > 0) + qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp); + +cleanup: + if (d) + FreeDir(d); + pfree(directory); + return filenames; +} + /* * Read and parse a single configuration file. This function recurses * to handle "include" directives. @@ -606,127 +710,30 @@ ParseConfigDirectory(const char *includedir, ConfigVariable **head_p, ConfigVariable **tail_p) { - char *directory; - DIR *d; - struct dirent *de; + char *err_msg; char **filenames; int num_filenames; - int size_filenames; - bool status; - - /* - * Reject directory name that is all-blank (including empty), as that - * leads to confusion --- we'd read the containing directory, typically - * resulting in recursive inclusion of the same file(s). - */ - if (strspn(includedir, " \t\r\n") == strlen(includedir)) - { - ereport(elevel, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("empty configuration directory name: \"%s\"", - includedir))); - record_config_file_error("empty configuration directory name", - calling_file, calling_lineno, - head_p, tail_p); - return false; - } - - /* - * We don't check for recursion or too-deep nesting depth here; the - * subsequent calls to ParseConfigFile will take care of that. - */ - - directory = AbsoluteConfigLocation(includedir, calling_file); - d = AllocateDir(directory); - if (d == NULL) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open configuration directory \"%s\": %m", - directory))); - record_config_file_error(psprintf("could not open directory \"%s\"", - directory), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } - /* - * Read the directory and put the filenames in an array, so we can sort - * them prior to processing the contents. - */ - size_filenames = 32; - filenames = (char **) palloc(size_filenames * sizeof(char *)); - num_filenames = 0; + filenames = GetDirConfFiles(includedir, calling_file, elevel, + &num_filenames, &err_msg); - while ((de = ReadDir(d, directory)) != NULL) + if (!filenames) { - PGFileType de_type; - char filename[MAXPGPATH]; - - /* - * Only parse files with names ending in ".conf". Explicitly reject - * files starting with ".". This excludes things like "." and "..", - * as well as typical hidden files, backup files, and editor debris. - */ - if (strlen(de->d_name) < 6) - continue; - if (de->d_name[0] == '.') - continue; - if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) - continue; - - join_path_components(filename, directory, de->d_name); - canonicalize_path(filename); - de_type = get_dirent_type(filename, de, true, elevel); - if (de_type == PGFILETYPE_ERROR) - { - record_config_file_error(psprintf("could not stat file \"%s\"", - filename), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } - else if (de_type != PGFILETYPE_DIR) - { - /* Add file to array, increasing its size in blocks of 32 */ - if (num_filenames >= size_filenames) - { - size_filenames += 32; - filenames = (char **) repalloc(filenames, - size_filenames * sizeof(char *)); - } - filenames[num_filenames] = pstrdup(filename); - num_filenames++; - } + record_config_file_error(err_msg, calling_file, calling_lineno, head_p, + tail_p); + return false; } - if (num_filenames > 0) + for (int i = 0; i < num_filenames; i++) { - int i; - - qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp); - for (i = 0; i < num_filenames; i++) - { - if (!ParseConfigFile(filenames[i], true, - calling_file, calling_lineno, - depth, elevel, - head_p, tail_p)) - { - status = false; - goto cleanup; - } - } + if (!ParseConfigFile(filenames[i], true, + calling_file, calling_lineno, + depth, elevel, + head_p, tail_p)) + return false; } - status = true; -cleanup: - if (d) - FreeDir(d); - pfree(directory); - return status; + return true; } /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 060bafaced..e6a54cc3d6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6128,16 +6128,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o}', - proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '6250', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,text,text,text}', - proargmodes => '{o,o,o,o,o,o}', - proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index 90036f7bcd..0ea100d1b8 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -79,6 +79,7 @@ typedef enum ClientCertName typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -155,6 +156,7 @@ typedef struct AuthToken typedef struct TokenizedAuthLine { List *fields; /* List of lists of AuthTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ @@ -174,6 +176,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel); extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel); extern bool pg_isblank(const char c); extern MemoryContext tokenize_auth_file(const char *filename, FILE *file, - List **tok_lines, int elevel); + List **tok_lines, int depth, + int elevel); #endif /* HBA_H */ diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h index 45ae1b537f..cb3bbcf949 100644 --- a/src/include/utils/guc.h +++ b/src/include/utils/guc.h @@ -145,6 +145,8 @@ typedef struct ConfigVariable struct ConfigVariable *next; } ConfigVariable; +extern char **GetDirConfFiles(const char *includedir, const char *calling_file, + int elevel, int *num_filenames, char **err_msg); extern bool ParseConfigFile(const char *config_file, bool strict, const char *calling_file, int calling_lineno, int depth, int elevel, diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl new file mode 100644 index 0000000000..8eae72b8d4 --- /dev/null +++ b/src/test/authentication/t/003_file_inclusion.pl @@ -0,0 +1,657 @@ + +# Copyright (c) 2021-2022, PostgreSQL Global Development Group + +# Set of tests for authentication and pg_hba.conf inclusion. +# This test can only run with Unix-domain sockets. + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use Time::HiRes qw(usleep); +use IPC::Run qw(pump finish timer); +use Data::Dumper; + +if (!$use_unix_sockets) +{ + plan skip_all => + "authentication tests cannot run without Unix-domain sockets"; +} + +# stores the current line counter for each file. hba_rule and ident_rule are +# fake file names used for the global rule number for each auth view. +my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + +my $hba_file = 'subdir1/pg_hba_custom.conf'; +my $ident_file = 'subdir2/pg_ident_custom.conf'; + +# Initialize primary node +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init; +$node->start; + +my $data_dir = $node->data_dir; + +# Normalize the data directory for Windows +$data_dir =~ s/\/\.\//\//g; # reduce /./ to / +$data_dir =~ s/\/\//\//g; # reduce // to / +$data_dir =~ s/\/$//; # remove trailing / + + +# Add the given payload to the given relative HBA file of the given node. +# This function maintains the %cur_line metadata, so it has to be called in the +# expected inclusion evaluation order in order to keep it in sync. +# +# If the payload starts with "include" or "ignore", the function doesn't +# increase the general hba rule number. +# +# If an err_str is provided, it returns an arrayref containing the provided +# filename, the current line number in that file and the provided err_str. The +# err_str has to be a valid regex string. +# Otherwise it only returns the line number of the payload in the wanted file. +# This function has to be called in the expected inclusion evaluation order to +# keep the %cur_line information in sync. +sub add_hba_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'hba_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_hba_file_rules line + @tokens = split(/ /, $payload); + $tokens[1] = '{' . $tokens[1] . '}'; # database + $tokens[2] = '{' . $tokens[2] . '}'; # user_name + + # add empty address and netmask betweed user_name and auth_method + splice @tokens, 3, 0, ''; + splice @tokens, 3, 0, ''; + + # append empty options and error + push @tokens, ''; + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Add the given payload to the given relative ident file of the given node. +# Same as add_hba_line but for pg_ident files +sub add_ident_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'ident_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_ident_file_mappings line + @tokens = split(/ /, $payload); + + # append empty error + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Delete pg_hba.conf from the given node, add various entries to test the +# include infrastructure and then execute a reload to refresh it. +sub generate_valid_auth_files +{ + my $node = shift; + my $hba_expected = ''; + my $ident_expected = ''; + + # customise main auth file names + $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'"); + $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'"); + + # and make original ones invalid to be sure they're not used anywhere + $node->append_conf('pg_hba.conf', "some invalid line"); + $node->append_conf('pg_ident.conf', "some invalid line"); + + # pg_hba stuff + mkdir("$data_dir/subdir1"); + mkdir("$data_dir/hba_inc"); + mkdir("$data_dir/hba_inc_if"); + mkdir("$data_dir/hba_pos"); + + # Make sure we will still be able to connect + $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust'); + + # Add include data + add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf"); + $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject"); + + $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject"); + + add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject"); + # include is relative to current path + add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject"); + + # include_if_exists data + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none"); + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some"); + $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject"); + + # include_dir data + add_hba_line($node, "$hba_file", "include_dir ../hba_inc"); + add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included"); + $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject"); + $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject"); + + # secondary auth file + add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject'); + $node->append_conf('dbnames.conf', "db1"); + $node->append_conf('dbnames.conf', "db3"); + $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1) + . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1) + . '|local|{db1,db3}|{all}|||reject||'; + + # pg_ident stuff + mkdir("$data_dir/subdir2"); + mkdir("$data_dir/ident_inc"); + mkdir("$data_dir/ident_inc_if"); + mkdir("$data_dir/ident_pos"); + + # Add include data + add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf"); + $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar"); + + $ident_expected .= add_ident_line($node, "$ident_file", "test a b"); + + add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar"); + # include is relative to current path + add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar"); + + # include_if_exists data + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none"); + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some"); + $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar"); + + # include_dir data + add_ident_line($node, "$ident_file", "include_dir ../ident_inc"); + add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included"); + $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar"); + $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar"); + + $node->restart; + $node->connect_ok('dbname=postgres', + 'Connection ok after generating valid auth files'); + + return ($hba_expected, $ident_expected); +} + +# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal +# entries to allow authentication. +sub reset_auth_files +{ + my $node = shift; + + unlink("$data_dir/$hba_file"); + unlink("$data_dir/$ident_file"); + + %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + + return add_hba_line($node, "$hba_file", 'local all all trust'); +} + +# Generate a list of expected error regex for the given array of error +# conditions, as generated by add_hba_line/add_ident_line with an err_str. +# +# 2 regex are generated per array entry: one for the given err_str, and one for +# the expected line in the specific file. Since all lines are independant, +# there's no guarantee that a specific failure regex and the per-line regex +# will match the same error. Calling code should add at least one test with a +# single error to make sure that the line number / file name is correct. +# +# On top of that, an extra line is generated for the general failure to process +# the main auth file. +sub generate_log_err_patterns +{ + my $node = shift; + my $raw_errors = shift; + my $is_hba_err = shift; + my @errors; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + push @errors, qr/$err_str/; + + # Context messages with the file / line location aren't always emitted + if ($err_str !~ /maximum nesting depth exceeded/ and + $err_str !~ /could not open secondary authentication file/) + { + push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/ + } + } + + push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err); + + return \@errors; +} + +# Generate the expected output for the auth file view error reporting (file +# name, file line, error), for the given array of error conditions, as +# generated generated by add_hba_line/add_ident_line with an err_str. +sub generate_log_err_rows +{ + my $node = shift; + my $raw_errors = shift; + my $exp_rows = ''; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + $exp_rows .= "\n" if ($exp_rows ne ""); + + # Unescape regex patterns if any + $err_str =~ s/\\([\(\)])/$1/g; + $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str" + } + + return $exp_rows; +} + +# Reset the main auth files, append the given payload to the given config file, +# and check that the instance cannot start, raising the expected error line(s). +sub start_errors_like +{ + my $node = shift; + my $file = shift; + my $payload = shift; + my $pattern = shift; + my $should_fail = shift; + + reset_auth_files($node); + $node->append_conf($file, $payload); + + unlink($node->logfile); + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir, + '-l', $node->logfile, 'start'); + + if ($should_fail) + { + ok($ret != 0, "Cannot start postgres with faulty $file"); + } + else + { + ok($ret == 0, "postgres can start with faulty $file"); + } + + my $log_contents = slurp_file($node->logfile); + + foreach (@{$pattern}) + { + like($log_contents, + $_, + "Expected failure found in the logs"); + } + + if (not $should_fail) + { + # We can't simply call $node->stop here as the call is optimized out + # when the server isn't started with $node->start. + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', + $data_dir, 'stop', '-m', 'fast'); + ok($ret == 0, "Could stop postgres"); + } +} + +# We should be able to connect, and see an empty pg_ident.conf +is($node->psql( + 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'), + qq(0), + 'pg_ident.conf is empty'); + +############################################ +# part 1, test view reporting for valid data +############################################ +my ($exp_hba, $exp_ident) = generate_valid_auth_files($node); + +$node->connect_ok('dbname=postgres', 'Connection still ok'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_hba_file_rules'), + qq($exp_hba), + 'pg_hba_file_rules content is expected'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_ident_file_mappings'), + qq($exp_ident), + 'pg_ident_file_mappings content is expected'); + +############################################# +# part 2, test log reporting for invalid data +############################################# +reset_auth_files($node); +$node->restart('fast'); +$node->connect_ok('dbname=postgres', + 'Connection ok after resetting auth files'); + +$node->stop('fast'); + +start_errors_like($node, $hba_file, "include ../not_a_file", + [ + qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file +mkdir("$data_dir/hba_inc_fail"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token"); +start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail", + [ + qr/invalid connection type "not_a_token"/, + qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file with nested inclusion +unlink("$data_dir/hba_inc_fail/inc_dir.conf"); +my @hba_raw_errors_step1; + +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1"); + +add_hba_line($node, "hba_inc_fail/file1", "include file2"); +add_hba_line($node, "hba_inc_fail/file2", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file2", "include file3"); + +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3", + "local all all zuul", + 'invalid authentication method "zuul"'); + +start_errors_like( + $node, $hba_file, "include_dir ../hba_inc_fail", + generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @hba_raw_errors_step2; + +# include_if_exists, with various problems +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local", + "end-of-line before database specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local,host", + "multiple values specified for connection type"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all", + "end-of-line before role specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all all", + "end-of-line before authentication method"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "host all all test/42", + 'specifying both host name and CIDR mask is invalid: "test/42"'); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + 'local @dbnames_fails.conf all reject', + "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory"); + +add_hba_line($node, "hba_if_exists.conf", "include recurse.conf"); +push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf", + "include recurse.conf", + 'could not open configuration file "recurse.conf": maximum nesting depth exceeded'); + +# Generate the regex for the expected errors in the logs. There's no guarantee +# that the generated "line X of file..." will be emitted for the expected line, +# but previous tests already ensured that the correct line number / file name +# was emitted, so ensuring that there's an error in all expected lines is +# enough here. +my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2, + 1); + +# Not an error, but it should raise a message in the logs. Manually add an +# extra log message to detect +add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none"); +push @{$expected_errors}, + qr/skipping missing authentication file "$data_dir\/if_exists_none"/; + +start_errors_like( + $node, $hba_file, "include_if_exists ../hba_if_exists.conf", + $expected_errors, 1); + +# Mostly the same, but for ident files +reset_auth_files($node); + +my @ident_raw_errors_step1; + +# include_dir, single included file with nested inclusion +mkdir("$data_dir/ident_inc_fail"); +add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1"); + +add_ident_line($node, "ident_inc_fail/file1", "include file2"); +add_ident_line($node, "ident_inc_fail/file2", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file2", "include file3"); + +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3", + "failmap /(fail postgres", + 'invalid regular expression "\(fail": parentheses \(\) not balanced'); + +start_errors_like( + $node, $ident_file, "include_dir ../ident_inc_fail", + generate_log_err_patterns($node, \@ident_raw_errors_step1, 0), + 0); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @ident_raw_errors_step2; + +# include_if_exists, with various problems +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map", + "missing entry at end of line"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2", + "multiple values in ident field"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", + 'map @osnames_fails.conf postgres', + "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory"); + +add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf", + 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded'); + +start_errors_like( + $node, $ident_file, "include_if_exists ../ident_if_exists.conf", + # There's no guarantee that the generated "line X of file..." will be + # emitted for the expected line, but previous tests already ensured that + # the correct line number / file name was emitted, so ensuring that there's + # an error in all expected lines is enough here. + generate_log_err_patterns($node, \@ident_raw_errors_step2, 0), + 0); + +##################################################### +# part 3, test reporting of various error scenario +# NOTE: this will be bypassed -DEXEC_BACKEND or win32 +##################################################### +reset_auth_files($node); + +$node->start; +$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset'); + +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_hba_file_rules'); + +add_ident_line($node, $ident_file, ''); +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_ident_file_mappings'); + +# The instance could be restarted and no error is detected. Now check if the +# build is compatible with the view error reporting (EXEC_BACKEND / win32 will +# fail when trying to connect as they always rely on the current auth files +# content) +my @hba_raw_errors; + +push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +my ($stdout, $stderr); +my $cmdret = $node->psql('postgres', 'SELECT 1', + stdout => \$stdout, stderr => \$stderr); + +if ($cmdret != 0) +{ + # Connection failed. Bail out, but make sure to raise a failure if it + # didn't fail for the expected hba file modification. + like($stderr, + qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/, + "Connection failed due to loading an invalid hba file"); + + done_testing(); + diag("Build not compatible with auth file view error reporting, bail out.\n"); + exit; +} + +# Combine errors generated at step 2, in the same order. +$node->append_conf($hba_file, "include_dir ../hba_inc_fail"); +push @hba_raw_errors, @hba_raw_errors_step1; + +$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf"); +push @hba_raw_errors, @hba_raw_errors_step2; + +my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules' + . ' WHERE error IS NOT NULL ORDER BY rule_number'), + qq($hba_expected), + 'Detected all error in hba file'); + +# and do the same for pg_ident +my @ident_raw_errors; + +push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +$node->append_conf($ident_file, "include_dir ../ident_inc_fail"); +push @ident_raw_errors, @ident_raw_errors_step1; + +$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf"); +push @ident_raw_errors, @ident_raw_errors_step2; + +my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings' + . ' WHERE error IS NOT NULL ORDER BY mapping_number'), + qq($ident_expected), + 'Detected all error in ident file'); + +done_testing(); diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 79408710e0..5ed2fe3704 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname, FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); pg_hba_file_rules| SELECT a.rule_number, + a.file_name, a.line_number, a.type, a.database, @@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error); + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, a.line_number, a.map_name, a.sys_name, a.pg_username, a.error - FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.37.0 --fy3hxyvyv2spu3wq Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v10-0003-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v12 2/3] Allow file inclusion in pg_hba and pg_ident files. @ 2022-05-30 03:15 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-05-30 03:15 UTC (permalink / raw) pg_hba.conf file now has support for "include", "include_dir" and "include_if_exists" directives, which work similarly to the same directives in the postgresql.conf file. This fixes a possible crash if a secondary file tries to include itself as there's now a nesting depth check in the inclusion code path, same as the postgresql.conf. Many regression tests added to cover both the new directives, but also error detection for the whole pg_hba / pg_ident files. Catversion is bumped. Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/client-auth.sgml | 86 ++- doc/src/sgml/system-views.sgml | 22 +- src/backend/libpq/hba.c | 483 ++++++++++--- src/backend/libpq/pg_hba.conf.sample | 25 +- src/backend/libpq/pg_ident.conf.sample | 15 +- src/backend/utils/adt/hbafuncs.c | 43 +- src/backend/utils/misc/guc-file.l | 229 +++--- src/include/catalog/pg_proc.dat | 12 +- src/include/libpq/hba.h | 5 +- src/include/utils/guc.h | 2 + .../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++ src/test/regress/expected/rules.out | 6 +- 12 files changed, 1311 insertions(+), 274 deletions(-) create mode 100644 src/test/authentication/t/003_file_inclusion.pl diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index 32d5d45863..2ae723de66 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,23 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication + record. Inclusion directives specify files that can be included, which + contains additional records. The records will be inserted in lieu of the + inclusion records. Those records only contains two fields: the + <literal>include</literal>, <literal>include_if_exists</literal> or + <literal>include_dir</literal> directive and the file or directory to be + included. The file or directory can be a relative of absolute path, and can + be double quoted if needed. For the <literal>include_dir</literal> form, + all files not starting with a <literal>.</literal> and ending with + <literal>.conf</literal> will be included. Multiple files within an include + directory are processed in file name order (according to C locale rules, + i.e., numbers before letters, and uppercase letters before lowercase ones). + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,21 +118,57 @@ <para> A record can have several formats: <synopsis> -local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +include <replaceable>file</replaceable> +include_if_exists <replaceable>file</replaceable> +include_dir <replaceable>directory</replaceable> +local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> </synopsis> The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_if_exists</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file if the + file exists and can be read. Otherwise, a message will be logged to + indicate that the file is skipped. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_dir</literal></term> + <listitem> + <para> + This line will be replaced with the content of all the files found in + the directory, if they don't start with a <literal>.</literal> and end + with <literal>.conf</literal>, processed in file name order (according + to C locale rules, i.e., numbers before letters, and uppercase letters + before lowercase ones). + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -863,8 +914,10 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> +<replaceable>include_dir</replaceable> <replaceable>directory</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -875,6 +928,11 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + As for <filename>pg_hba.conf</filename>, the lines in this file can either + be inclusion directives or user name map records, and follow the same + rules. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 4723f712a7..7d1cec8b7f 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -1003,12 +1003,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule the given <literal>file_name</literal> </para></entry> </row> @@ -1153,12 +1162,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this mapping + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_ident.conf</filename> + Line number of this mapping in the given <literal>file_name</literal> </para></entry> </row> diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index ea92f02a47..183312b10a 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -22,6 +22,7 @@ #include <sys/param.h> #include <sys/socket.h> #include <netdb.h> +#include <sys/stat.h> #include <netinet/in.h> #include <arpa/inet.h> #include <unistd.h> @@ -70,6 +71,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -115,14 +122,26 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int depth, + int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, - const char *inc_filename, int elevel, char **err_msg); + const char *inc_filename, int depth, int elevel, + char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); static int regcomp_auth_token(AuthToken *token, char *filename, int line_num, char **err_msg, int elevel); static int regexec_auth_token(const char *match, AuthToken *token, size_t nmatch, regmatch_t pmatch[]); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + bool strict, const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); +static char *process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, + int elevel, MemoryContext linecxt, + List **tok_lines); /* @@ -413,7 +432,7 @@ regexec_auth_token(const char *match, AuthToken *token, size_t nmatch, */ static List * next_field_expand(const char *filename, char **lineptr, - int elevel, char **err_msg) + int depth, int elevel, char **err_msg) { char buf[MAX_TOKEN]; bool trailing_comma; @@ -429,7 +448,7 @@ next_field_expand(const char *filename, char **lineptr, /* Is this referencing a file? */ if (!initial_quote && buf[0] == '@' && buf[1] != '\0') - tokens = tokenize_inc_file(tokens, filename, buf + 1, + tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1, elevel, err_msg); else tokens = lappend(tokens, make_auth_token(buf, initial_quote)); @@ -457,6 +476,7 @@ static List * tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, + int depth, int elevel, char **err_msg) { @@ -466,39 +486,30 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); + *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", *err_msg))); + return tokens; } - inc_file = AllocateFile(inc_fullname, "r"); - if (inc_file == NULL) - { - int save_errno = errno; + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true, + outer_filename, elevel, err_msg, &inc_fullname); - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); + if (inc_file == NULL) return tokens; - } /* There is possible recursion here if the file contains @ */ - linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); + linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1, + elevel); FreeFile(inc_file); pfree(inc_fullname); @@ -536,11 +547,38 @@ tokenize_inc_file(List *tokens, /* * tokenize_auth_file - * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a dedicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, + int depth, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, depth, + elevel); + + return linecxt; +} + +/* + * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see the struct definition * in libpq/hba.h. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -549,30 +587,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int depth, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -625,7 +655,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, { List *current_field; - current_field = next_field_expand(filename, &lineptr, + current_field = next_field_expand(filename, &lineptr, depth, elevel, &err_msg); /* add field to line, unless we are at EOL or comment start */ if (current_field != NIL) @@ -633,29 +663,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + AuthToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); + + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; + + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, true, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } + else if (strcmp(first->string, "include_dir") == 0) + { + char **filenames; + char *dir_name = second->string; + int num_filenames; + StringInfoData err_buf; + + filenames = GetDirConfFiles(dir_name, filename, elevel, + &num_filenames, &err_msg); + + if (!filenames) + { + /* We have the error in err_msg, simply process it */ + goto process_line; + } + + initStringInfo(&err_buf); + for (int i = 0; i < num_filenames; i++) + { + /* + * err_msg is used here as a temp buffer, it will be + * overwritten at the end of the loop with the + * cumulated errors, if any. + */ + err_msg = process_included_authfile(filenames[i], true, + filename, depth + 1, elevel, + linecxt, tok_lines); + + /* Cumulate errors if any. */ + if (err_msg) + { + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfoString(&err_buf, err_msg); + } + } + + /* + * If there were no errors, the line is fully processed, bypass + * the general TokenizedAuthLine processing. + */ + if (err_buf.len == 0) + goto next_line; + + /* Otherwise, process the cumulated errors, if any. */ + err_msg = err_buf.data; + } + else if (strcmp(first->string, "include_if_exists") == 0) + { + char *inc_filename; - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, false, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } } +process_line: + /* + * General processing: report the error if any and emit line to the + * TokenizedAuthLine + */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -984,7 +1112,7 @@ do { \ errmsg("authentication option \"%s\" is only valid for authentication methods %s", \ optname, _(validmethods)), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication option \"%s\" is only valid for authentication methods %s", \ optname, validmethods); \ return false; \ @@ -1004,7 +1132,7 @@ do { \ errmsg("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, HbaFileName))); \ + line_num, file_name))); \ *err_msg = psprintf("authentication method \"%s\" requires argument \"%s\" to be set", \ authname, argname); \ return NULL; \ @@ -1027,7 +1155,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("missing entry at end of line"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, file_name))); \ *err_msg = pstrdup("missing entry at end of line"); \ return NULL; \ } \ @@ -1040,7 +1168,7 @@ do { \ (errcode(ERRCODE_CONFIG_FILE_ERROR), \ errmsg("multiple values in ident field"), \ errcontext("line %d of configuration file \"%s\"", \ - line_num, IdentFileName))); \ + line_num, file_name))); \ *err_msg = pstrdup("multiple values in ident field"); \ return NULL; \ } \ @@ -1063,6 +1191,7 @@ HbaLine * parse_hba_line(TokenizedAuthLine *tok_line, int elevel) { int line_num = tok_line->line_num; + char *file_name = tok_line->file_name; char **err_msg = &tok_line->err_msg; char *str; struct addrinfo *gai_result; @@ -1077,6 +1206,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) HbaLine *parsedline; parsedline = palloc0(sizeof(HbaLine)); + parsedline->sourcefile = pstrdup(file_name); parsedline->linenumber = line_num; parsedline->rawline = pstrdup(tok_line->raw_line); @@ -1091,7 +1221,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for connection type"), errhint("Specify exactly one connection type per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for connection type"; return NULL; } @@ -1119,7 +1249,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("hostssl record cannot match because SSL is disabled"), errhint("Set ssl = on in postgresql.conf."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostssl record cannot match because SSL is disabled"; } #else @@ -1127,7 +1257,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("hostssl record cannot match because SSL is not supported by this build"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostssl record cannot match because SSL is not supported by this build"; #endif } @@ -1139,7 +1269,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("hostgssenc record cannot match because GSSAPI is not supported by this build"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "hostgssenc record cannot match because GSSAPI is not supported by this build"; #endif } @@ -1160,7 +1290,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid connection type \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid connection type \"%s\"", token->string); return NULL; } @@ -1173,7 +1303,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before database specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before database specification"; return NULL; } @@ -1198,7 +1328,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before role specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before role specification"; return NULL; } @@ -1225,7 +1355,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before IP address specification"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before IP address specification"; return NULL; } @@ -1237,7 +1367,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for host address"), errhint("Specify one address range per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for host address"; return NULL; } @@ -1296,7 +1426,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid IP address \"%s\": %s", str, gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid IP address \"%s\": %s", str, gai_strerror(ret)); if (gai_result) @@ -1316,7 +1446,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("specifying both host name and CIDR mask is invalid: \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("specifying both host name and CIDR mask is invalid: \"%s\"", token->string); return NULL; @@ -1330,7 +1460,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid CIDR mask in address \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid CIDR mask in address \"%s\"", token->string); return NULL; @@ -1350,7 +1480,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("end-of-line before netmask specification"), errhint("Specify an address range in CIDR notation, or provide a separate netmask."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before netmask specification"; return NULL; } @@ -1361,7 +1491,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("multiple values specified for netmask"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for netmask"; return NULL; } @@ -1376,7 +1506,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid IP mask \"%s\": %s", token->string, gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid IP mask \"%s\": %s", token->string, gai_strerror(ret)); if (gai_result) @@ -1395,7 +1525,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("IP address and mask do not match"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "IP address and mask do not match"; return NULL; } @@ -1411,7 +1541,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("end-of-line before authentication method"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "end-of-line before authentication method"; return NULL; } @@ -1423,7 +1553,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("multiple values specified for authentication type"), errhint("Specify exactly one authentication type per line."), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "multiple values specified for authentication type"; return NULL; } @@ -1460,7 +1590,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("MD5 authentication is not supported when \"db_user_namespace\" is enabled"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "MD5 authentication is not supported when \"db_user_namespace\" is enabled"; return NULL; } @@ -1501,7 +1631,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid authentication method \"%s\"", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid authentication method \"%s\"", token->string); return NULL; @@ -1514,7 +1644,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) errmsg("invalid authentication method \"%s\": not supported by this build", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid authentication method \"%s\": not supported by this build", token->string); return NULL; @@ -1536,7 +1666,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("gssapi authentication is not supported on local sockets"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "gssapi authentication is not supported on local sockets"; return NULL; } @@ -1548,7 +1678,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("peer authentication is only supported on local sockets"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "peer authentication is only supported on local sockets"; return NULL; } @@ -1566,7 +1696,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cert authentication is only supported on hostssl connections"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cert authentication is only supported on hostssl connections"; return NULL; } @@ -1616,7 +1746,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("authentication option not in name=value format: %s", token->string), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("authentication option not in name=value format: %s", token->string); return NULL; @@ -1660,7 +1790,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cannot use ldapbasedn, ldapbinddn, ldapbindpasswd, ldapsearchattribute, ldapsearchfilter, or ldapurl together with ldapprefix"; return NULL; } @@ -1671,7 +1801,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "authentication method \"ldap\" requires argument \"ldapbasedn\", \"ldapprefix\", or \"ldapsuffix\" to be set"; return NULL; } @@ -1687,7 +1817,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("cannot use ldapsearchattribute together with ldapsearchfilter"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "cannot use ldapsearchattribute together with ldapsearchfilter"; return NULL; } @@ -1704,7 +1834,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("list of RADIUS servers cannot be empty"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "list of RADIUS servers cannot be empty"; return NULL; } @@ -1715,7 +1845,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("list of RADIUS secrets cannot be empty"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "list of RADIUS secrets cannot be empty"; return NULL; } @@ -1734,7 +1864,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiussecrets), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS secrets (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiussecrets), list_length(parsedline->radiusservers)); @@ -1750,7 +1880,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiusports), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS ports (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiusports), list_length(parsedline->radiusservers)); @@ -1766,7 +1896,7 @@ parse_hba_line(TokenizedAuthLine *tok_line, int elevel) list_length(parsedline->radiusidentifiers), list_length(parsedline->radiusservers)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("the number of RADIUS identifiers (%d) must be 1 or the same as the number of RADIUS servers (%d)", list_length(parsedline->radiusidentifiers), list_length(parsedline->radiusservers)); @@ -1801,6 +1931,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg) { int line_num = hbaline->linenumber; + char *file_name = hbaline->sourcefile; #ifdef USE_LDAP hbaline->ldapscope = LDAP_SCOPE_SUBTREE; @@ -1824,7 +1955,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientcert can only be configured for \"hostssl\" rows"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientcert can only be configured for \"hostssl\" rows"; return false; } @@ -1841,7 +1972,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientcert only accepts \"verify-full\" when using \"cert\" authentication"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientcert can only be set to \"verify-full\" when using \"cert\" authentication"; return false; } @@ -1854,7 +1985,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid value for clientcert: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } } @@ -1866,7 +1997,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("clientname can only be configured for \"hostssl\" rows"), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = "clientname can only be configured for \"hostssl\" rows"; return false; } @@ -1885,7 +2016,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid value for clientname: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } } @@ -1971,7 +2102,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid ldapscheme value: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); hbaline->ldapscheme = pstrdup(val); } else if (strcmp(name, "ldapserver") == 0) @@ -1989,7 +2120,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid LDAP port number: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid LDAP port number: \"%s\"", val); return false; } @@ -2083,7 +2214,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS server list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2102,7 +2233,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not translate RADIUS server name \"%s\" to address: %s", (char *) lfirst(l), gai_strerror(ret)), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); if (gai_result) pg_freeaddrinfo_all(hints.ai_family, gai_result); @@ -2131,7 +2262,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS port list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("invalid RADIUS port number: \"%s\"", val); return false; } @@ -2144,7 +2275,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, (errcode(ERRCODE_CONFIG_FILE_ERROR), errmsg("invalid RADIUS port number: \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2167,7 +2298,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS secret list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2189,7 +2320,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("could not parse RADIUS identifiers list \"%s\"", val), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); return false; } @@ -2203,7 +2334,7 @@ parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, errmsg("unrecognized authentication option name: \"%s\"", name), errcontext("line %d of configuration file \"%s\"", - line_num, HbaFileName))); + line_num, file_name))); *err_msg = psprintf("unrecognized authentication option name: \"%s\"", name); return false; @@ -2351,7 +2482,7 @@ load_hba(void) return false; } - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ @@ -2441,6 +2572,135 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + if (strict) + { + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + } + else + { + Assert(kind == IncludedAuthFile); + ereport(LOG, + (errmsg("skipping missing authentication file \"%s\"", + *inc_fullname))); + } + + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} + +/* + * Try to open an included file, and tokenize it using the given context. + * Returns NULL if no error happens during tokenization, otherwise the error. + */ +static char * +process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, int elevel, + MemoryContext linecxt, List **tok_lines) +{ + char *inc_fullname; + FILE *inc_file; + char *err_msg = NULL; + + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) + { + err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", err_msg))); + return err_msg; + } + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict, + outer_filename, elevel, &err_msg, &inc_fullname); + + if (inc_file == NULL) + { + if (strict) + { + /* open_inc_file should have reported an error. */ + Assert(err_msg != NULL); + return err_msg; + } + else + return NULL; + } + else + { + /* No error message should have been reported. */ + Assert(err_msg == NULL); + } + + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, depth, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + return NULL; +} /* * Parse one tokenised line from the ident config file and store the result in @@ -2459,6 +2719,7 @@ load_hba(void) IdentLine * parse_ident_line(TokenizedAuthLine *tok_line, int elevel) { + char *file_name = tok_line->file_name; int line_num = tok_line->line_num; char **err_msg = &tok_line->err_msg; ListCell *field; @@ -2500,8 +2761,8 @@ parse_ident_line(TokenizedAuthLine *tok_line, int elevel) * Now that the field validation is done, compile a regex from the user * token, if necessary. */ - if (regcomp_auth_token(parsedline->token, IdentFileName, line_num, - err_msg, elevel)) + if (regcomp_auth_token(parsedline->token, file_name, line_num, err_msg, + elevel)) { /* err_msg includes the error to report */ return NULL; @@ -2722,7 +2983,7 @@ load_ident(void) return false; } - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..7433050112 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,16 +9,27 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # -# local DATABASE USER METHOD [OPTIONS] -# host DATABASE USER ADDRESS METHOD [OPTIONS] -# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# local DATABASE USER METHOD [OPTIONS] +# host DATABASE USER ADDRESS METHOD [OPTIONS] +# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..8e3fa29135 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,23 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # -# MAPNAME SYSTEM-USERNAME PG-USERNAME +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index 21a451e391..ea8de9a057 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -26,12 +26,12 @@ static ArrayType *get_hba_options(HbaLine *hba); static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int rule_number, int lineno, HbaLine *hba, - const char *err_msg); + int rule_number, const char *filename, int lineno, + HbaLine *hba, const char *err_msg); static void fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int mapping_number, int lineno, IdentLine *ident, - const char *err_msg); + int mapping_number, const char *filename, + int lineno, IdentLine *ident, const char *err_msg); static void fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc); @@ -159,7 +159,7 @@ get_hba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 10 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line @@ -168,7 +168,8 @@ get_hba_options(HbaLine *hba) * tuple_store: where to store data * tupdesc: tuple descriptor for the view * rule_number: unique rule identifier among all valid rules - * lineno: pg_hba.conf line number (must always be valid) + * filename: name of the file containing that line + * lineno: line number in that file (must always be valid) * hba: parsed line data (can be NULL, in which case err_msg should be set) * err_msg: error message (NULL if none) * @@ -177,7 +178,7 @@ get_hba_options(HbaLine *hba) */ static void fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int rule_number, int lineno, HbaLine *hba, + int rule_number, const char *filename, int lineno, HbaLine *hba, const char *err_msg) { Datum values[NUM_PG_HBA_FILE_RULES_ATTS]; @@ -202,6 +203,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -345,7 +348,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -386,7 +389,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open configuration file \"%s\": %m", HbaFileName))); - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -407,8 +410,8 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) rule_number++; - fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->line_num, - hbaline, tok_line->err_msg); + fill_hba_line(tuple_store, tupdesc, rule_number, tok_line->file_name, + tok_line->line_num, hbaline, tok_line->err_msg); } /* Free tokenizer memory */ @@ -445,7 +448,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) } /* Number of columns in pg_hba_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6 +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -454,7 +457,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) * tuple_store: where to store data * tupdesc: tuple descriptor for the view * mapping_number: unique rule identifier among all valid rules - * lineno: pg_ident.conf line number (must always be valid) + * filename: name of the file containing that line + * lineno: line number in that file (must always be valid) * ident: parsed line data (can be NULL, in which case err_msg should be set) * err_msg: error message (NULL if none) * @@ -463,8 +467,8 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) */ static void fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, - int mapping_number, int lineno, IdentLine *ident, - const char *err_msg) + int mapping_number, const char *filename, int lineno, + IdentLine *ident, const char *err_msg) { Datum values[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; bool nulls[NUM_PG_IDENT_FILE_MAPPINGS_ATTS]; @@ -482,6 +486,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(filename); /* line_number */ values[index++] = Int32GetDatum(lineno); @@ -494,7 +500,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -534,7 +540,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open usermap file \"%s\": %m", IdentFileName))); - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -556,7 +562,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) mapping_number++; fill_ident_line(tuple_store, tupdesc, mapping_number, - tok_line->line_num, identline, tok_line->err_msg); + tok_line->file_name, tok_line->line_num, identline, + tok_line->err_msg); } /* Free tokenizer memory */ diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l index 721628c0cf..86b6cc1c8a 100644 --- a/src/backend/utils/misc/guc-file.l +++ b/src/backend/utils/misc/guc-file.l @@ -345,6 +345,110 @@ GUC_flex_fatal(const char *msg) return 0; /* keep compiler quiet */ } +/* + * Returns the list of config files located in a directory, in alphabetical + * order. + * + * We don't check for recursion or too-deep nesting depth here, its up to the + * caller to take care of that. + */ +char ** +GetDirConfFiles(const char *includedir, const char *calling_file, int elevel, + int *num_filenames, char **err_msg) +{ + char *directory; + DIR *d; + struct dirent *de; + char **filenames; + int size_filenames; + + /* + * Reject directory name that is all-blank (including empty), as that + * leads to confusion --- we'd read the containing directory, typically + * resulting in recursive inclusion of the same file(s). + */ + if (strspn(includedir, " \t\r\n") == strlen(includedir)) + { + ereport(elevel, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("empty configuration directory name: \"%s\"", + includedir))); + *err_msg = "empty configuration directory name"; + return NULL; + } + + directory = AbsoluteConfigLocation(includedir, calling_file); + d = AllocateDir(directory); + if (d == NULL) + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not open configuration directory \"%s\": %m", + directory))); + *err_msg = psprintf("could not open directory \"%s\"", directory); + filenames = NULL; + goto cleanup; + } + + /* + * Read the directory and put the filenames in an array, so we can sort + * them prior to caller processing the contents. + */ + size_filenames = 32; + filenames = (char **) palloc(size_filenames * sizeof(char *)); + *num_filenames = 0; + + while ((de = ReadDir(d, directory)) != NULL) + { + PGFileType de_type; + char filename[MAXPGPATH]; + + /* + * Only parse files with names ending in ".conf". Explicitly reject + * files starting with ".". This excludes things like "." and "..", + * as well as typical hidden files, backup files, and editor debris. + */ + if (strlen(de->d_name) < 6) + continue; + if (de->d_name[0] == '.') + continue; + if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) + continue; + + join_path_components(filename, directory, de->d_name); + canonicalize_path(filename); + de_type = get_dirent_type(filename, de, true, elevel); + if (de_type == PGFILETYPE_ERROR) + { + *err_msg = psprintf("could not stat file \"%s\"", filename); + pfree(filenames); + filenames = NULL; + goto cleanup; + } + else if (de_type != PGFILETYPE_DIR) + { + /* Add file to array, increasing its size in blocks of 32 */ + if (*num_filenames >= size_filenames) + { + size_filenames += 32; + filenames = (char **) repalloc(filenames, + size_filenames * sizeof(char *)); + } + filenames[*num_filenames] = pstrdup(filename); + (*num_filenames)++; + } + } + + if (*num_filenames > 0) + qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp); + +cleanup: + if (d) + FreeDir(d); + pfree(directory); + return filenames; +} + /* * Read and parse a single configuration file. This function recurses * to handle "include" directives. @@ -606,127 +710,30 @@ ParseConfigDirectory(const char *includedir, ConfigVariable **head_p, ConfigVariable **tail_p) { - char *directory; - DIR *d; - struct dirent *de; + char *err_msg; char **filenames; int num_filenames; - int size_filenames; - bool status; - - /* - * Reject directory name that is all-blank (including empty), as that - * leads to confusion --- we'd read the containing directory, typically - * resulting in recursive inclusion of the same file(s). - */ - if (strspn(includedir, " \t\r\n") == strlen(includedir)) - { - ereport(elevel, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("empty configuration directory name: \"%s\"", - includedir))); - record_config_file_error("empty configuration directory name", - calling_file, calling_lineno, - head_p, tail_p); - return false; - } - - /* - * We don't check for recursion or too-deep nesting depth here; the - * subsequent calls to ParseConfigFile will take care of that. - */ - - directory = AbsoluteConfigLocation(includedir, calling_file); - d = AllocateDir(directory); - if (d == NULL) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open configuration directory \"%s\": %m", - directory))); - record_config_file_error(psprintf("could not open directory \"%s\"", - directory), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } - /* - * Read the directory and put the filenames in an array, so we can sort - * them prior to processing the contents. - */ - size_filenames = 32; - filenames = (char **) palloc(size_filenames * sizeof(char *)); - num_filenames = 0; + filenames = GetDirConfFiles(includedir, calling_file, elevel, + &num_filenames, &err_msg); - while ((de = ReadDir(d, directory)) != NULL) + if (!filenames) { - PGFileType de_type; - char filename[MAXPGPATH]; - - /* - * Only parse files with names ending in ".conf". Explicitly reject - * files starting with ".". This excludes things like "." and "..", - * as well as typical hidden files, backup files, and editor debris. - */ - if (strlen(de->d_name) < 6) - continue; - if (de->d_name[0] == '.') - continue; - if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) - continue; - - join_path_components(filename, directory, de->d_name); - canonicalize_path(filename); - de_type = get_dirent_type(filename, de, true, elevel); - if (de_type == PGFILETYPE_ERROR) - { - record_config_file_error(psprintf("could not stat file \"%s\"", - filename), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } - else if (de_type != PGFILETYPE_DIR) - { - /* Add file to array, increasing its size in blocks of 32 */ - if (num_filenames >= size_filenames) - { - size_filenames += 32; - filenames = (char **) repalloc(filenames, - size_filenames * sizeof(char *)); - } - filenames[num_filenames] = pstrdup(filename); - num_filenames++; - } + record_config_file_error(err_msg, calling_file, calling_lineno, head_p, + tail_p); + return false; } - if (num_filenames > 0) + for (int i = 0; i < num_filenames; i++) { - int i; - - qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp); - for (i = 0; i < num_filenames; i++) - { - if (!ParseConfigFile(filenames[i], true, - calling_file, calling_lineno, - depth, elevel, - head_p, tail_p)) - { - status = false; - goto cleanup; - } - } + if (!ParseConfigFile(filenames[i], true, + calling_file, calling_lineno, + depth, elevel, + head_p, tail_p)) + return false; } - status = true; -cleanup: - if (d) - FreeDir(d); - pfree(directory); - return status; + return true; } /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4f5d05d0ce..2ad06c4d3e 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6135,16 +6135,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o}', - proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '6250', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,text,text,text}', - proargmodes => '{o,o,o,o,o,o}', - proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index cec2e2665f..6b3bf814b5 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -93,6 +93,7 @@ typedef struct AuthToken typedef struct HbaLine { + char *sourcefile; int linenumber; char *rawline; ConnType conntype; @@ -157,6 +158,7 @@ typedef struct IdentLine typedef struct TokenizedAuthLine { List *fields; /* List of lists of AuthTokens */ + char *file_name; /* File name */ int line_num; /* Line number */ char *raw_line; /* Raw line text */ char *err_msg; /* Error message if any */ @@ -176,6 +178,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel); extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel); extern bool pg_isblank(const char c); extern MemoryContext tokenize_auth_file(const char *filename, FILE *file, - List **tok_lines, int elevel); + List **tok_lines, int depth, + int elevel); #endif /* HBA_H */ diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h index b3aaff9665..59ca39d908 100644 --- a/src/include/utils/guc.h +++ b/src/include/utils/guc.h @@ -144,6 +144,8 @@ typedef struct ConfigVariable struct ConfigVariable *next; } ConfigVariable; +extern char **GetDirConfFiles(const char *includedir, const char *calling_file, + int elevel, int *num_filenames, char **err_msg); extern bool ParseConfigFile(const char *config_file, bool strict, const char *calling_file, int calling_lineno, int depth, int elevel, diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl new file mode 100644 index 0000000000..8eae72b8d4 --- /dev/null +++ b/src/test/authentication/t/003_file_inclusion.pl @@ -0,0 +1,657 @@ + +# Copyright (c) 2021-2022, PostgreSQL Global Development Group + +# Set of tests for authentication and pg_hba.conf inclusion. +# This test can only run with Unix-domain sockets. + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use Time::HiRes qw(usleep); +use IPC::Run qw(pump finish timer); +use Data::Dumper; + +if (!$use_unix_sockets) +{ + plan skip_all => + "authentication tests cannot run without Unix-domain sockets"; +} + +# stores the current line counter for each file. hba_rule and ident_rule are +# fake file names used for the global rule number for each auth view. +my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + +my $hba_file = 'subdir1/pg_hba_custom.conf'; +my $ident_file = 'subdir2/pg_ident_custom.conf'; + +# Initialize primary node +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init; +$node->start; + +my $data_dir = $node->data_dir; + +# Normalize the data directory for Windows +$data_dir =~ s/\/\.\//\//g; # reduce /./ to / +$data_dir =~ s/\/\//\//g; # reduce // to / +$data_dir =~ s/\/$//; # remove trailing / + + +# Add the given payload to the given relative HBA file of the given node. +# This function maintains the %cur_line metadata, so it has to be called in the +# expected inclusion evaluation order in order to keep it in sync. +# +# If the payload starts with "include" or "ignore", the function doesn't +# increase the general hba rule number. +# +# If an err_str is provided, it returns an arrayref containing the provided +# filename, the current line number in that file and the provided err_str. The +# err_str has to be a valid regex string. +# Otherwise it only returns the line number of the payload in the wanted file. +# This function has to be called in the expected inclusion evaluation order to +# keep the %cur_line information in sync. +sub add_hba_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'hba_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_hba_file_rules line + @tokens = split(/ /, $payload); + $tokens[1] = '{' . $tokens[1] . '}'; # database + $tokens[2] = '{' . $tokens[2] . '}'; # user_name + + # add empty address and netmask betweed user_name and auth_method + splice @tokens, 3, 0, ''; + splice @tokens, 3, 0, ''; + + # append empty options and error + push @tokens, ''; + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Add the given payload to the given relative ident file of the given node. +# Same as add_hba_line but for pg_ident files +sub add_ident_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'ident_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_ident_file_mappings line + @tokens = split(/ /, $payload); + + # append empty error + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Delete pg_hba.conf from the given node, add various entries to test the +# include infrastructure and then execute a reload to refresh it. +sub generate_valid_auth_files +{ + my $node = shift; + my $hba_expected = ''; + my $ident_expected = ''; + + # customise main auth file names + $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'"); + $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'"); + + # and make original ones invalid to be sure they're not used anywhere + $node->append_conf('pg_hba.conf', "some invalid line"); + $node->append_conf('pg_ident.conf', "some invalid line"); + + # pg_hba stuff + mkdir("$data_dir/subdir1"); + mkdir("$data_dir/hba_inc"); + mkdir("$data_dir/hba_inc_if"); + mkdir("$data_dir/hba_pos"); + + # Make sure we will still be able to connect + $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust'); + + # Add include data + add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf"); + $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject"); + + $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject"); + + add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject"); + # include is relative to current path + add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject"); + + # include_if_exists data + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none"); + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some"); + $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject"); + + # include_dir data + add_hba_line($node, "$hba_file", "include_dir ../hba_inc"); + add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included"); + $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject"); + $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject"); + + # secondary auth file + add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject'); + $node->append_conf('dbnames.conf', "db1"); + $node->append_conf('dbnames.conf', "db3"); + $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1) + . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1) + . '|local|{db1,db3}|{all}|||reject||'; + + # pg_ident stuff + mkdir("$data_dir/subdir2"); + mkdir("$data_dir/ident_inc"); + mkdir("$data_dir/ident_inc_if"); + mkdir("$data_dir/ident_pos"); + + # Add include data + add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf"); + $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar"); + + $ident_expected .= add_ident_line($node, "$ident_file", "test a b"); + + add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar"); + # include is relative to current path + add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar"); + + # include_if_exists data + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none"); + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some"); + $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar"); + + # include_dir data + add_ident_line($node, "$ident_file", "include_dir ../ident_inc"); + add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included"); + $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar"); + $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar"); + + $node->restart; + $node->connect_ok('dbname=postgres', + 'Connection ok after generating valid auth files'); + + return ($hba_expected, $ident_expected); +} + +# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal +# entries to allow authentication. +sub reset_auth_files +{ + my $node = shift; + + unlink("$data_dir/$hba_file"); + unlink("$data_dir/$ident_file"); + + %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + + return add_hba_line($node, "$hba_file", 'local all all trust'); +} + +# Generate a list of expected error regex for the given array of error +# conditions, as generated by add_hba_line/add_ident_line with an err_str. +# +# 2 regex are generated per array entry: one for the given err_str, and one for +# the expected line in the specific file. Since all lines are independant, +# there's no guarantee that a specific failure regex and the per-line regex +# will match the same error. Calling code should add at least one test with a +# single error to make sure that the line number / file name is correct. +# +# On top of that, an extra line is generated for the general failure to process +# the main auth file. +sub generate_log_err_patterns +{ + my $node = shift; + my $raw_errors = shift; + my $is_hba_err = shift; + my @errors; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + push @errors, qr/$err_str/; + + # Context messages with the file / line location aren't always emitted + if ($err_str !~ /maximum nesting depth exceeded/ and + $err_str !~ /could not open secondary authentication file/) + { + push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/ + } + } + + push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err); + + return \@errors; +} + +# Generate the expected output for the auth file view error reporting (file +# name, file line, error), for the given array of error conditions, as +# generated generated by add_hba_line/add_ident_line with an err_str. +sub generate_log_err_rows +{ + my $node = shift; + my $raw_errors = shift; + my $exp_rows = ''; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + $exp_rows .= "\n" if ($exp_rows ne ""); + + # Unescape regex patterns if any + $err_str =~ s/\\([\(\)])/$1/g; + $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str" + } + + return $exp_rows; +} + +# Reset the main auth files, append the given payload to the given config file, +# and check that the instance cannot start, raising the expected error line(s). +sub start_errors_like +{ + my $node = shift; + my $file = shift; + my $payload = shift; + my $pattern = shift; + my $should_fail = shift; + + reset_auth_files($node); + $node->append_conf($file, $payload); + + unlink($node->logfile); + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir, + '-l', $node->logfile, 'start'); + + if ($should_fail) + { + ok($ret != 0, "Cannot start postgres with faulty $file"); + } + else + { + ok($ret == 0, "postgres can start with faulty $file"); + } + + my $log_contents = slurp_file($node->logfile); + + foreach (@{$pattern}) + { + like($log_contents, + $_, + "Expected failure found in the logs"); + } + + if (not $should_fail) + { + # We can't simply call $node->stop here as the call is optimized out + # when the server isn't started with $node->start. + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', + $data_dir, 'stop', '-m', 'fast'); + ok($ret == 0, "Could stop postgres"); + } +} + +# We should be able to connect, and see an empty pg_ident.conf +is($node->psql( + 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'), + qq(0), + 'pg_ident.conf is empty'); + +############################################ +# part 1, test view reporting for valid data +############################################ +my ($exp_hba, $exp_ident) = generate_valid_auth_files($node); + +$node->connect_ok('dbname=postgres', 'Connection still ok'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_hba_file_rules'), + qq($exp_hba), + 'pg_hba_file_rules content is expected'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_ident_file_mappings'), + qq($exp_ident), + 'pg_ident_file_mappings content is expected'); + +############################################# +# part 2, test log reporting for invalid data +############################################# +reset_auth_files($node); +$node->restart('fast'); +$node->connect_ok('dbname=postgres', + 'Connection ok after resetting auth files'); + +$node->stop('fast'); + +start_errors_like($node, $hba_file, "include ../not_a_file", + [ + qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file +mkdir("$data_dir/hba_inc_fail"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token"); +start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail", + [ + qr/invalid connection type "not_a_token"/, + qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file with nested inclusion +unlink("$data_dir/hba_inc_fail/inc_dir.conf"); +my @hba_raw_errors_step1; + +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1"); + +add_hba_line($node, "hba_inc_fail/file1", "include file2"); +add_hba_line($node, "hba_inc_fail/file2", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file2", "include file3"); + +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3", + "local all all zuul", + 'invalid authentication method "zuul"'); + +start_errors_like( + $node, $hba_file, "include_dir ../hba_inc_fail", + generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @hba_raw_errors_step2; + +# include_if_exists, with various problems +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local", + "end-of-line before database specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local,host", + "multiple values specified for connection type"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all", + "end-of-line before role specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all all", + "end-of-line before authentication method"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "host all all test/42", + 'specifying both host name and CIDR mask is invalid: "test/42"'); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + 'local @dbnames_fails.conf all reject', + "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory"); + +add_hba_line($node, "hba_if_exists.conf", "include recurse.conf"); +push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf", + "include recurse.conf", + 'could not open configuration file "recurse.conf": maximum nesting depth exceeded'); + +# Generate the regex for the expected errors in the logs. There's no guarantee +# that the generated "line X of file..." will be emitted for the expected line, +# but previous tests already ensured that the correct line number / file name +# was emitted, so ensuring that there's an error in all expected lines is +# enough here. +my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2, + 1); + +# Not an error, but it should raise a message in the logs. Manually add an +# extra log message to detect +add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none"); +push @{$expected_errors}, + qr/skipping missing authentication file "$data_dir\/if_exists_none"/; + +start_errors_like( + $node, $hba_file, "include_if_exists ../hba_if_exists.conf", + $expected_errors, 1); + +# Mostly the same, but for ident files +reset_auth_files($node); + +my @ident_raw_errors_step1; + +# include_dir, single included file with nested inclusion +mkdir("$data_dir/ident_inc_fail"); +add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1"); + +add_ident_line($node, "ident_inc_fail/file1", "include file2"); +add_ident_line($node, "ident_inc_fail/file2", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file2", "include file3"); + +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3", + "failmap /(fail postgres", + 'invalid regular expression "\(fail": parentheses \(\) not balanced'); + +start_errors_like( + $node, $ident_file, "include_dir ../ident_inc_fail", + generate_log_err_patterns($node, \@ident_raw_errors_step1, 0), + 0); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @ident_raw_errors_step2; + +# include_if_exists, with various problems +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map", + "missing entry at end of line"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2", + "multiple values in ident field"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", + 'map @osnames_fails.conf postgres', + "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory"); + +add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf", + 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded'); + +start_errors_like( + $node, $ident_file, "include_if_exists ../ident_if_exists.conf", + # There's no guarantee that the generated "line X of file..." will be + # emitted for the expected line, but previous tests already ensured that + # the correct line number / file name was emitted, so ensuring that there's + # an error in all expected lines is enough here. + generate_log_err_patterns($node, \@ident_raw_errors_step2, 0), + 0); + +##################################################### +# part 3, test reporting of various error scenario +# NOTE: this will be bypassed -DEXEC_BACKEND or win32 +##################################################### +reset_auth_files($node); + +$node->start; +$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset'); + +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_hba_file_rules'); + +add_ident_line($node, $ident_file, ''); +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_ident_file_mappings'); + +# The instance could be restarted and no error is detected. Now check if the +# build is compatible with the view error reporting (EXEC_BACKEND / win32 will +# fail when trying to connect as they always rely on the current auth files +# content) +my @hba_raw_errors; + +push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +my ($stdout, $stderr); +my $cmdret = $node->psql('postgres', 'SELECT 1', + stdout => \$stdout, stderr => \$stderr); + +if ($cmdret != 0) +{ + # Connection failed. Bail out, but make sure to raise a failure if it + # didn't fail for the expected hba file modification. + like($stderr, + qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/, + "Connection failed due to loading an invalid hba file"); + + done_testing(); + diag("Build not compatible with auth file view error reporting, bail out.\n"); + exit; +} + +# Combine errors generated at step 2, in the same order. +$node->append_conf($hba_file, "include_dir ../hba_inc_fail"); +push @hba_raw_errors, @hba_raw_errors_step1; + +$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf"); +push @hba_raw_errors, @hba_raw_errors_step2; + +my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules' + . ' WHERE error IS NOT NULL ORDER BY rule_number'), + qq($hba_expected), + 'Detected all error in hba file'); + +# and do the same for pg_ident +my @ident_raw_errors; + +push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +$node->append_conf($ident_file, "include_dir ../ident_inc_fail"); +push @ident_raw_errors, @ident_raw_errors_step1; + +$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf"); +push @ident_raw_errors, @ident_raw_errors_step2; + +my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings' + . ' WHERE error IS NOT NULL ORDER BY mapping_number'), + qq($ident_expected), + 'Detected all error in ident file'); + +done_testing(); diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 178e536e21..3d8f182674 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname, FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); pg_hba_file_rules| SELECT a.rule_number, + a.file_name, a.line_number, a.type, a.database, @@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error); + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, a.line_number, a.map_name, a.sys_name, a.pg_username, a.error - FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.37.0 --mkp7bqke54pr5bne Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v12-0003-POC-Add-a-pg_hba_matches-function.patch" ^ permalink raw reply [nested|flat] 41+ messages in thread
* [PATCH v14 4/5] Allow file inclusion in pg_hba and pg_ident files. @ 2022-10-25 06:26 Julien Rouhaud <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Julien Rouhaud @ 2022-10-25 06:26 UTC (permalink / raw) pg_hba.conf file now has support for "include", "include_dir" and "include_if_exists" directives, which work similarly to the same directives in the postgresql.conf file. This fixes a possible crash if a secondary file tries to include itself as there's now a nesting depth check in the inclusion code path, same as the postgresql.conf. Many regression tests added to cover both the new directives, but also error detection for the whole pg_hba / pg_ident files. Catversion is bumped. Author: Julien Rouhaud Reviewed-by: FIXME Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya%40jrouhaud --- doc/src/sgml/client-auth.sgml | 86 ++- doc/src/sgml/system-views.sgml | 22 +- src/backend/libpq/hba.c | 374 ++++++++-- src/backend/libpq/pg_hba.conf.sample | 25 +- src/backend/libpq/pg_ident.conf.sample | 15 +- src/backend/utils/adt/hbafuncs.c | 20 +- src/backend/utils/misc/guc-file.l | 229 +++--- src/include/catalog/pg_proc.dat | 12 +- src/include/libpq/hba.h | 3 +- src/include/utils/guc.h | 2 + .../authentication/t/003_file_inclusion.pl | 657 ++++++++++++++++++ src/test/regress/expected/rules.out | 6 +- 12 files changed, 1239 insertions(+), 212 deletions(-) create mode 100644 src/test/authentication/t/003_file_inclusion.pl diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index 32d5d45863..2ae723de66 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -89,8 +89,23 @@ </para> <para> - Each record specifies a connection type, a client IP address range - (if relevant for the connection type), a database name, a user name, + Each record can either be an inclusion directive or an authentication + record. Inclusion directives specify files that can be included, which + contains additional records. The records will be inserted in lieu of the + inclusion records. Those records only contains two fields: the + <literal>include</literal>, <literal>include_if_exists</literal> or + <literal>include_dir</literal> directive and the file or directory to be + included. The file or directory can be a relative of absolute path, and can + be double quoted if needed. For the <literal>include_dir</literal> form, + all files not starting with a <literal>.</literal> and ending with + <literal>.conf</literal> will be included. Multiple files within an include + directory are processed in file name order (according to C locale rules, + i.e., numbers before letters, and uppercase letters before lowercase ones). + </para> + + <para> + Each authentication record specifies a connection type, a client IP address + range (if relevant for the connection type), a database name, a user name, and the authentication method to be used for connections matching these parameters. The first record with a matching connection type, client address, requested database, and user name is used to perform @@ -103,21 +118,57 @@ <para> A record can have several formats: <synopsis> -local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> -hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +include <replaceable>file</replaceable> +include_if_exists <replaceable>file</replaceable> +include_dir <replaceable>directory</replaceable> +local <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>address</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +host <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostssl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnossl <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostgssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> +hostnogssenc <replaceable>database</replaceable> <replaceable>user</replaceable> <replaceable>IP-address</replaceable> <replaceable>IP-mask</replaceable> <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional> </synopsis> The meaning of the fields is as follows: <variablelist> + <varlistentry> + <term><literal>include</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_if_exists</literal></term> + <listitem> + <para> + This line will be replaced with the content of the given file if the + file exists and can be read. Otherwise, a message will be logged to + indicate that the file is skipped. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>include_dir</literal></term> + <listitem> + <para> + This line will be replaced with the content of all the files found in + the directory, if they don't start with a <literal>.</literal> and end + with <literal>.conf</literal>, processed in file name order (according + to C locale rules, i.e., numbers before letters, and uppercase letters + before lowercase ones). + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>local</literal></term> <listitem> @@ -863,8 +914,10 @@ local db1,db2,@demodbs all md5 cluster's data directory. (It is possible to place the map file elsewhere, however; see the <xref linkend="guc-ident-file"/> configuration parameter.) - The ident map file contains lines of the general form: + The ident map file contains lines of two general form: <synopsis> +<replaceable>include</replaceable> <replaceable>file</replaceable> +<replaceable>include_dir</replaceable> <replaceable>directory</replaceable> <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable> </synopsis> Comments, whitespace and line continuations are handled in the same way as in @@ -875,6 +928,11 @@ local db1,db2,@demodbs all md5 database user name. The same <replaceable>map-name</replaceable> can be used repeatedly to specify multiple user-mappings within a single map. </para> + <para> + As for <filename>pg_hba.conf</filename>, the lines in this file can either + be inclusion directives or user name map records, and follow the same + rules. + </para> <para> There is no restriction regarding how many database users a given operating system user can correspond to, nor vice versa. Thus, entries diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 4723f712a7..7d1cec8b7f 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -1003,12 +1003,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this rule + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_hba.conf</filename> + Line number of this rule the given <literal>file_name</literal> </para></entry> </row> @@ -1153,12 +1162,21 @@ </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>file_name</structfield> <type>text</type> + </para> + <para> + Name of the file containing this mapping + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>line_number</structfield> <type>int4</type> </para> <para> - Line number of this rule in <filename>pg_ident.conf</filename> + Line number of this mapping in the given <literal>file_name</literal> </para></entry> </row> diff --git a/src/backend/libpq/hba.c b/src/backend/libpq/hba.c index 56bbe31dfd..74f46ffb04 100644 --- a/src/backend/libpq/hba.c +++ b/src/backend/libpq/hba.c @@ -22,6 +22,7 @@ #include <sys/param.h> #include <sys/socket.h> #include <netdb.h> +#include <sys/stat.h> #include <netinet/in.h> #include <arpa/inet.h> #include <unistd.h> @@ -70,6 +71,12 @@ typedef struct check_network_data #define token_is_keyword(t, k) (!t->quoted && strcmp(t->string, k) == 0) #define token_matches(t, k) (strcmp(t->string, k) == 0) +typedef enum HbaIncludeKind +{ + SecondaryAuthFile, + IncludedAuthFile +} HbaIncludeKind; + /* * pre-parsed content of HBA config file: list of HbaLine structs. * parsed_hba_context is the memory context where it lives. @@ -115,14 +122,26 @@ static const char *const UserAuthName[] = }; +static void tokenize_file_with_context(MemoryContext linecxt, + const char *filename, FILE *file, + List **tok_lines, int depth, + int elevel); static List *tokenize_inc_file(List *tokens, const char *outer_filename, - const char *inc_filename, int elevel, char **err_msg); + const char *inc_filename, int depth, int elevel, + char **err_msg); static bool parse_hba_auth_opt(char *name, char *val, HbaLine *hbaline, int elevel, char **err_msg); static int regcomp_auth_token(AuthToken *token, char *filename, int line_num, char **err_msg, int elevel); static int regexec_auth_token(const char *match, AuthToken *token, size_t nmatch, regmatch_t pmatch[]); +static FILE *open_inc_file(HbaIncludeKind kind, const char *inc_filename, + bool strict, const char *outer_filename, int elevel, + char **err_msg, char **inc_fullname); +static char *process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, + int elevel, MemoryContext linecxt, + List **tok_lines); /* @@ -413,7 +432,7 @@ regexec_auth_token(const char *match, AuthToken *token, size_t nmatch, */ static List * next_field_expand(const char *filename, char **lineptr, - int elevel, char **err_msg) + int depth, int elevel, char **err_msg) { char buf[MAX_TOKEN]; bool trailing_comma; @@ -429,7 +448,7 @@ next_field_expand(const char *filename, char **lineptr, /* Is this referencing a file? */ if (!initial_quote && buf[0] == '@' && buf[1] != '\0') - tokens = tokenize_inc_file(tokens, filename, buf + 1, + tokens = tokenize_inc_file(tokens, filename, buf + 1, depth + 1, elevel, err_msg); else tokens = lappend(tokens, make_auth_token(buf, initial_quote)); @@ -457,6 +476,7 @@ static List * tokenize_inc_file(List *tokens, const char *outer_filename, const char *inc_filename, + int depth, int elevel, char **err_msg) { @@ -466,39 +486,30 @@ tokenize_inc_file(List *tokens, ListCell *inc_line; MemoryContext linecxt; - if (is_absolute_path(inc_filename)) - { - /* absolute path is taken as-is */ - inc_fullname = pstrdup(inc_filename); - } - else + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) { - /* relative path is relative to dir of calling file */ - inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + - strlen(inc_filename) + 1); - strcpy(inc_fullname, outer_filename); - get_parent_directory(inc_fullname); - join_path_components(inc_fullname, inc_fullname, inc_filename); - canonicalize_path(inc_fullname); + *err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", *err_msg))); + return tokens; } - inc_file = AllocateFile(inc_fullname, "r"); - if (inc_file == NULL) - { - int save_errno = errno; + inc_file = open_inc_file(SecondaryAuthFile, inc_filename, true, + outer_filename, elevel, err_msg, &inc_fullname); - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open secondary authentication file \"@%s\" as \"%s\": %m", - inc_filename, inc_fullname))); - *err_msg = psprintf("could not open secondary authentication file \"@%s\" as \"%s\": %s", - inc_filename, inc_fullname, strerror(save_errno)); - pfree(inc_fullname); + if (inc_file == NULL) return tokens; - } /* There is possible recursion here if the file contains @ */ - linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, elevel); + linecxt = tokenize_auth_file(inc_fullname, inc_file, &inc_lines, depth + 1, + elevel); FreeFile(inc_file); pfree(inc_fullname); @@ -536,11 +547,38 @@ tokenize_inc_file(List *tokens, /* * tokenize_auth_file - * Tokenize the given file. + * + * Wrapper around tokenize_file_with_context, creating a dedicated memory + * context. + * + * Return value is this memory context which contains all memory allocated by + * this function (it's a child of caller's context). + */ +MemoryContext +tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, + int depth, int elevel) +{ + MemoryContext linecxt; + linecxt = AllocSetContextCreate(CurrentMemoryContext, + "tokenize_auth_file", + ALLOCSET_SMALL_SIZES); + + *tok_lines = NIL; + + tokenize_file_with_context(linecxt, filename, file, tok_lines, depth, + elevel); + + return linecxt; +} + +/* + * Tokenize the given file. * * The output is a list of TokenizedAuthLine structs; see the struct definition * in libpq/hba.h. * + * linecxt: memory context which must contain all memory allocated by the + * function * filename: the absolute path to the target file * file: the already-opened target file * tok_lines: receives output list @@ -549,30 +587,22 @@ tokenize_inc_file(List *tokens, * Errors are reported by logging messages at ereport level elevel and by * adding TokenizedAuthLine structs containing non-null err_msg fields to the * output list. - * - * Return value is a memory context which contains all memory allocated by - * this function (it's a child of caller's context). */ -MemoryContext -tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, - int elevel) +static void +tokenize_file_with_context(MemoryContext linecxt, const char *filename, + FILE *file, List **tok_lines, int depth, int elevel) { - int line_number = 1; StringInfoData buf; - MemoryContext linecxt; + int line_number = 1; MemoryContext oldcxt; - linecxt = AllocSetContextCreate(CurrentMemoryContext, - "tokenize_auth_file", - ALLOCSET_SMALL_SIZES); oldcxt = MemoryContextSwitchTo(linecxt); initStringInfo(&buf); - *tok_lines = NIL; - while (!feof(file) && !ferror(file)) { + TokenizedAuthLine *tok_line; char *lineptr; List *current_line = NIL; char *err_msg = NULL; @@ -625,7 +655,7 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, { List *current_field; - current_field = next_field_expand(filename, &lineptr, + current_field = next_field_expand(filename, &lineptr, depth, elevel, &err_msg); /* add field to line, unless we are at EOL or comment start */ if (current_field != NIL) @@ -633,30 +663,127 @@ tokenize_auth_file(const char *filename, FILE *file, List **tok_lines, } /* - * Reached EOL; emit line to TokenizedAuthLine list unless it's boring + * Reached EOL; no need to emit line to TokenizedAuthLine list if it's + * boring. */ - if (current_line != NIL || err_msg != NULL) + if (current_line == NIL && err_msg == NULL) + goto next_line; + + /* If the line is valid, check if that's an include directive */ + if (err_msg == NULL && list_length(current_line) == 2) { - TokenizedAuthLine *tok_line; + AuthToken *first, *second; + + first = linitial(linitial_node(List, current_line)); + second = linitial(lsecond_node(List, current_line)); + + if (strcmp(first->string, "include") == 0) + { + char *inc_filename; + + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, true, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } + else if (strcmp(first->string, "include_dir") == 0) + { + char **filenames; + char *dir_name = second->string; + int num_filenames; + StringInfoData err_buf; + + filenames = GetDirConfFiles(dir_name, filename, elevel, + &num_filenames, &err_msg); + + if (!filenames) + { + /* We have the error in err_msg, simply process it */ + goto process_line; + } + + initStringInfo(&err_buf); + for (int i = 0; i < num_filenames; i++) + { + /* + * err_msg is used here as a temp buffer, it will be + * overwritten at the end of the loop with the + * cumulated errors, if any. + */ + err_msg = process_included_authfile(filenames[i], true, + filename, depth + 1, elevel, + linecxt, tok_lines); + + /* Cumulate errors if any. */ + if (err_msg) + { + if (err_buf.len > 0) + appendStringInfoChar(&err_buf, '\n'); + appendStringInfoString(&err_buf, err_msg); + } + } + + /* + * If there were no errors, the line is fully processed, bypass + * the general TokenizedAuthLine processing. + */ + if (err_buf.len == 0) + goto next_line; + + /* Otherwise, process the cumulated errors, if any. */ + err_msg = err_buf.data; + } + else if (strcmp(first->string, "include_if_exists") == 0) + { + char *inc_filename; - tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); - tok_line->fields = current_line; - tok_line->file_name = pstrdup(filename); - tok_line->line_num = line_number; - tok_line->raw_line = pstrdup(buf.data); - tok_line->err_msg = err_msg; - *tok_lines = lappend(*tok_lines, tok_line); + inc_filename = second->string; + + err_msg = process_included_authfile(inc_filename, false, + filename, depth + 1, elevel, linecxt, + tok_lines); + + if (!err_msg) + { + /* + * The line is fully processed, bypass the general + * TokenizedAuthLine processing. + */ + goto next_line; + } + } } +process_line: + /* + * General processing: report the error if any and emit line to the + * TokenizedAuthLine + */ + tok_line = (TokenizedAuthLine *) palloc(sizeof(TokenizedAuthLine)); + tok_line->fields = current_line; + tok_line->file_name = pstrdup(filename); + tok_line->line_num = line_number; + tok_line->raw_line = pstrdup(buf.data); + tok_line->err_msg = err_msg; + *tok_lines = lappend(*tok_lines, tok_line); + +next_line: line_number += continuations + 1; } MemoryContextSwitchTo(oldcxt); - - return linecxt; } - /* * Does user belong to role? * @@ -2355,7 +2482,7 @@ load_hba(void) return false; } - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, LOG); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ @@ -2445,6 +2572,135 @@ load_hba(void) return true; } +/* + * Open the given file for inclusion in an authentication file, whether + * secondary or included. + */ +static FILE * +open_inc_file(HbaIncludeKind kind, const char *inc_filename, bool strict, + const char *outer_filename, int elevel, char **err_msg, + char **inc_fullname) +{ + FILE *inc_file; + + if (is_absolute_path(inc_filename)) + { + /* absolute path is taken as-is */ + *inc_fullname = pstrdup(inc_filename); + } + else + { + /* relative path is relative to dir of calling file */ + *inc_fullname = (char *) palloc(strlen(outer_filename) + 1 + + strlen(inc_filename) + 1); + strcpy(*inc_fullname, outer_filename); + get_parent_directory(*inc_fullname); + join_path_components(*inc_fullname, *inc_fullname, inc_filename); + canonicalize_path(*inc_fullname); + } + + inc_file = AllocateFile(*inc_fullname, "r"); + if (inc_file == NULL) + { + int save_errno = errno; + const char *msglog; + const char *msgview; + + if (strict) + { + switch (kind) + { + case SecondaryAuthFile: + msglog = "could not open secondary authentication file \"@%s\" as \"%s\": %m"; + msgview = "could not open secondary authentication file \"@%s\" as \"%s\": %s"; + break; + case IncludedAuthFile: + msglog = "could not open included authentication file \"%s\" as \"%s\": %m"; + msgview = "could not open included authentication file \"%s\" as \"%s\": %s"; + break; + default: + elog(ERROR, "unknown HbaIncludeKind: %d", kind); + break; + } + + ereport(elevel, + (errcode_for_file_access(), + errmsg(msglog, inc_filename, *inc_fullname))); + *err_msg = psprintf(msgview, inc_filename, *inc_fullname, + strerror(save_errno)); + } + else + { + Assert(kind == IncludedAuthFile); + ereport(LOG, + (errmsg("skipping missing authentication file \"%s\"", + *inc_fullname))); + } + + pfree(*inc_fullname); + *inc_fullname = NULL; + return NULL; + } + + return inc_file; +} + +/* + * Try to open an included file, and tokenize it using the given context. + * Returns NULL if no error happens during tokenization, otherwise the error. + */ +static char * +process_included_authfile(const char *inc_filename, bool strict, + const char *outer_filename, int depth, int elevel, + MemoryContext linecxt, List **tok_lines) +{ + char *inc_fullname; + FILE *inc_file; + char *err_msg = NULL; + + /* + * Reject too-deep include nesting depth. This is just a safety check to + * avoid dumping core due to stack overflow if an include file loops back + * to itself. The maximum nesting depth is pretty arbitrary. + */ + if (depth > 10) + { + err_msg = psprintf("could not open configuration file \"%s\": maximum nesting depth exceeded", + inc_filename); + ereport(elevel, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("%s", err_msg))); + return err_msg; + } + + inc_file = open_inc_file(IncludedAuthFile, inc_filename, strict, + outer_filename, elevel, &err_msg, &inc_fullname); + + if (inc_file == NULL) + { + if (strict) + { + /* open_inc_file should have reported an error. */ + Assert(err_msg != NULL); + return err_msg; + } + else + return NULL; + } + else + { + /* No error message should have been reported. */ + Assert(err_msg == NULL); + } + + tokenize_file_with_context(linecxt, inc_fullname, inc_file, + tok_lines, depth, elevel); + + FreeFile(inc_file); + pfree(inc_fullname); + + return NULL; +} /* * Parse one tokenised line from the ident config file and store the result in @@ -2728,7 +2984,7 @@ load_ident(void) return false; } - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, LOG); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, LOG); FreeFile(file); /* Now parse all the lines */ diff --git a/src/backend/libpq/pg_hba.conf.sample b/src/backend/libpq/pg_hba.conf.sample index 5f3f63eb0c..7433050112 100644 --- a/src/backend/libpq/pg_hba.conf.sample +++ b/src/backend/libpq/pg_hba.conf.sample @@ -9,16 +9,27 @@ # are authenticated, which PostgreSQL user names they can use, which # databases they can access. Records take one of these forms: # -# local DATABASE USER METHOD [OPTIONS] -# host DATABASE USER ADDRESS METHOD [OPTIONS] -# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] -# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] -# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# local DATABASE USER METHOD [OPTIONS] +# host DATABASE USER ADDRESS METHOD [OPTIONS] +# hostssl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS] +# hostgssenc DATABASE USER ADDRESS METHOD [OPTIONS] +# hostnogssenc DATABASE USER ADDRESS METHOD [OPTIONS] # # (The uppercase items must be replaced by actual values.) # -# The first field is the connection type: +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# +# Otherwise the first field is the connection type: # - "local" is a Unix-domain socket # - "host" is a TCP/IP socket (encrypted or not) # - "hostssl" is a TCP/IP socket that is SSL-encrypted diff --git a/src/backend/libpq/pg_ident.conf.sample b/src/backend/libpq/pg_ident.conf.sample index a5870e6448..8e3fa29135 100644 --- a/src/backend/libpq/pg_ident.conf.sample +++ b/src/backend/libpq/pg_ident.conf.sample @@ -7,12 +7,23 @@ # # This file controls PostgreSQL user name mapping. It maps external # user names to their corresponding PostgreSQL user names. Records -# are of the form: +# are one of these forms: # -# MAPNAME SYSTEM-USERNAME PG-USERNAME +# include FILE +# include_if_exists FILE +# include_dir DIRECTORY +# MAPNAME SYSTEM-USERNAME PG-USERNAME # # (The uppercase quantities must be replaced by actual values.) # +# If the first field is "include", "include_if_exists" or "include_dir", it's +# not a mapping record but a directive to include records from respectively +# another file, another file if it exists or all the files in the given +# directory ending in '.conf'. FILE is the file name to include, and +# DIR is the directory name containing the file(s) to include. FILE and +# DIRECTORY can be specified with a relative or absolute path, and can be +# double quoted if they contains spaces. +# # MAPNAME is the (otherwise freely chosen) map name that was used in # pg_hba.conf. SYSTEM-USERNAME is the detected user name of the # client. PG-USERNAME is the requested PostgreSQL user name. The diff --git a/src/backend/utils/adt/hbafuncs.c b/src/backend/utils/adt/hbafuncs.c index 255109ab07..d356804c5a 100644 --- a/src/backend/utils/adt/hbafuncs.c +++ b/src/backend/utils/adt/hbafuncs.c @@ -158,7 +158,7 @@ get_hba_options(HbaLine *hba) } /* Number of columns in pg_hba_file_rules view */ -#define NUM_PG_HBA_FILE_RULES_ATTS 10 +#define NUM_PG_HBA_FILE_RULES_ATTS 11 /* * fill_hba_line @@ -199,6 +199,8 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(rule_number); + /* file_name */ + values[index++] = CStringGetTextDatum(hba->sourcefile); /* line_number */ values[index++] = Int32GetDatum(hba->linenumber); @@ -342,7 +344,7 @@ fill_hba_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_HBA_FILE_RULES_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_HBA_FILE_RULES_ATTS - 4) * sizeof(bool)); } /* error */ @@ -383,7 +385,7 @@ fill_hba_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open configuration file \"%s\": %m", HbaFileName))); - linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, DEBUG3); + linecxt = tokenize_auth_file(HbaFileName, file, &hba_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -442,7 +444,7 @@ pg_hba_file_rules(PG_FUNCTION_ARGS) } /* Number of columns in pg_hba_file_mappings view */ -#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 6 +#define NUM_PG_IDENT_FILE_MAPPINGS_ATTS 7 /* * fill_ident_line: build one row of pg_ident_file_mappings view, add it to @@ -477,6 +479,8 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, nulls[index++] = true; else values[index++] = Int32GetDatum(mapping_number); + /* file_name */ + values[index++] = CStringGetTextDatum(ident->sourcefile); /* line_number */ values[index++] = Int32GetDatum(ident->linenumber); @@ -489,7 +493,7 @@ fill_ident_line(Tuplestorestate *tuple_store, TupleDesc tupdesc, else { /* no parsing result, so set relevant fields to nulls */ - memset(&nulls[2], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 3) * sizeof(bool)); + memset(&nulls[3], true, (NUM_PG_IDENT_FILE_MAPPINGS_ATTS - 4) * sizeof(bool)); } /* error */ @@ -529,7 +533,7 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) errmsg("could not open usermap file \"%s\": %m", IdentFileName))); - linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, DEBUG3); + linecxt = tokenize_auth_file(IdentFileName, file, &ident_lines, 0, DEBUG3); FreeFile(file); /* Now parse all the lines */ @@ -550,8 +554,8 @@ fill_ident_view(Tuplestorestate *tuple_store, TupleDesc tupdesc) if (tok_line->err_msg == NULL) mapping_number++; - fill_ident_line(tuple_store, tupdesc, mapping_number, identline, - tok_line->err_msg); + fill_ident_line(tuple_store, tupdesc, mapping_number, + identline, tok_line->err_msg); } /* Free tokenizer memory */ diff --git a/src/backend/utils/misc/guc-file.l b/src/backend/utils/misc/guc-file.l index 721628c0cf..86b6cc1c8a 100644 --- a/src/backend/utils/misc/guc-file.l +++ b/src/backend/utils/misc/guc-file.l @@ -345,6 +345,110 @@ GUC_flex_fatal(const char *msg) return 0; /* keep compiler quiet */ } +/* + * Returns the list of config files located in a directory, in alphabetical + * order. + * + * We don't check for recursion or too-deep nesting depth here, its up to the + * caller to take care of that. + */ +char ** +GetDirConfFiles(const char *includedir, const char *calling_file, int elevel, + int *num_filenames, char **err_msg) +{ + char *directory; + DIR *d; + struct dirent *de; + char **filenames; + int size_filenames; + + /* + * Reject directory name that is all-blank (including empty), as that + * leads to confusion --- we'd read the containing directory, typically + * resulting in recursive inclusion of the same file(s). + */ + if (strspn(includedir, " \t\r\n") == strlen(includedir)) + { + ereport(elevel, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("empty configuration directory name: \"%s\"", + includedir))); + *err_msg = "empty configuration directory name"; + return NULL; + } + + directory = AbsoluteConfigLocation(includedir, calling_file); + d = AllocateDir(directory); + if (d == NULL) + { + ereport(elevel, + (errcode_for_file_access(), + errmsg("could not open configuration directory \"%s\": %m", + directory))); + *err_msg = psprintf("could not open directory \"%s\"", directory); + filenames = NULL; + goto cleanup; + } + + /* + * Read the directory and put the filenames in an array, so we can sort + * them prior to caller processing the contents. + */ + size_filenames = 32; + filenames = (char **) palloc(size_filenames * sizeof(char *)); + *num_filenames = 0; + + while ((de = ReadDir(d, directory)) != NULL) + { + PGFileType de_type; + char filename[MAXPGPATH]; + + /* + * Only parse files with names ending in ".conf". Explicitly reject + * files starting with ".". This excludes things like "." and "..", + * as well as typical hidden files, backup files, and editor debris. + */ + if (strlen(de->d_name) < 6) + continue; + if (de->d_name[0] == '.') + continue; + if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) + continue; + + join_path_components(filename, directory, de->d_name); + canonicalize_path(filename); + de_type = get_dirent_type(filename, de, true, elevel); + if (de_type == PGFILETYPE_ERROR) + { + *err_msg = psprintf("could not stat file \"%s\"", filename); + pfree(filenames); + filenames = NULL; + goto cleanup; + } + else if (de_type != PGFILETYPE_DIR) + { + /* Add file to array, increasing its size in blocks of 32 */ + if (*num_filenames >= size_filenames) + { + size_filenames += 32; + filenames = (char **) repalloc(filenames, + size_filenames * sizeof(char *)); + } + filenames[*num_filenames] = pstrdup(filename); + (*num_filenames)++; + } + } + + if (*num_filenames > 0) + qsort(filenames, *num_filenames, sizeof(char *), pg_qsort_strcmp); + +cleanup: + if (d) + FreeDir(d); + pfree(directory); + return filenames; +} + /* * Read and parse a single configuration file. This function recurses * to handle "include" directives. @@ -606,127 +710,30 @@ ParseConfigDirectory(const char *includedir, ConfigVariable **head_p, ConfigVariable **tail_p) { - char *directory; - DIR *d; - struct dirent *de; + char *err_msg; char **filenames; int num_filenames; - int size_filenames; - bool status; - - /* - * Reject directory name that is all-blank (including empty), as that - * leads to confusion --- we'd read the containing directory, typically - * resulting in recursive inclusion of the same file(s). - */ - if (strspn(includedir, " \t\r\n") == strlen(includedir)) - { - ereport(elevel, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("empty configuration directory name: \"%s\"", - includedir))); - record_config_file_error("empty configuration directory name", - calling_file, calling_lineno, - head_p, tail_p); - return false; - } - - /* - * We don't check for recursion or too-deep nesting depth here; the - * subsequent calls to ParseConfigFile will take care of that. - */ - - directory = AbsoluteConfigLocation(includedir, calling_file); - d = AllocateDir(directory); - if (d == NULL) - { - ereport(elevel, - (errcode_for_file_access(), - errmsg("could not open configuration directory \"%s\": %m", - directory))); - record_config_file_error(psprintf("could not open directory \"%s\"", - directory), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } - /* - * Read the directory and put the filenames in an array, so we can sort - * them prior to processing the contents. - */ - size_filenames = 32; - filenames = (char **) palloc(size_filenames * sizeof(char *)); - num_filenames = 0; + filenames = GetDirConfFiles(includedir, calling_file, elevel, + &num_filenames, &err_msg); - while ((de = ReadDir(d, directory)) != NULL) + if (!filenames) { - PGFileType de_type; - char filename[MAXPGPATH]; - - /* - * Only parse files with names ending in ".conf". Explicitly reject - * files starting with ".". This excludes things like "." and "..", - * as well as typical hidden files, backup files, and editor debris. - */ - if (strlen(de->d_name) < 6) - continue; - if (de->d_name[0] == '.') - continue; - if (strcmp(de->d_name + strlen(de->d_name) - 5, ".conf") != 0) - continue; - - join_path_components(filename, directory, de->d_name); - canonicalize_path(filename); - de_type = get_dirent_type(filename, de, true, elevel); - if (de_type == PGFILETYPE_ERROR) - { - record_config_file_error(psprintf("could not stat file \"%s\"", - filename), - calling_file, calling_lineno, - head_p, tail_p); - status = false; - goto cleanup; - } - else if (de_type != PGFILETYPE_DIR) - { - /* Add file to array, increasing its size in blocks of 32 */ - if (num_filenames >= size_filenames) - { - size_filenames += 32; - filenames = (char **) repalloc(filenames, - size_filenames * sizeof(char *)); - } - filenames[num_filenames] = pstrdup(filename); - num_filenames++; - } + record_config_file_error(err_msg, calling_file, calling_lineno, head_p, + tail_p); + return false; } - if (num_filenames > 0) + for (int i = 0; i < num_filenames; i++) { - int i; - - qsort(filenames, num_filenames, sizeof(char *), pg_qsort_strcmp); - for (i = 0; i < num_filenames; i++) - { - if (!ParseConfigFile(filenames[i], true, - calling_file, calling_lineno, - depth, elevel, - head_p, tail_p)) - { - status = false; - goto cleanup; - } - } + if (!ParseConfigFile(filenames[i], true, + calling_file, calling_lineno, + depth, elevel, + head_p, tail_p)) + return false; } - status = true; -cleanup: - if (d) - FreeDir(d); - pfree(directory); - return status; + return true; } /* diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 4f5d05d0ce..2ad06c4d3e 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6135,16 +6135,16 @@ { oid => '3401', descr => 'show pg_hba.conf rules', proname => 'pg_hba_file_rules', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,_text,_text,text,text,text,_text,text}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o}', - proargnames => '{rule_number,line_number,type,database,user_name,address,netmask,auth_method,options,error}', + proallargtypes => '{int4,text,int4,text,_text,_text,text,text,text,_text,text}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{rule_number,file_name,line_number,type,database,user_name,address,netmask,auth_method,options,error}', prosrc => 'pg_hba_file_rules' }, { oid => '6250', descr => 'show pg_ident.conf mappings', proname => 'pg_ident_file_mappings', prorows => '1000', proretset => 't', provolatile => 'v', prorettype => 'record', proargtypes => '', - proallargtypes => '{int4,int4,text,text,text,text}', - proargmodes => '{o,o,o,o,o,o}', - proargnames => '{mapping_number,line_number,map_name,sys_name,pg_username,error}', + proallargtypes => '{int4,text,int4,text,text,text,text}', + proargmodes => '{o,o,o,o,o,o,o}', + proargnames => '{mapping_number,file_name,line_number,map_name,sys_name,pg_username,error}', prosrc => 'pg_ident_file_mappings' }, { oid => '1371', descr => 'view system lock information', proname => 'pg_lock_status', prorows => '1000', proretset => 't', diff --git a/src/include/libpq/hba.h b/src/include/libpq/hba.h index bf896ac084..7108cd2dae 100644 --- a/src/include/libpq/hba.h +++ b/src/include/libpq/hba.h @@ -179,6 +179,7 @@ extern HbaLine *parse_hba_line(TokenizedAuthLine *tok_line, int elevel); extern IdentLine *parse_ident_line(TokenizedAuthLine *tok_line, int elevel); extern bool pg_isblank(const char c); extern MemoryContext tokenize_auth_file(const char *filename, FILE *file, - List **tok_lines, int elevel); + List **tok_lines, int depth, + int elevel); #endif /* HBA_H */ diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h index b3aaff9665..59ca39d908 100644 --- a/src/include/utils/guc.h +++ b/src/include/utils/guc.h @@ -144,6 +144,8 @@ typedef struct ConfigVariable struct ConfigVariable *next; } ConfigVariable; +extern char **GetDirConfFiles(const char *includedir, const char *calling_file, + int elevel, int *num_filenames, char **err_msg); extern bool ParseConfigFile(const char *config_file, bool strict, const char *calling_file, int calling_lineno, int depth, int elevel, diff --git a/src/test/authentication/t/003_file_inclusion.pl b/src/test/authentication/t/003_file_inclusion.pl new file mode 100644 index 0000000000..8eae72b8d4 --- /dev/null +++ b/src/test/authentication/t/003_file_inclusion.pl @@ -0,0 +1,657 @@ + +# Copyright (c) 2021-2022, PostgreSQL Global Development Group + +# Set of tests for authentication and pg_hba.conf inclusion. +# This test can only run with Unix-domain sockets. + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; +use Time::HiRes qw(usleep); +use IPC::Run qw(pump finish timer); +use Data::Dumper; + +if (!$use_unix_sockets) +{ + plan skip_all => + "authentication tests cannot run without Unix-domain sockets"; +} + +# stores the current line counter for each file. hba_rule and ident_rule are +# fake file names used for the global rule number for each auth view. +my %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + +my $hba_file = 'subdir1/pg_hba_custom.conf'; +my $ident_file = 'subdir2/pg_ident_custom.conf'; + +# Initialize primary node +my $node = PostgreSQL::Test::Cluster->new('primary'); +$node->init; +$node->start; + +my $data_dir = $node->data_dir; + +# Normalize the data directory for Windows +$data_dir =~ s/\/\.\//\//g; # reduce /./ to / +$data_dir =~ s/\/\//\//g; # reduce // to / +$data_dir =~ s/\/$//; # remove trailing / + + +# Add the given payload to the given relative HBA file of the given node. +# This function maintains the %cur_line metadata, so it has to be called in the +# expected inclusion evaluation order in order to keep it in sync. +# +# If the payload starts with "include" or "ignore", the function doesn't +# increase the general hba rule number. +# +# If an err_str is provided, it returns an arrayref containing the provided +# filename, the current line number in that file and the provided err_str. The +# err_str has to be a valid regex string. +# Otherwise it only returns the line number of the payload in the wanted file. +# This function has to be called in the expected inclusion evaluation order to +# keep the %cur_line information in sync. +sub add_hba_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'hba_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_hba_file_rules line + @tokens = split(/ /, $payload); + $tokens[1] = '{' . $tokens[1] . '}'; # database + $tokens[2] = '{' . $tokens[2] . '}'; # user_name + + # add empty address and netmask betweed user_name and auth_method + splice @tokens, 3, 0, ''; + splice @tokens, 3, 0, ''; + + # append empty options and error + push @tokens, ''; + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Add the given payload to the given relative ident file of the given node. +# Same as add_hba_line but for pg_ident files +sub add_ident_line +{ + my $node = shift; + my $filename = shift; + my $payload = shift; + my $err_str = shift; + my $globline; + my $fileline; + my @tokens; + my $line; + + # Append the payload to the given file + $node->append_conf($filename, $payload); + + # Get the current %cur_line counter for the file + if (not defined $cur_line{$filename}) + { + $cur_line{$filename} = 1; + } + $fileline = $cur_line{$filename}++; + + # Include directive, don't generate an underlying pg_hba_file_rules line + # but make sure we incremented the %cur_line counter. + # Also ignore line beginning with "ignore", for content of files that + # should not being included + if ($payload =~ qr/^(include|ignore)/) + { + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + else + { + return $fileline; + } + } + + # Get (and increment) the global rule number + $globline = $cur_line{'ident_rule'}++; + + # If caller provided an err_str, just returns the needed metadata + if (defined $err_str) + { + return [$filename, $fileline, $err_str]; + } + + # Otherwise, generate the expected pg_ident_file_mappings line + @tokens = split(/ /, $payload); + + # append empty error + push @tokens, ''; + + # generate the expected final line + $line = ""; + $line .= "\n" if ($globline > 1); + $line .= "$globline|$data_dir/$filename|$fileline|"; + $line .= join('|', @tokens); + + return $line; +} + +# Delete pg_hba.conf from the given node, add various entries to test the +# include infrastructure and then execute a reload to refresh it. +sub generate_valid_auth_files +{ + my $node = shift; + my $hba_expected = ''; + my $ident_expected = ''; + + # customise main auth file names + $node->safe_psql('postgres', "ALTER SYSTEM SET hba_file = '$data_dir/$hba_file'"); + $node->safe_psql('postgres', "ALTER SYSTEM SET ident_file = '$data_dir/$ident_file'"); + + # and make original ones invalid to be sure they're not used anywhere + $node->append_conf('pg_hba.conf', "some invalid line"); + $node->append_conf('pg_ident.conf', "some invalid line"); + + # pg_hba stuff + mkdir("$data_dir/subdir1"); + mkdir("$data_dir/hba_inc"); + mkdir("$data_dir/hba_inc_if"); + mkdir("$data_dir/hba_pos"); + + # Make sure we will still be able to connect + $hba_expected .= add_hba_line($node, "$hba_file", 'local all all trust'); + + # Add include data + add_hba_line($node, "$hba_file", "include ../pg_hba_pre.conf"); + $hba_expected .= add_hba_line($node, 'pg_hba_pre.conf', "local pre all reject"); + + $hba_expected .= add_hba_line($node, "$hba_file", "local all all reject"); + + add_hba_line($node, "$hba_file", "include ../hba_pos/pg_hba_pos.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "local pos all reject"); + # include is relative to current path + add_hba_line($node, 'hba_pos/pg_hba_pos.conf', "include pg_hba_pos2.conf"); + $hba_expected .= add_hba_line($node, 'hba_pos/pg_hba_pos2.conf', "local pos2 all reject"); + + # include_if_exists data + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/none"); + add_hba_line($node, "$hba_file", "include_if_exists ../hba_inc_if/some"); + $hba_expected .= add_hba_line($node, 'hba_inc_if/some', "local if_some all reject"); + + # include_dir data + add_hba_line($node, "$hba_file", "include_dir ../hba_inc"); + add_hba_line($node, 'hba_inc/garbageconf', "ignore - should not be included"); + $hba_expected .= add_hba_line($node, 'hba_inc/01_z.conf', "local dir_z all reject"); + $hba_expected .= add_hba_line($node, 'hba_inc/02_a.conf', "local dir_a all reject"); + + # secondary auth file + add_hba_line($node, $hba_file, 'local @../dbnames.conf all reject'); + $node->append_conf('dbnames.conf', "db1"); + $node->append_conf('dbnames.conf', "db3"); + $hba_expected .= "\n" . ($cur_line{'hba_rule'} - 1) + . "|$data_dir/$hba_file|" . ($cur_line{$hba_file} - 1) + . '|local|{db1,db3}|{all}|||reject||'; + + # pg_ident stuff + mkdir("$data_dir/subdir2"); + mkdir("$data_dir/ident_inc"); + mkdir("$data_dir/ident_inc_if"); + mkdir("$data_dir/ident_pos"); + + # Add include data + add_ident_line($node, "$ident_file", "include ../pg_ident_pre.conf"); + $ident_expected .= add_ident_line($node, 'pg_ident_pre.conf', "pre foo bar"); + + $ident_expected .= add_ident_line($node, "$ident_file", "test a b"); + + add_ident_line($node, "$ident_file", "include ../ident_pos/pg_ident_pos.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "pos foo bar"); + # include is relative to current path + add_ident_line($node, 'ident_pos/pg_ident_pos.conf', "include pg_ident_pos2.conf"); + $ident_expected .= add_ident_line($node, 'ident_pos/pg_ident_pos2.conf', "pos2 foo bar"); + + # include_if_exists data + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/none"); + add_ident_line($node, "$ident_file", "include_if_exists ../ident_inc_if/some"); + $ident_expected .= add_ident_line($node, 'ident_inc_if/some', "if_some foo bar"); + + # include_dir data + add_ident_line($node, "$ident_file", "include_dir ../ident_inc"); + add_ident_line($node, 'ident_inc/garbageconf', "ignore - should not be included"); + $ident_expected .= add_ident_line($node, 'ident_inc/01_z.conf', "dir_z foo bar"); + $ident_expected .= add_ident_line($node, 'ident_inc/02_a.conf', "dir_a foo bar"); + + $node->restart; + $node->connect_ok('dbname=postgres', + 'Connection ok after generating valid auth files'); + + return ($hba_expected, $ident_expected); +} + +# Delete pg_hba.conf and pg_ident.conf from the given node and add minimal +# entries to allow authentication. +sub reset_auth_files +{ + my $node = shift; + + unlink("$data_dir/$hba_file"); + unlink("$data_dir/$ident_file"); + + %cur_line = ('hba_rule' => 1, 'ident_rule' => 1); + + return add_hba_line($node, "$hba_file", 'local all all trust'); +} + +# Generate a list of expected error regex for the given array of error +# conditions, as generated by add_hba_line/add_ident_line with an err_str. +# +# 2 regex are generated per array entry: one for the given err_str, and one for +# the expected line in the specific file. Since all lines are independant, +# there's no guarantee that a specific failure regex and the per-line regex +# will match the same error. Calling code should add at least one test with a +# single error to make sure that the line number / file name is correct. +# +# On top of that, an extra line is generated for the general failure to process +# the main auth file. +sub generate_log_err_patterns +{ + my $node = shift; + my $raw_errors = shift; + my $is_hba_err = shift; + my @errors; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + push @errors, qr/$err_str/; + + # Context messages with the file / line location aren't always emitted + if ($err_str !~ /maximum nesting depth exceeded/ and + $err_str !~ /could not open secondary authentication file/) + { + push @errors, qr/line $fileline of configuration file "$data_dir\/$filename"/ + } + } + + push @errors, qr/could not load $data_dir\/$hba_file/ if ($is_hba_err); + + return \@errors; +} + +# Generate the expected output for the auth file view error reporting (file +# name, file line, error), for the given array of error conditions, as +# generated generated by add_hba_line/add_ident_line with an err_str. +sub generate_log_err_rows +{ + my $node = shift; + my $raw_errors = shift; + my $exp_rows = ''; + + foreach my $arr (@{$raw_errors}) + { + my $filename = @{$arr}[0]; + my $fileline = @{$arr}[1]; + my $err_str = @{$arr}[2]; + + $exp_rows .= "\n" if ($exp_rows ne ""); + + # Unescape regex patterns if any + $err_str =~ s/\\([\(\)])/$1/g; + $exp_rows .= "|$data_dir\/$filename|$fileline|$err_str" + } + + return $exp_rows; +} + +# Reset the main auth files, append the given payload to the given config file, +# and check that the instance cannot start, raising the expected error line(s). +sub start_errors_like +{ + my $node = shift; + my $file = shift; + my $payload = shift; + my $pattern = shift; + my $should_fail = shift; + + reset_auth_files($node); + $node->append_conf($file, $payload); + + unlink($node->logfile); + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', $data_dir, + '-l', $node->logfile, 'start'); + + if ($should_fail) + { + ok($ret != 0, "Cannot start postgres with faulty $file"); + } + else + { + ok($ret == 0, "postgres can start with faulty $file"); + } + + my $log_contents = slurp_file($node->logfile); + + foreach (@{$pattern}) + { + like($log_contents, + $_, + "Expected failure found in the logs"); + } + + if (not $should_fail) + { + # We can't simply call $node->stop here as the call is optimized out + # when the server isn't started with $node->start. + my $ret = + PostgreSQL::Test::Utils::system_log('pg_ctl', '-D', + $data_dir, 'stop', '-m', 'fast'); + ok($ret == 0, "Could stop postgres"); + } +} + +# We should be able to connect, and see an empty pg_ident.conf +is($node->psql( + 'postgres', 'SELECT count(*) FROM pg_ident_file_mappings'), + qq(0), + 'pg_ident.conf is empty'); + +############################################ +# part 1, test view reporting for valid data +############################################ +my ($exp_hba, $exp_ident) = generate_valid_auth_files($node); + +$node->connect_ok('dbname=postgres', 'Connection still ok'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_hba_file_rules'), + qq($exp_hba), + 'pg_hba_file_rules content is expected'); + +is($node->safe_psql( + 'postgres', 'SELECT * FROM pg_ident_file_mappings'), + qq($exp_ident), + 'pg_ident_file_mappings content is expected'); + +############################################# +# part 2, test log reporting for invalid data +############################################# +reset_auth_files($node); +$node->restart('fast'); +$node->connect_ok('dbname=postgres', + 'Connection ok after resetting auth files'); + +$node->stop('fast'); + +start_errors_like($node, $hba_file, "include ../not_a_file", + [ + qr/could not open included authentication file "\.\.\/not_a_file" as "$data_dir\/not_a_file": No such file or directory/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file +mkdir("$data_dir/hba_inc_fail"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "local all all reject"); +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "not_a_token"); +start_errors_like($node, $hba_file, "include_dir ../hba_inc_fail", + [ + qr/invalid connection type "not_a_token"/, + qr/line 4 of configuration file "$data_dir\/hba_inc_fail\/inc_dir\.conf"/, + qr/could not load $data_dir\/$hba_file/ + ], 1); + +# include_dir, single included file with nested inclusion +unlink("$data_dir/hba_inc_fail/inc_dir.conf"); +my @hba_raw_errors_step1; + +add_hba_line($node, "hba_inc_fail/inc_dir.conf", "include file1"); + +add_hba_line($node, "hba_inc_fail/file1", "include file2"); +add_hba_line($node, "hba_inc_fail/file2", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file2", "include file3"); + +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +add_hba_line($node, "hba_inc_fail/file3", "local all all reject"); +push @hba_raw_errors_step1, add_hba_line($node, "hba_inc_fail/file3", + "local all all zuul", + 'invalid authentication method "zuul"'); + +start_errors_like( + $node, $hba_file, "include_dir ../hba_inc_fail", + generate_log_err_patterns($node, \@hba_raw_errors_step1, 1), 1); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @hba_raw_errors_step2; + +# include_if_exists, with various problems +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local", + "end-of-line before database specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local,host", + "multiple values specified for connection type"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all", + "end-of-line before role specification"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "local all all", + "end-of-line before authentication method"); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + "host all all test/42", + 'specifying both host name and CIDR mask is invalid: "test/42"'); +push @hba_raw_errors_step2, add_hba_line($node, "hba_if_exists.conf", + 'local @dbnames_fails.conf all reject', + "could not open secondary authentication file \"\@dbnames_fails.conf\" as \"$data_dir/dbnames_fails.conf\": No such file or directory"); + +add_hba_line($node, "hba_if_exists.conf", "include recurse.conf"); +push @hba_raw_errors_step2, add_hba_line($node, "recurse.conf", + "include recurse.conf", + 'could not open configuration file "recurse.conf": maximum nesting depth exceeded'); + +# Generate the regex for the expected errors in the logs. There's no guarantee +# that the generated "line X of file..." will be emitted for the expected line, +# but previous tests already ensured that the correct line number / file name +# was emitted, so ensuring that there's an error in all expected lines is +# enough here. +my $expected_errors = generate_log_err_patterns($node, \@hba_raw_errors_step2, + 1); + +# Not an error, but it should raise a message in the logs. Manually add an +# extra log message to detect +add_hba_line($node, "hba_if_exists.conf", "include_if_exists if_exists_none"); +push @{$expected_errors}, + qr/skipping missing authentication file "$data_dir\/if_exists_none"/; + +start_errors_like( + $node, $hba_file, "include_if_exists ../hba_if_exists.conf", + $expected_errors, 1); + +# Mostly the same, but for ident files +reset_auth_files($node); + +my @ident_raw_errors_step1; + +# include_dir, single included file with nested inclusion +mkdir("$data_dir/ident_inc_fail"); +add_ident_line($node, "ident_inc_fail/inc_dir.conf", "include file1"); + +add_ident_line($node, "ident_inc_fail/file1", "include file2"); +add_ident_line($node, "ident_inc_fail/file2", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file2", "include file3"); + +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +add_ident_line($node, "ident_inc_fail/file3", "ok ok ok"); +push @ident_raw_errors_step1, add_ident_line($node, "ident_inc_fail/file3", + "failmap /(fail postgres", + 'invalid regular expression "\(fail": parentheses \(\) not balanced'); + +start_errors_like( + $node, $ident_file, "include_dir ../ident_inc_fail", + generate_log_err_patterns($node, \@ident_raw_errors_step1, 0), + 0); + +# start_errors_like will reset the main auth files, so the previous error won't +# occur again. We keep it around as we will put back both bogus inclusions for +# the tests at step 3. +my @ident_raw_errors_step2; + +# include_if_exists, with various problems +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map", + "missing entry at end of line"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", "map1,map2", + "multiple values in ident field"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_if_exists.conf", + 'map @osnames_fails.conf postgres', + "could not open secondary authentication file \"\@osnames_fails.conf\" as \"$data_dir/osnames_fails.conf\": No such file or directory"); + +add_ident_line($node, "ident_if_exists.conf", "include ident_recurse.conf"); +push @ident_raw_errors_step2, add_ident_line($node, "ident_recurse.conf", "include ident_recurse.conf", + 'could not open configuration file "ident_recurse.conf": maximum nesting depth exceeded'); + +start_errors_like( + $node, $ident_file, "include_if_exists ../ident_if_exists.conf", + # There's no guarantee that the generated "line X of file..." will be + # emitted for the expected line, but previous tests already ensured that + # the correct line number / file name was emitted, so ensuring that there's + # an error in all expected lines is enough here. + generate_log_err_patterns($node, \@ident_raw_errors_step2, 0), + 0); + +##################################################### +# part 3, test reporting of various error scenario +# NOTE: this will be bypassed -DEXEC_BACKEND or win32 +##################################################### +reset_auth_files($node); + +$node->start; +$node->connect_ok('dbname=postgres', 'Can connect after an auth file reset'); + +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_hba_file_rules WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_hba_file_rules'); + +add_ident_line($node, $ident_file, ''); +is($node->safe_psql( + 'postgres', + 'SELECT count(*) FROM pg_ident_file_mappings WHERE error IS NOT NULL'), + qq(0), + 'No error expected in pg_ident_file_mappings'); + +# The instance could be restarted and no error is detected. Now check if the +# build is compatible with the view error reporting (EXEC_BACKEND / win32 will +# fail when trying to connect as they always rely on the current auth files +# content) +my @hba_raw_errors; + +push @hba_raw_errors, add_hba_line($node, $hba_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +my ($stdout, $stderr); +my $cmdret = $node->psql('postgres', 'SELECT 1', + stdout => \$stdout, stderr => \$stderr); + +if ($cmdret != 0) +{ + # Connection failed. Bail out, but make sure to raise a failure if it + # didn't fail for the expected hba file modification. + like($stderr, + qr/connection to server.* failed: FATAL: could not load $data_dir\/$hba_file/, + "Connection failed due to loading an invalid hba file"); + + done_testing(); + diag("Build not compatible with auth file view error reporting, bail out.\n"); + exit; +} + +# Combine errors generated at step 2, in the same order. +$node->append_conf($hba_file, "include_dir ../hba_inc_fail"); +push @hba_raw_errors, @hba_raw_errors_step1; + +$node->append_conf($hba_file, "include_if_exists ../hba_if_exists.conf"); +push @hba_raw_errors, @hba_raw_errors_step2; + +my $hba_expected = generate_log_err_rows($node, \@hba_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT rule_number, file_name, line_number, error FROM pg_hba_file_rules' + . ' WHERE error IS NOT NULL ORDER BY rule_number'), + qq($hba_expected), + 'Detected all error in hba file'); + +# and do the same for pg_ident +my @ident_raw_errors; + +push @ident_raw_errors, add_ident_line($node, $ident_file, "include ../not_a_file", + "could not open included authentication file \"../not_a_file\" as \"$data_dir/not_a_file\": No such file or directory"); + +$node->append_conf($ident_file, "include_dir ../ident_inc_fail"); +push @ident_raw_errors, @ident_raw_errors_step1; + +$node->append_conf($ident_file, "include_if_exists ../ident_if_exists.conf"); +push @ident_raw_errors, @ident_raw_errors_step2; + +my $ident_expected = generate_log_err_rows($node, \@ident_raw_errors); +is($node->safe_psql( + 'postgres', + 'SELECT mapping_number, file_name, line_number, error FROM pg_ident_file_mappings' + . ' WHERE error IS NOT NULL ORDER BY mapping_number'), + qq($ident_expected), + 'Detected all error in ident file'); + +done_testing(); diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 178e536e21..3d8f182674 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1338,6 +1338,7 @@ pg_group| SELECT pg_authid.rolname AS groname, FROM pg_authid WHERE (NOT pg_authid.rolcanlogin); pg_hba_file_rules| SELECT a.rule_number, + a.file_name, a.line_number, a.type, a.database, @@ -1347,14 +1348,15 @@ pg_hba_file_rules| SELECT a.rule_number, a.auth_method, a.options, a.error - FROM pg_hba_file_rules() a(rule_number, line_number, type, database, user_name, address, netmask, auth_method, options, error); + FROM pg_hba_file_rules() a(rule_number, file_name, line_number, type, database, user_name, address, netmask, auth_method, options, error); pg_ident_file_mappings| SELECT a.mapping_number, + a.file_name, a.line_number, a.map_name, a.sys_name, a.pg_username, a.error - FROM pg_ident_file_mappings() a(mapping_number, line_number, map_name, sys_name, pg_username, error); + FROM pg_ident_file_mappings() a(mapping_number, file_name, line_number, map_name, sys_name, pg_username, error); pg_indexes| SELECT n.nspname AS schemaname, c.relname AS tablename, i.relname AS indexname, -- 2.37.0 --4aj2zfv7etn2tv3k-- ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations @ 2025-04-25 12:45 Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Sutou Kouhei @ 2025-04-25 12:45 UTC (permalink / raw) To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers Hi, I've updated the patch set. See the attached v40 patch set. In <CAD21AoAXzwPC7jjPMTcT80hnzmPa2SUJkiqdYHweEY8sZscEMA@mail.gmail.com> "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 23 Apr 2025 23:44:55 -0700, Masahiko Sawada <[email protected]> wrote: >> Are the followings correct? >> >> 1. Move invalid input patterns in >> src/test/modules/test_copy_format/sql/invalid.sql to >> src/test/regress/sql/copy.sql as much as possible. >> 2. Create >> src/test/modules/test_copy_format/sql/test_copy_format.sql >> and move all contents in existing *.sql to the file. >> 3. Add comments what the tests expect to >> src/test/modules/test_copy_format/sql/test_copy_format.sql. >> 4. Remove CopyFormatOptions::{binary,csv_mode}. > > Agreed with the above items. Done except 1. because 1. is removed by 3. in the following list: ---- >> There are 3 unconfirmed suggested changes for tests in: >> https://www.postgresql.org/message-id/20250330.113126.433742864258096312.kou%40clear-code.com >> >> Here are my opinions for them: >> >> > 1.: There is no difference between single-quoting and >> > double-quoting here. Because the information what quote >> > was used for the given FORMAT value isn't remained >> > here. Should we update gram.y? >> > >> > 2.: I don't have a strong opinion for it. If nobody objects >> > it, I'll remove them. >> > >> > 3.: I don't have a strong opinion for it. If nobody objects >> > it, I'll remove them. ---- 0005 is added for 4. Could you squash 0004 ("Use copy handler for bult-in formats") and 0005 ("Remove CopyFormatOptions::{binary,csv_mode}") if needed when you push? >> 6. Use handler OID for detecting the default built-in format >> instead of comparing the given format as string. Done. >> 7. Update documentation. Could someone help this? 0007 is the draft commit for this. >> There are 3 unconfirmed suggested changes for tests in: >> https://www.postgresql.org/message-id/20250330.113126.433742864258096312.kou%40clear-code.com >> >> Here are my opinions for them: >> >> > 1.: There is no difference between single-quoting and >> > double-quoting here. Because the information what quote >> > was used for the given FORMAT value isn't remained >> > here. Should we update gram.y? >> > >> > 2.: I don't have a strong opinion for it. If nobody objects >> > it, I'll remove them. >> > >> > 3.: I don't have a strong opinion for it. If nobody objects >> > it, I'll remove them. >> >> Is the 1. required for "ready for merge"? If so, is there >> any suggestion? I don't have a strong opinion for it. >> >> If there are no more opinions for 2. and 3., I'll remove >> them. > > Agreed. 1.: I didn't do anything. Because there is no suggestion. 2., 3.: Done. > I think we would still need some rounds of reviews but the patch is > getting in good shape. I hope that this is completed in this year... Thanks, -- kou Attachments: [text/x-patch] v40-0001-Export-CopyDest-as-private-data.patch (8.6K, ../../[email protected]/2-v40-0001-Export-CopyDest-as-private-data.patch) download | inline diff: From a81eb07a4c92b8b34ed6fbe6610c54bb9b3bb2e4 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei <[email protected]> Date: Mon, 25 Nov 2024 13:58:33 +0900 Subject: [PATCH v40 1/6] Export CopyDest as private data This is a preparation to export CopyToStateData as private data. CopyToStateData depends on CopyDest. So we need to export CopyDest too. But CopyDest and CopySource has the same names. So we can't export CopyDest as-is. This uses the COPY_DEST_ prefix for CopyDest enum values. CopySource uses the COPY_FROM_ prefix for consistency. --- src/backend/commands/copyfrom.c | 4 ++-- src/backend/commands/copyfromparse.c | 10 ++++---- src/backend/commands/copyto.c | 30 ++++++++---------------- src/include/commands/copyfrom_internal.h | 8 +++---- src/include/commands/copyto_internal.h | 28 ++++++++++++++++++++++ 5 files changed, 49 insertions(+), 31 deletions(-) create mode 100644 src/include/commands/copyto_internal.h diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index fbbbc09a97b..b4dad744547 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -1709,7 +1709,7 @@ BeginCopyFrom(ParseState *pstate, pg_encoding_to_char(GetDatabaseEncoding())))); } - cstate->copy_src = COPY_FILE; /* default */ + cstate->copy_src = COPY_SOURCE_FILE; /* default */ cstate->whereClause = whereClause; @@ -1837,7 +1837,7 @@ BeginCopyFrom(ParseState *pstate, if (data_source_cb) { progress_vals[1] = PROGRESS_COPY_TYPE_CALLBACK; - cstate->copy_src = COPY_CALLBACK; + cstate->copy_src = COPY_SOURCE_CALLBACK; cstate->data_source_cb = data_source_cb; } else if (pipe) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index f5fc346e201..9f7171d1478 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -180,7 +180,7 @@ ReceiveCopyBegin(CopyFromState cstate) for (i = 0; i < natts; i++) pq_sendint16(&buf, format); /* per-column formats */ pq_endmessage(&buf); - cstate->copy_src = COPY_FRONTEND; + cstate->copy_src = COPY_SOURCE_FRONTEND; cstate->fe_msgbuf = makeStringInfo(); /* We *must* flush here to ensure FE knows it can send. */ pq_flush(); @@ -248,7 +248,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) switch (cstate->copy_src) { - case COPY_FILE: + case COPY_SOURCE_FILE: bytesread = fread(databuf, 1, maxread, cstate->copy_file); if (ferror(cstate->copy_file)) ereport(ERROR, @@ -257,7 +257,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) if (bytesread == 0) cstate->raw_reached_eof = true; break; - case COPY_FRONTEND: + case COPY_SOURCE_FRONTEND: while (maxread > 0 && bytesread < minread && !cstate->raw_reached_eof) { int avail; @@ -340,7 +340,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) bytesread += avail; } break; - case COPY_CALLBACK: + case COPY_SOURCE_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); break; } @@ -1172,7 +1172,7 @@ CopyReadLine(CopyFromState cstate, bool is_csv) * after \. up to the protocol end of copy data. (XXX maybe better * not to treat \. as special?) */ - if (cstate->copy_src == COPY_FRONTEND) + if (cstate->copy_src == COPY_SOURCE_FRONTEND) { int inbytes; diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index f87e405351d..d739826afbc 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -20,6 +20,7 @@ #include "access/tableam.h" #include "commands/copyapi.h" +#include "commands/copyto_internal.h" #include "commands/progress.h" #include "executor/execdesc.h" #include "executor/executor.h" @@ -36,17 +37,6 @@ #include "utils/rel.h" #include "utils/snapmgr.h" -/* - * Represents the different dest cases we need to worry about at - * the bottom level - */ -typedef enum CopyDest -{ - COPY_FILE, /* to file (or a piped program) */ - COPY_FRONTEND, /* to frontend */ - COPY_CALLBACK, /* to callback function */ -} CopyDest; - /* * This struct contains all the state variables used throughout a COPY TO * operation. @@ -69,7 +59,7 @@ typedef struct CopyToStateData /* low-level state data */ CopyDest copy_dest; /* type of copy source/destination */ - FILE *copy_file; /* used if copy_dest == COPY_FILE */ + FILE *copy_file; /* used if copy_dest == COPY_DEST_FILE */ StringInfo fe_msgbuf; /* used for all dests during COPY TO */ int file_encoding; /* file or remote side's character encoding */ @@ -401,7 +391,7 @@ SendCopyBegin(CopyToState cstate) for (i = 0; i < natts; i++) pq_sendint16(&buf, format); /* per-column formats */ pq_endmessage(&buf); - cstate->copy_dest = COPY_FRONTEND; + cstate->copy_dest = COPY_DEST_FRONTEND; } static void @@ -448,7 +438,7 @@ CopySendEndOfRow(CopyToState cstate) switch (cstate->copy_dest) { - case COPY_FILE: + case COPY_DEST_FILE: if (fwrite(fe_msgbuf->data, fe_msgbuf->len, 1, cstate->copy_file) != 1 || ferror(cstate->copy_file)) @@ -482,11 +472,11 @@ CopySendEndOfRow(CopyToState cstate) errmsg("could not write to COPY file: %m"))); } break; - case COPY_FRONTEND: + case COPY_DEST_FRONTEND: /* Dump the accumulated row as one CopyData message */ (void) pq_putmessage(PqMsg_CopyData, fe_msgbuf->data, fe_msgbuf->len); break; - case COPY_CALLBACK: + case COPY_DEST_CALLBACK: cstate->data_dest_cb(fe_msgbuf->data, fe_msgbuf->len); break; } @@ -507,7 +497,7 @@ CopySendTextLikeEndOfRow(CopyToState cstate) { switch (cstate->copy_dest) { - case COPY_FILE: + case COPY_DEST_FILE: /* Default line termination depends on platform */ #ifndef WIN32 CopySendChar(cstate, '\n'); @@ -515,7 +505,7 @@ CopySendTextLikeEndOfRow(CopyToState cstate) CopySendString(cstate, "\r\n"); #endif break; - case COPY_FRONTEND: + case COPY_DEST_FRONTEND: /* The FE/BE protocol uses \n as newline for all platforms */ CopySendChar(cstate, '\n'); break; @@ -903,12 +893,12 @@ BeginCopyTo(ParseState *pstate, /* See Multibyte encoding comment above */ cstate->encoding_embeds_ascii = PG_ENCODING_IS_CLIENT_ONLY(cstate->file_encoding); - cstate->copy_dest = COPY_FILE; /* default */ + cstate->copy_dest = COPY_DEST_FILE; /* default */ if (data_dest_cb) { progress_vals[1] = PROGRESS_COPY_TYPE_CALLBACK; - cstate->copy_dest = COPY_CALLBACK; + cstate->copy_dest = COPY_DEST_CALLBACK; cstate->data_dest_cb = data_dest_cb; } else if (pipe) diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index c8b22af22d8..24157e11a73 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -24,9 +24,9 @@ */ typedef enum CopySource { - COPY_FILE, /* from file (or a piped program) */ - COPY_FRONTEND, /* from frontend */ - COPY_CALLBACK, /* from callback function */ + COPY_SOURCE_FILE, /* from file (or a piped program) */ + COPY_SOURCE_FRONTEND, /* from frontend */ + COPY_SOURCE_CALLBACK, /* from callback function */ } CopySource; /* @@ -64,7 +64,7 @@ typedef struct CopyFromStateData /* low-level state data */ CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ - StringInfo fe_msgbuf; /* used if copy_src == COPY_FRONTEND */ + StringInfo fe_msgbuf; /* used if copy_src == COPY_SOURCE_FRONTEND */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ diff --git a/src/include/commands/copyto_internal.h b/src/include/commands/copyto_internal.h new file mode 100644 index 00000000000..42ddb37a8a2 --- /dev/null +++ b/src/include/commands/copyto_internal.h @@ -0,0 +1,28 @@ +/*------------------------------------------------------------------------- + * + * copyto_internal.h + * Internal definitions for COPY TO command. + * + * + * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/commands/copyto_internal.h + * + *------------------------------------------------------------------------- + */ +#ifndef COPYTO_INTERNAL_H +#define COPYTO_INTERNAL_H + +/* + * Represents the different dest cases we need to worry about at + * the bottom level + */ +typedef enum CopyDest +{ + COPY_DEST_FILE, /* to file (or a piped program) */ + COPY_DEST_FRONTEND, /* to frontend */ + COPY_DEST_CALLBACK, /* to callback function */ +} CopyDest; + +#endif /* COPYTO_INTERNAL_H */ -- 2.47.2 [text/x-patch] v40-0002-Add-support-for-adding-custom-COPY-format.patch (33.7K, ../../[email protected]/3-v40-0002-Add-support-for-adding-custom-COPY-format.patch) download | inline diff: From 398994b555e3b508ce26fc33199bf9badbfc82d5 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei <[email protected]> Date: Thu, 27 Mar 2025 11:14:43 +0900 Subject: [PATCH v40 2/6] Add support for adding custom COPY format This uses the handler approach like tablesample. The approach creates an internal function that returns an internal struct. In this case, a handler returns a CopyToRoutine for COPY TO and a CopyFromRoutine for COPY FROM. Whether COPY TO or COPY FROM is passed as the "is_from" argument: copy_handler(true) returns CopyToRoutine copy_handler(false) returns CopyFromRoutine This also add a test module for custom COPY handler. --- src/backend/commands/copy.c | 31 ++++- src/backend/commands/copyfrom.c | 20 +++- src/backend/commands/copyto.c | 72 +++-------- src/backend/nodes/Makefile | 1 + src/backend/nodes/gen_node_support.pl | 2 + src/backend/utils/adt/pseudotypes.c | 1 + src/include/catalog/pg_proc.dat | 6 + src/include/catalog/pg_type.dat | 6 + src/include/commands/copy.h | 3 +- src/include/commands/copyapi.h | 4 + src/include/commands/copyto_internal.h | 55 +++++++++ src/include/nodes/meson.build | 1 + src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + src/test/modules/test_copy_format/.gitignore | 4 + src/test/modules/test_copy_format/Makefile | 23 ++++ .../expected/test_copy_format.out | 107 +++++++++++++++++ src/test/modules/test_copy_format/meson.build | 33 +++++ .../test_copy_format/sql/test_copy_format.sql | 52 ++++++++ .../test_copy_format--1.0.sql | 24 ++++ .../test_copy_format/test_copy_format.c | 113 ++++++++++++++++++ .../test_copy_format/test_copy_format.control | 4 + 22 files changed, 505 insertions(+), 59 deletions(-) mode change 100644 => 100755 src/backend/nodes/gen_node_support.pl create mode 100644 src/test/modules/test_copy_format/.gitignore create mode 100644 src/test/modules/test_copy_format/Makefile create mode 100644 src/test/modules/test_copy_format/expected/test_copy_format.out create mode 100644 src/test/modules/test_copy_format/meson.build create mode 100644 src/test/modules/test_copy_format/sql/test_copy_format.sql create mode 100644 src/test/modules/test_copy_format/test_copy_format--1.0.sql create mode 100644 src/test/modules/test_copy_format/test_copy_format.c create mode 100644 src/test/modules/test_copy_format/test_copy_format.control diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 74ae42b19a7..9515c4d5786 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -32,10 +32,12 @@ #include "parser/parse_coerce.h" #include "parser/parse_collate.h" #include "parser/parse_expr.h" +#include "parser/parse_func.h" #include "parser/parse_relation.h" #include "utils/acl.h" #include "utils/builtins.h" #include "utils/lsyscache.h" +#include "utils/regproc.h" #include "utils/rel.h" #include "utils/rls.h" @@ -531,10 +533,31 @@ ProcessCopyOptions(ParseState *pstate, else if (strcmp(fmt, "binary") == 0) opts_out->binary = true; else - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("COPY format \"%s\" not recognized", fmt), - parser_errposition(pstate, defel->location))); + { + List *qualified_format; + Oid arg_types[1]; + Oid handler = InvalidOid; + + qualified_format = stringToQualifiedNameList(fmt, NULL); + arg_types[0] = INTERNALOID; + handler = LookupFuncName(qualified_format, 1, + arg_types, true); + if (!OidIsValid(handler)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY format \"%s\" not recognized", fmt), + parser_errposition(pstate, defel->location))); + + /* check that handler has correct return type */ + if (get_func_rettype(handler) != COPY_HANDLEROID) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("function %s must return type %s", + fmt, "copy_handler"), + parser_errposition(pstate, defel->location))); + + opts_out->handler = handler; + } } else if (strcmp(defel->defname, "freeze") == 0) { diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index b4dad744547..3d86e8a8328 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -129,6 +129,7 @@ static void CopyFromBinaryEnd(CopyFromState cstate); /* text format */ static const CopyFromRoutine CopyFromRoutineText = { + .type = T_CopyFromRoutine, .CopyFromInFunc = CopyFromTextLikeInFunc, .CopyFromStart = CopyFromTextLikeStart, .CopyFromOneRow = CopyFromTextOneRow, @@ -137,6 +138,7 @@ static const CopyFromRoutine CopyFromRoutineText = { /* CSV format */ static const CopyFromRoutine CopyFromRoutineCSV = { + .type = T_CopyFromRoutine, .CopyFromInFunc = CopyFromTextLikeInFunc, .CopyFromStart = CopyFromTextLikeStart, .CopyFromOneRow = CopyFromCSVOneRow, @@ -145,6 +147,7 @@ static const CopyFromRoutine CopyFromRoutineCSV = { /* binary format */ static const CopyFromRoutine CopyFromRoutineBinary = { + .type = T_CopyFromRoutine, .CopyFromInFunc = CopyFromBinaryInFunc, .CopyFromStart = CopyFromBinaryStart, .CopyFromOneRow = CopyFromBinaryOneRow, @@ -155,7 +158,22 @@ static const CopyFromRoutine CopyFromRoutineBinary = { static const CopyFromRoutine * CopyFromGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (OidIsValid(opts->handler)) + { + Datum datum; + Node *routine; + + datum = OidFunctionCall1(opts->handler, BoolGetDatum(true)); + routine = (Node *) DatumGetPointer(datum); + if (routine == NULL || !IsA(routine, CopyFromRoutine)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY handler function %s.%s did not return CopyFromRoutine struct", + get_namespace_name(get_func_namespace(opts->handler)), + get_func_name(opts->handler)))); + return castNode(CopyFromRoutine, routine); + } + else if (opts->csv_mode) return &CopyFromRoutineCSV; else if (opts->binary) return &CopyFromRoutineBinary; diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index d739826afbc..265b847e255 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -22,9 +22,7 @@ #include "commands/copyapi.h" #include "commands/copyto_internal.h" #include "commands/progress.h" -#include "executor/execdesc.h" #include "executor/executor.h" -#include "executor/tuptable.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -37,56 +35,6 @@ #include "utils/rel.h" #include "utils/snapmgr.h" -/* - * This struct contains all the state variables used throughout a COPY TO - * operation. - * - * Multi-byte encodings: all supported client-side encodings encode multi-byte - * characters by having the first byte's high bit set. Subsequent bytes of the - * character can have the high bit not set. When scanning data in such an - * encoding to look for a match to a single-byte (ie ASCII) character, we must - * use the full pg_encoding_mblen() machinery to skip over multibyte - * characters, else we might find a false match to a trailing byte. In - * supported server encodings, there is no possibility of a false match, and - * it's faster to make useless comparisons to trailing bytes than it is to - * invoke pg_encoding_mblen() to skip over them. encoding_embeds_ascii is true - * when we have to do it the hard way. - */ -typedef struct CopyToStateData -{ - /* format-specific routines */ - const CopyToRoutine *routine; - - /* low-level state data */ - CopyDest copy_dest; /* type of copy source/destination */ - FILE *copy_file; /* used if copy_dest == COPY_DEST_FILE */ - StringInfo fe_msgbuf; /* used for all dests during COPY TO */ - - int file_encoding; /* file or remote side's character encoding */ - bool need_transcoding; /* file encoding diff from server? */ - bool encoding_embeds_ascii; /* ASCII can be non-first byte? */ - - /* parameters from the COPY command */ - Relation rel; /* relation to copy to */ - QueryDesc *queryDesc; /* executable query to copy from */ - List *attnumlist; /* integer list of attnums to copy */ - char *filename; /* filename, or NULL for STDOUT */ - bool is_program; /* is 'filename' a program to popen? */ - copy_data_dest_cb data_dest_cb; /* function for writing data */ - - CopyFormatOptions opts; - Node *whereClause; /* WHERE condition (or NULL) */ - - /* - * Working state - */ - MemoryContext copycontext; /* per-copy execution context */ - - FmgrInfo *out_functions; /* lookup info for output functions */ - MemoryContext rowcontext; /* per-row evaluation context */ - uint64 bytes_processed; /* number of bytes processed so far */ -} CopyToStateData; - /* DestReceiver for COPY (query) TO */ typedef struct { @@ -140,6 +88,7 @@ static void CopySendInt16(CopyToState cstate, int16 val); /* text format */ static const CopyToRoutine CopyToRoutineText = { + .type = T_CopyToRoutine, .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToTextOneRow, @@ -148,6 +97,7 @@ static const CopyToRoutine CopyToRoutineText = { /* CSV format */ static const CopyToRoutine CopyToRoutineCSV = { + .type = T_CopyToRoutine, .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, .CopyToOneRow = CopyToCSVOneRow, @@ -156,6 +106,7 @@ static const CopyToRoutine CopyToRoutineCSV = { /* binary format */ static const CopyToRoutine CopyToRoutineBinary = { + .type = T_CopyToRoutine, .CopyToStart = CopyToBinaryStart, .CopyToOutFunc = CopyToBinaryOutFunc, .CopyToOneRow = CopyToBinaryOneRow, @@ -166,7 +117,22 @@ static const CopyToRoutine CopyToRoutineBinary = { static const CopyToRoutine * CopyToGetRoutine(const CopyFormatOptions *opts) { - if (opts->csv_mode) + if (OidIsValid(opts->handler)) + { + Datum datum; + Node *routine; + + datum = OidFunctionCall1(opts->handler, BoolGetDatum(false)); + routine = (Node *) DatumGetPointer(datum); + if (routine == NULL || !IsA(routine, CopyToRoutine)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY handler function %s.%s did not return CopyToRoutine struct", + get_namespace_name(get_func_namespace(opts->handler)), + get_func_name(opts->handler)))); + return castNode(CopyToRoutine, routine); + } + else if (opts->csv_mode) return &CopyToRoutineCSV; else if (opts->binary) return &CopyToRoutineBinary; diff --git a/src/backend/nodes/Makefile b/src/backend/nodes/Makefile index 77ddb9ca53f..dc6c1087361 100644 --- a/src/backend/nodes/Makefile +++ b/src/backend/nodes/Makefile @@ -50,6 +50,7 @@ node_headers = \ access/sdir.h \ access/tableam.h \ access/tsmapi.h \ + commands/copyapi.h \ commands/event_trigger.h \ commands/trigger.h \ executor/tuptable.h \ diff --git a/src/backend/nodes/gen_node_support.pl b/src/backend/nodes/gen_node_support.pl old mode 100644 new mode 100755 index 77659b0f760..d688bbea3a0 --- a/src/backend/nodes/gen_node_support.pl +++ b/src/backend/nodes/gen_node_support.pl @@ -62,6 +62,7 @@ my @all_input_files = qw( access/sdir.h access/tableam.h access/tsmapi.h + commands/copyapi.h commands/event_trigger.h commands/trigger.h executor/tuptable.h @@ -86,6 +87,7 @@ my @nodetag_only_files = qw( access/sdir.h access/tableam.h access/tsmapi.h + commands/copyapi.h commands/event_trigger.h commands/trigger.h executor/tuptable.h diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c index 317a1f2b282..f2ebc21ca56 100644 --- a/src/backend/utils/adt/pseudotypes.c +++ b/src/backend/utils/adt/pseudotypes.c @@ -370,6 +370,7 @@ PSEUDOTYPE_DUMMY_IO_FUNCS(fdw_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(table_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(index_am_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(tsm_handler); +PSEUDOTYPE_DUMMY_IO_FUNCS(copy_handler); PSEUDOTYPE_DUMMY_IO_FUNCS(internal); PSEUDOTYPE_DUMMY_IO_FUNCS(anyelement); PSEUDOTYPE_DUMMY_IO_FUNCS(anynonarray); diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 62beb71da28..ba46bfa48a8 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7888,6 +7888,12 @@ { oid => '3312', descr => 'I/O', proname => 'tsm_handler_out', prorettype => 'cstring', proargtypes => 'tsm_handler', prosrc => 'tsm_handler_out' }, +{ oid => '8753', descr => 'I/O', + proname => 'copy_handler_in', proisstrict => 'f', prorettype => 'copy_handler', + proargtypes => 'cstring', prosrc => 'copy_handler_in' }, +{ oid => '8754', descr => 'I/O', + proname => 'copy_handler_out', prorettype => 'cstring', + proargtypes => 'copy_handler', prosrc => 'copy_handler_out' }, { oid => '267', descr => 'I/O', proname => 'table_am_handler_in', proisstrict => 'f', prorettype => 'table_am_handler', proargtypes => 'cstring', diff --git a/src/include/catalog/pg_type.dat b/src/include/catalog/pg_type.dat index 6dca77e0a22..bddf9fb4fbe 100644 --- a/src/include/catalog/pg_type.dat +++ b/src/include/catalog/pg_type.dat @@ -633,6 +633,12 @@ typcategory => 'P', typinput => 'tsm_handler_in', typoutput => 'tsm_handler_out', typreceive => '-', typsend => '-', typalign => 'i' }, +{ oid => '8752', + descr => 'pseudo-type for the result of a COPY TO/FROM handler function', + typname => 'copy_handler', typlen => '4', typbyval => 't', typtype => 'p', + typcategory => 'P', typinput => 'copy_handler_in', + typoutput => 'copy_handler_out', typreceive => '-', typsend => '-', + typalign => 'i' }, { oid => '269', descr => 'pseudo-type for the result of a table AM handler function', typname => 'table_am_handler', typlen => '4', typbyval => 't', typtype => 'p', diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 06dfdfef721..6df1f8a3b9b 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -87,9 +87,10 @@ typedef struct CopyFormatOptions CopyLogVerbosityChoice log_verbosity; /* verbosity of logged messages */ int64 reject_limit; /* maximum tolerable number of errors */ List *convert_select; /* list of column names (can be NIL) */ + Oid handler; /* handler function for custom format routine */ } CopyFormatOptions; -/* These are private in commands/copy[from|to].c */ +/* These are private in commands/copy[from|to]_internal.h */ typedef struct CopyFromStateData *CopyFromState; typedef struct CopyToStateData *CopyToState; diff --git a/src/include/commands/copyapi.h b/src/include/commands/copyapi.h index 2a2d2f9876b..53ad3337f86 100644 --- a/src/include/commands/copyapi.h +++ b/src/include/commands/copyapi.h @@ -22,6 +22,8 @@ */ typedef struct CopyToRoutine { + NodeTag type; + /* * Set output function information. This callback is called once at the * beginning of COPY TO. @@ -60,6 +62,8 @@ typedef struct CopyToRoutine */ typedef struct CopyFromRoutine { + NodeTag type; + /* * Set input function information. This callback is called once at the * beginning of COPY FROM. diff --git a/src/include/commands/copyto_internal.h b/src/include/commands/copyto_internal.h index 42ddb37a8a2..da796131988 100644 --- a/src/include/commands/copyto_internal.h +++ b/src/include/commands/copyto_internal.h @@ -14,6 +14,11 @@ #ifndef COPYTO_INTERNAL_H #define COPYTO_INTERNAL_H +#include "commands/copy.h" +#include "executor/execdesc.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" + /* * Represents the different dest cases we need to worry about at * the bottom level @@ -25,4 +30,54 @@ typedef enum CopyDest COPY_DEST_CALLBACK, /* to callback function */ } CopyDest; +/* + * This struct contains all the state variables used throughout a COPY TO + * operation. + * + * Multi-byte encodings: all supported client-side encodings encode multi-byte + * characters by having the first byte's high bit set. Subsequent bytes of the + * character can have the high bit not set. When scanning data in such an + * encoding to look for a match to a single-byte (ie ASCII) character, we must + * use the full pg_encoding_mblen() machinery to skip over multibyte + * characters, else we might find a false match to a trailing byte. In + * supported server encodings, there is no possibility of a false match, and + * it's faster to make useless comparisons to trailing bytes than it is to + * invoke pg_encoding_mblen() to skip over them. encoding_embeds_ascii is true + * when we have to do it the hard way. + */ +typedef struct CopyToStateData +{ + /* format-specific routines */ + const CopyToRoutine *routine; + + /* low-level state data */ + CopyDest copy_dest; /* type of copy source/destination */ + FILE *copy_file; /* used if copy_dest == COPY_DEST_FILE */ + StringInfo fe_msgbuf; /* used for all dests during COPY TO */ + + int file_encoding; /* file or remote side's character encoding */ + bool need_transcoding; /* file encoding diff from server? */ + bool encoding_embeds_ascii; /* ASCII can be non-first byte? */ + + /* parameters from the COPY command */ + Relation rel; /* relation to copy to */ + QueryDesc *queryDesc; /* executable query to copy from */ + List *attnumlist; /* integer list of attnums to copy */ + char *filename; /* filename, or NULL for STDOUT */ + bool is_program; /* is 'filename' a program to popen? */ + copy_data_dest_cb data_dest_cb; /* function for writing data */ + + CopyFormatOptions opts; + Node *whereClause; /* WHERE condition (or NULL) */ + + /* + * Working state + */ + MemoryContext copycontext; /* per-copy execution context */ + + FmgrInfo *out_functions; /* lookup info for output functions */ + MemoryContext rowcontext; /* per-row evaluation context */ + uint64 bytes_processed; /* number of bytes processed so far */ +} CopyToStateData; + #endif /* COPYTO_INTERNAL_H */ diff --git a/src/include/nodes/meson.build b/src/include/nodes/meson.build index d1ca24dd32f..96e70e7f38b 100644 --- a/src/include/nodes/meson.build +++ b/src/include/nodes/meson.build @@ -12,6 +12,7 @@ node_support_input_i = [ 'access/sdir.h', 'access/tableam.h', 'access/tsmapi.h', + 'commands/copyapi.h', 'commands/event_trigger.h', 'commands/trigger.h', 'executor/tuptable.h', diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index aa1d27bbed3..9bf5d58cdae 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -17,6 +17,7 @@ SUBDIRS = \ test_aio \ test_bloomfilter \ test_copy_callbacks \ + test_copy_format \ test_custom_rmgrs \ test_ddl_deparse \ test_dsa \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index 9de0057bd1d..5fd06de2737 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -16,6 +16,7 @@ subdir('ssl_passphrase_callback') subdir('test_aio') subdir('test_bloomfilter') subdir('test_copy_callbacks') +subdir('test_copy_format') subdir('test_custom_rmgrs') subdir('test_ddl_deparse') subdir('test_dsa') diff --git a/src/test/modules/test_copy_format/.gitignore b/src/test/modules/test_copy_format/.gitignore new file mode 100644 index 00000000000..5dcb3ff9723 --- /dev/null +++ b/src/test/modules/test_copy_format/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_copy_format/Makefile b/src/test/modules/test_copy_format/Makefile new file mode 100644 index 00000000000..8497f91624d --- /dev/null +++ b/src/test/modules/test_copy_format/Makefile @@ -0,0 +1,23 @@ +# src/test/modules/test_copy_format/Makefile + +MODULE_big = test_copy_format +OBJS = \ + $(WIN32RES) \ + test_copy_format.o +PGFILEDESC = "test_copy_format - test custom COPY FORMAT" + +EXTENSION = test_copy_format +DATA = test_copy_format--1.0.sql + +REGRESS = test_copy_format + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_copy_format +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_copy_format/expected/test_copy_format.out b/src/test/modules/test_copy_format/expected/test_copy_format.out new file mode 100644 index 00000000000..3916b766615 --- /dev/null +++ b/src/test/modules/test_copy_format/expected/test_copy_format.out @@ -0,0 +1,107 @@ +CREATE TABLE copy_data (a smallint, b integer, c bigint); +INSERT INTO copy_data VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789); +-- No WITH SCHEMA. It installs custom COPY handlers to the current +-- schema. +CREATE EXTENSION test_copy_format; +-- We can find a custom COPY handler without schema. +COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format'); +NOTICE: test_copy_format: is_from=true +NOTICE: CopyFromInFunc: attribute: smallint +NOTICE: CopyFromInFunc: attribute: integer +NOTICE: CopyFromInFunc: attribute: bigint +NOTICE: CopyFromStart: the number of attributes: 3 +NOTICE: CopyFromOneRow +NOTICE: CopyFromEnd +COPY copy_data TO stdout WITH (FORMAT 'test_copy_format'); +NOTICE: test_copy_format: is_from=false +NOTICE: CopyToOutFunc: attribute: smallint +NOTICE: CopyToOutFunc: attribute: integer +NOTICE: CopyToOutFunc: attribute: bigint +NOTICE: CopyToStart: the number of attributes: 3 +NOTICE: CopyToOneRow: the number of valid values: 3 +NOTICE: CopyToOneRow: the number of valid values: 3 +NOTICE: CopyToOneRow: the number of valid values: 3 +NOTICE: CopyToEnd +DROP EXTENSION test_copy_format; +-- Install custom COPY handlers to a schema that isn't included in +-- search_path. +CREATE SCHEMA test_schema; +CREATE EXTENSION test_copy_format WITH SCHEMA test_schema; +-- We can find a custom COPY handler by qualified name. +COPY copy_data FROM stdin WITH (FORMAT 'test_schema.test_copy_format'); +NOTICE: test_copy_format: is_from=true +NOTICE: CopyFromInFunc: attribute: smallint +NOTICE: CopyFromInFunc: attribute: integer +NOTICE: CopyFromInFunc: attribute: bigint +NOTICE: CopyFromStart: the number of attributes: 3 +NOTICE: CopyFromOneRow +NOTICE: CopyFromEnd +COPY copy_data TO stdout WITH (FORMAT 'test_schema.test_copy_format'); +NOTICE: test_copy_format: is_from=false +NOTICE: CopyToOutFunc: attribute: smallint +NOTICE: CopyToOutFunc: attribute: integer +NOTICE: CopyToOutFunc: attribute: bigint +NOTICE: CopyToStart: the number of attributes: 3 +NOTICE: CopyToOneRow: the number of valid values: 3 +NOTICE: CopyToOneRow: the number of valid values: 3 +NOTICE: CopyToOneRow: the number of valid values: 3 +NOTICE: CopyToEnd +-- We can't find a custom COPY handler without schema when search_path +-- doesn't include the schema where we installed custom COPY handlers. +COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format'); +ERROR: COPY format "test_copy_format" not recognized +LINE 1: COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format'); + ^ +COPY copy_data TO stdout WITH (FORMAT 'test_copy_format'); +ERROR: COPY format "test_copy_format" not recognized +LINE 1: COPY copy_data TO stdout WITH (FORMAT 'test_copy_format'); + ^ +-- We can find a custom COPY handler without schema when search_path +-- includes the schema where we installed custom COPY handlers. +SET search_path = test_schema,public; +COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format'); +NOTICE: test_copy_format: is_from=true +NOTICE: CopyFromInFunc: attribute: smallint +NOTICE: CopyFromInFunc: attribute: integer +NOTICE: CopyFromInFunc: attribute: bigint +NOTICE: CopyFromStart: the number of attributes: 3 +NOTICE: CopyFromOneRow +NOTICE: CopyFromEnd +COPY copy_data TO stdout WITH (FORMAT 'test_copy_format'); +NOTICE: test_copy_format: is_from=false +NOTICE: CopyToOutFunc: attribute: smallint +NOTICE: CopyToOutFunc: attribute: integer +NOTICE: CopyToOutFunc: attribute: bigint +NOTICE: CopyToStart: the number of attributes: 3 +NOTICE: CopyToOneRow: the number of valid values: 3 +NOTICE: CopyToOneRow: the number of valid values: 3 +NOTICE: CopyToOneRow: the number of valid values: 3 +NOTICE: CopyToEnd +RESET search_path; +-- Invalid cases with qualified name. +-- Input type is wrong +COPY copy_data FROM stdin WITH (FORMAT 'test_schema.test_copy_format_wrong_input_type'); +ERROR: COPY format "test_schema.test_copy_format_wrong_input_type" not recognized +LINE 1: COPY copy_data FROM stdin WITH (FORMAT 'test_schema.test_cop... + ^ +COPY copy_data TO stdout WITH (FORMAT 'test_schema.test_copy_format_wrong_input_type'); +ERROR: COPY format "test_schema.test_copy_format_wrong_input_type" not recognized +LINE 1: COPY copy_data TO stdout WITH (FORMAT 'test_schema.test_copy... + ^ +-- Return type is wrong +COPY copy_data FROM stdin WITH (FORMAT 'test_schema.test_copy_format_wrong_return_type'); +ERROR: function test_schema.test_copy_format_wrong_return_type must return type copy_handler +LINE 1: COPY copy_data FROM stdin WITH (FORMAT 'test_schema.test_cop... + ^ +COPY copy_data TO stdout WITH (FORMAT 'test_schema.test_copy_format_wrong_return_type'); +ERROR: function test_schema.test_copy_format_wrong_return_type must return type copy_handler +LINE 1: COPY copy_data TO stdout WITH (FORMAT 'test_schema.test_copy... + ^ +-- Returned value is wrong +COPY copy_data FROM stdin WITH (FORMAT 'test_schema.test_copy_format_wrong_return_value'); +ERROR: COPY handler function test_schema.test_copy_format_wrong_return_value did not return CopyFromRoutine struct +COPY copy_data TO stdout WITH (FORMAT 'test_schema.test_copy_format_wrong_return_value'); +ERROR: COPY handler function test_schema.test_copy_format_wrong_return_value did not return CopyToRoutine struct +DROP TABLE copy_data; +DROP EXTENSION test_copy_format; +DROP SCHEMA test_schema; diff --git a/src/test/modules/test_copy_format/meson.build b/src/test/modules/test_copy_format/meson.build new file mode 100644 index 00000000000..a45a2e0a039 --- /dev/null +++ b/src/test/modules/test_copy_format/meson.build @@ -0,0 +1,33 @@ +# Copyright (c) 2025, PostgreSQL Global Development Group + +test_copy_format_sources = files( + 'test_copy_format.c', +) + +if host_system == 'windows' + test_copy_format_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_copy_format', + '--FILEDESC', 'test_copy_format - test custom COPY FORMAT',]) +endif + +test_copy_format = shared_module('test_copy_format', + test_copy_format_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_copy_format + +test_install_data += files( + 'test_copy_format.control', + 'test_copy_format--1.0.sql', +) + +tests += { + 'name': 'test_copy_format', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'regress': { + 'sql': [ + 'test_copy_format', + ], + }, +} diff --git a/src/test/modules/test_copy_format/sql/test_copy_format.sql b/src/test/modules/test_copy_format/sql/test_copy_format.sql new file mode 100644 index 00000000000..b262794f878 --- /dev/null +++ b/src/test/modules/test_copy_format/sql/test_copy_format.sql @@ -0,0 +1,52 @@ +CREATE TABLE copy_data (a smallint, b integer, c bigint); +INSERT INTO copy_data VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789); + +-- No WITH SCHEMA. It installs custom COPY handlers to the current +-- schema. +CREATE EXTENSION test_copy_format; +-- We can find a custom COPY handler without schema. +COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format'); +\. +COPY copy_data TO stdout WITH (FORMAT 'test_copy_format'); +DROP EXTENSION test_copy_format; + + +-- Install custom COPY handlers to a schema that isn't included in +-- search_path. +CREATE SCHEMA test_schema; +CREATE EXTENSION test_copy_format WITH SCHEMA test_schema; + +-- We can find a custom COPY handler by qualified name. +COPY copy_data FROM stdin WITH (FORMAT 'test_schema.test_copy_format'); +\. +COPY copy_data TO stdout WITH (FORMAT 'test_schema.test_copy_format'); + +-- We can't find a custom COPY handler without schema when search_path +-- doesn't include the schema where we installed custom COPY handlers. +COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format'); +COPY copy_data TO stdout WITH (FORMAT 'test_copy_format'); + +-- We can find a custom COPY handler without schema when search_path +-- includes the schema where we installed custom COPY handlers. +SET search_path = test_schema,public; +COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format'); +\. +COPY copy_data TO stdout WITH (FORMAT 'test_copy_format'); +RESET search_path; + +-- Invalid cases with qualified name. + +-- Input type is wrong +COPY copy_data FROM stdin WITH (FORMAT 'test_schema.test_copy_format_wrong_input_type'); +COPY copy_data TO stdout WITH (FORMAT 'test_schema.test_copy_format_wrong_input_type'); +-- Return type is wrong +COPY copy_data FROM stdin WITH (FORMAT 'test_schema.test_copy_format_wrong_return_type'); +COPY copy_data TO stdout WITH (FORMAT 'test_schema.test_copy_format_wrong_return_type'); +-- Returned value is wrong +COPY copy_data FROM stdin WITH (FORMAT 'test_schema.test_copy_format_wrong_return_value'); +COPY copy_data TO stdout WITH (FORMAT 'test_schema.test_copy_format_wrong_return_value'); + + +DROP TABLE copy_data; +DROP EXTENSION test_copy_format; +DROP SCHEMA test_schema; diff --git a/src/test/modules/test_copy_format/test_copy_format--1.0.sql b/src/test/modules/test_copy_format/test_copy_format--1.0.sql new file mode 100644 index 00000000000..c1a137181f8 --- /dev/null +++ b/src/test/modules/test_copy_format/test_copy_format--1.0.sql @@ -0,0 +1,24 @@ +/* src/test/modules/test_copy_format/test_copy_format--1.0.sql */ + +-- complain if script is sourced in psql, rather than via CREATE EXTENSION +\echo Use "CREATE EXTENSION test_copy_format" to load this file. \quit + +CREATE FUNCTION test_copy_format(internal) + RETURNS copy_handler + AS 'MODULE_PATHNAME', 'test_copy_format' + LANGUAGE C; + +CREATE FUNCTION test_copy_format_wrong_input_type(bool) + RETURNS copy_handler + AS 'MODULE_PATHNAME', 'test_copy_format' + LANGUAGE C; + +CREATE FUNCTION test_copy_format_wrong_return_type(internal) + RETURNS bool + AS 'MODULE_PATHNAME', 'test_copy_format' + LANGUAGE C; + +CREATE FUNCTION test_copy_format_wrong_return_value(internal) + RETURNS copy_handler + AS 'MODULE_PATHNAME', 'test_copy_format_wrong_return_value' + LANGUAGE C; diff --git a/src/test/modules/test_copy_format/test_copy_format.c b/src/test/modules/test_copy_format/test_copy_format.c new file mode 100644 index 00000000000..1d754201336 --- /dev/null +++ b/src/test/modules/test_copy_format/test_copy_format.c @@ -0,0 +1,113 @@ +/*-------------------------------------------------------------------------- + * + * test_copy_format.c + * Code for testing custom COPY format. + * + * Portions Copyright (c) 2025, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/test/modules/test_copy_format/test_copy_format.c + * + * ------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "commands/copyapi.h" +#include "commands/defrem.h" +#include "utils/builtins.h" + +PG_MODULE_MAGIC; + +static void +TestCopyFromInFunc(CopyFromState cstate, Oid atttypid, + FmgrInfo *finfo, Oid *typioparam) +{ + ereport(NOTICE, (errmsg("CopyFromInFunc: attribute: %s", format_type_be(atttypid)))); +} + +static void +TestCopyFromStart(CopyFromState cstate, TupleDesc tupDesc) +{ + ereport(NOTICE, (errmsg("CopyFromStart: the number of attributes: %d", tupDesc->natts))); +} + +static bool +TestCopyFromOneRow(CopyFromState cstate, ExprContext *econtext, Datum *values, bool *nulls) +{ + ereport(NOTICE, (errmsg("CopyFromOneRow"))); + return false; +} + +static void +TestCopyFromEnd(CopyFromState cstate) +{ + ereport(NOTICE, (errmsg("CopyFromEnd"))); +} + +static const CopyFromRoutine CopyFromRoutineTestCopyFormat = { + .type = T_CopyFromRoutine, + .CopyFromInFunc = TestCopyFromInFunc, + .CopyFromStart = TestCopyFromStart, + .CopyFromOneRow = TestCopyFromOneRow, + .CopyFromEnd = TestCopyFromEnd, +}; + +static void +TestCopyToOutFunc(CopyToState cstate, Oid atttypid, FmgrInfo *finfo) +{ + ereport(NOTICE, (errmsg("CopyToOutFunc: attribute: %s", format_type_be(atttypid)))); +} + +static void +TestCopyToStart(CopyToState cstate, TupleDesc tupDesc) +{ + ereport(NOTICE, (errmsg("CopyToStart: the number of attributes: %d", tupDesc->natts))); +} + +static void +TestCopyToOneRow(CopyToState cstate, TupleTableSlot *slot) +{ + ereport(NOTICE, (errmsg("CopyToOneRow: the number of valid values: %u", slot->tts_nvalid))); +} + +static void +TestCopyToEnd(CopyToState cstate) +{ + ereport(NOTICE, (errmsg("CopyToEnd"))); +} + +static const CopyToRoutine CopyToRoutineTestCopyFormat = { + .type = T_CopyToRoutine, + .CopyToOutFunc = TestCopyToOutFunc, + .CopyToStart = TestCopyToStart, + .CopyToOneRow = TestCopyToOneRow, + .CopyToEnd = TestCopyToEnd, +}; + +PG_FUNCTION_INFO_V1(test_copy_format); +Datum +test_copy_format(PG_FUNCTION_ARGS) +{ + bool is_from = PG_GETARG_BOOL(0); + + ereport(NOTICE, + (errmsg("test_copy_format: is_from=%s", is_from ? "true" : "false"))); + + if (is_from) + PG_RETURN_POINTER(&CopyFromRoutineTestCopyFormat); + else + PG_RETURN_POINTER(&CopyToRoutineTestCopyFormat); +} + +PG_FUNCTION_INFO_V1(test_copy_format_wrong_return_value); +Datum +test_copy_format_wrong_return_value(PG_FUNCTION_ARGS) +{ + bool is_from = PG_GETARG_BOOL(0); + + if (is_from) + PG_RETURN_CSTRING(pstrdup("is_from=true")); + else + PG_RETURN_CSTRING(pstrdup("is_from=false")); +} diff --git a/src/test/modules/test_copy_format/test_copy_format.control b/src/test/modules/test_copy_format/test_copy_format.control new file mode 100644 index 00000000000..f05a6362358 --- /dev/null +++ b/src/test/modules/test_copy_format/test_copy_format.control @@ -0,0 +1,4 @@ +comment = 'Test code for custom COPY format' +default_version = '1.0' +module_pathname = '$libdir/test_copy_format' +relocatable = true -- 2.47.2 [text/x-patch] v40-0003-Add-support-for-implementing-custom-COPY-handler.patch (15.1K, ../../[email protected]/4-v40-0003-Add-support-for-implementing-custom-COPY-handler.patch) download | inline diff: From 18618368721678d78934251ff8243705013458f0 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei <[email protected]> Date: Thu, 27 Mar 2025 11:24:15 +0900 Subject: [PATCH v40 3/6] Add support for implementing custom COPY handler as extension * TO: Add CopyToStateData::opaque that can be used to keep data for custom COPY TO handler implementation * TO: Export CopySendEndOfRow() to send end of row data as CopyToStateFlush() * FROM: Add CopyFromStateData::opaque that can be used to keep data for custom COPY FROM handler implementation * FROM: Export CopyGetData() to get the next data as CopyFromStateGetData() * FROM: Add CopyFromSkipErrorRow() for "ON_ERROR stop" and "LOG_VERBOSITY verbose" COPY FROM extensions must call CopyFromSkipErrorRow() when CopyFromOneRow callback reports an error by errsave(). CopyFromSkipErrorRow() handles "ON_ERROR stop" and "LOG_VERBOSITY verbose" cases. --- src/backend/commands/copyfromparse.c | 93 ++++++++++++------- src/backend/commands/copyto.c | 12 +++ src/include/commands/copyapi.h | 6 ++ src/include/commands/copyfrom_internal.h | 3 + src/include/commands/copyto_internal.h | 3 + .../expected/test_copy_format.out | 50 ++++++++++ .../test_copy_format/sql/test_copy_format.sql | 35 +++++++ .../test_copy_format/test_copy_format.c | 80 +++++++++++++++- 8 files changed, 245 insertions(+), 37 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 9f7171d1478..de68b53b000 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -739,6 +739,17 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) return copied_bytes; } +/* + * Export CopyGetData() for extensions. We want to keep CopyGetData() as a + * static function for optimization. CopyGetData() calls in this file may be + * optimized by a compiler. + */ +int +CopyFromStateGetData(CopyFromState cstate, void *dest, int minread, int maxread) +{ + return CopyGetData(cstate, dest, minread, maxread); +} + /* * This function is exposed for use by extensions that read raw fields in the * next line. See NextCopyFromRawFieldsInternal() for details. @@ -927,6 +938,51 @@ CopyFromCSVOneRow(CopyFromState cstate, ExprContext *econtext, Datum *values, return CopyFromTextLikeOneRow(cstate, econtext, values, nulls, true); } +/* + * Call this when you report an error by errsave() in your CopyFromOneRow + * callback. This handles "ON_ERROR stop" and "LOG_VERBOSITY verbose" cases + * for you. + */ +void +CopyFromSkipErrorRow(CopyFromState cstate) +{ + Assert(cstate->opts.on_error != COPY_ON_ERROR_STOP); + + cstate->num_errors++; + + if (cstate->opts.log_verbosity == COPY_LOG_VERBOSITY_VERBOSE) + { + /* + * Since we emit line number and column info in the below notice + * message, we suppress error context information other than the + * relation name. + */ + Assert(!cstate->relname_only); + cstate->relname_only = true; + + if (cstate->cur_attval) + { + char *attval; + + attval = CopyLimitPrintoutLength(cstate->cur_attval); + ereport(NOTICE, + errmsg("skipping row due to data type incompatibility at line %" PRIu64 " for column \"%s\": \"%s\"", + cstate->cur_lineno, + cstate->cur_attname, + attval)); + pfree(attval); + } + else + ereport(NOTICE, + errmsg("skipping row due to data type incompatibility at line %" PRIu64 " for column \"%s\": null input", + cstate->cur_lineno, + cstate->cur_attname)); + + /* reset relname_only */ + cstate->relname_only = false; + } +} + /* * Workhorse for CopyFromTextOneRow() and CopyFromCSVOneRow(). * @@ -1033,42 +1089,7 @@ CopyFromTextLikeOneRow(CopyFromState cstate, ExprContext *econtext, (Node *) cstate->escontext, &values[m])) { - Assert(cstate->opts.on_error != COPY_ON_ERROR_STOP); - - cstate->num_errors++; - - if (cstate->opts.log_verbosity == COPY_LOG_VERBOSITY_VERBOSE) - { - /* - * Since we emit line number and column info in the below - * notice message, we suppress error context information other - * than the relation name. - */ - Assert(!cstate->relname_only); - cstate->relname_only = true; - - if (cstate->cur_attval) - { - char *attval; - - attval = CopyLimitPrintoutLength(cstate->cur_attval); - ereport(NOTICE, - errmsg("skipping row due to data type incompatibility at line %" PRIu64 " for column \"%s\": \"%s\"", - cstate->cur_lineno, - cstate->cur_attname, - attval)); - pfree(attval); - } - else - ereport(NOTICE, - errmsg("skipping row due to data type incompatibility at line %" PRIu64 " for column \"%s\": null input", - cstate->cur_lineno, - cstate->cur_attname)); - - /* reset relname_only */ - cstate->relname_only = false; - } - + CopyFromSkipErrorRow(cstate); return true; } diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 265b847e255..d6fcfdfb9b1 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -454,6 +454,18 @@ CopySendEndOfRow(CopyToState cstate) resetStringInfo(fe_msgbuf); } +/* + * Export CopySendEndOfRow() for extensions. We want to keep + * CopySendEndOfRow() as a static function for + * optimization. CopySendEndOfRow() calls in this file may be optimized by a + * compiler. + */ +void +CopyToStateFlush(CopyToState cstate) +{ + CopySendEndOfRow(cstate); +} + /* * Wrapper function of CopySendEndOfRow for text and CSV formats. Sends the * line termination and do common appropriate things for the end of row. diff --git a/src/include/commands/copyapi.h b/src/include/commands/copyapi.h index 53ad3337f86..500ece7d5bb 100644 --- a/src/include/commands/copyapi.h +++ b/src/include/commands/copyapi.h @@ -56,6 +56,8 @@ typedef struct CopyToRoutine void (*CopyToEnd) (CopyToState cstate); } CopyToRoutine; +extern void CopyToStateFlush(CopyToState cstate); + /* * API structure for a COPY FROM format implementation. Note this must be * allocated in a server-lifetime manner, typically as a static const struct. @@ -106,4 +108,8 @@ typedef struct CopyFromRoutine void (*CopyFromEnd) (CopyFromState cstate); } CopyFromRoutine; +extern int CopyFromStateGetData(CopyFromState cstate, void *dest, int minread, int maxread); + +extern void CopyFromSkipErrorRow(CopyFromState cstate); + #endif /* COPYAPI_H */ diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 24157e11a73..f9e27152313 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -181,6 +181,9 @@ typedef struct CopyFromStateData #define RAW_BUF_BYTES(cstate) ((cstate)->raw_buf_len - (cstate)->raw_buf_index) uint64 bytes_processed; /* number of bytes processed so far */ + + /* For custom format implementation */ + void *opaque; /* private space */ } CopyFromStateData; extern void ReceiveCopyBegin(CopyFromState cstate); diff --git a/src/include/commands/copyto_internal.h b/src/include/commands/copyto_internal.h index da796131988..3bd9d702bf0 100644 --- a/src/include/commands/copyto_internal.h +++ b/src/include/commands/copyto_internal.h @@ -78,6 +78,9 @@ typedef struct CopyToStateData FmgrInfo *out_functions; /* lookup info for output functions */ MemoryContext rowcontext; /* per-row evaluation context */ uint64 bytes_processed; /* number of bytes processed so far */ + + /* For custom format implementation */ + void *opaque; /* private space */ } CopyToStateData; #endif /* COPYTO_INTERNAL_H */ diff --git a/src/test/modules/test_copy_format/expected/test_copy_format.out b/src/test/modules/test_copy_format/expected/test_copy_format.out index 3916b766615..47a875f0ab1 100644 --- a/src/test/modules/test_copy_format/expected/test_copy_format.out +++ b/src/test/modules/test_copy_format/expected/test_copy_format.out @@ -4,6 +4,8 @@ INSERT INTO copy_data VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789); -- schema. CREATE EXTENSION test_copy_format; -- We can find a custom COPY handler without schema. +-- 987 is accepted. +-- 654 is a hard error because ON_ERROR is stop by default. COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format'); NOTICE: test_copy_format: is_from=true NOTICE: CopyFromInFunc: attribute: smallint @@ -11,7 +13,50 @@ NOTICE: CopyFromInFunc: attribute: integer NOTICE: CopyFromInFunc: attribute: bigint NOTICE: CopyFromStart: the number of attributes: 3 NOTICE: CopyFromOneRow +NOTICE: CopyFromOneRow +ERROR: invalid value: "6" +CONTEXT: COPY copy_data, line 2, column a: "6" +-- 987 is accepted. +-- 654 is a soft error because ON_ERROR is ignore. +COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format', ON_ERROR ignore); +NOTICE: test_copy_format: is_from=true +NOTICE: CopyFromInFunc: attribute: smallint +NOTICE: CopyFromInFunc: attribute: integer +NOTICE: CopyFromInFunc: attribute: bigint +NOTICE: CopyFromStart: the number of attributes: 3 +NOTICE: CopyFromOneRow +NOTICE: CopyFromOneRow +NOTICE: CopyFromOneRow +NOTICE: 1 row was skipped due to data type incompatibility NOTICE: CopyFromEnd +-- 987 is accepted. +-- 654 is a soft error because ON_ERROR is ignore. +COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format', ON_ERROR ignore, LOG_VERBOSITY verbose); +NOTICE: test_copy_format: is_from=true +NOTICE: CopyFromInFunc: attribute: smallint +NOTICE: CopyFromInFunc: attribute: integer +NOTICE: CopyFromInFunc: attribute: bigint +NOTICE: CopyFromStart: the number of attributes: 3 +NOTICE: CopyFromOneRow +NOTICE: CopyFromOneRow +NOTICE: skipping row due to data type incompatibility at line 2 for column "a": "6" +NOTICE: CopyFromOneRow +NOTICE: 1 row was skipped due to data type incompatibility +NOTICE: CopyFromEnd +-- 987 is accepted. +-- 654 is a soft error because ON_ERROR is ignore. +-- 321 is a hard error. +COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format', ON_ERROR ignore); +NOTICE: test_copy_format: is_from=true +NOTICE: CopyFromInFunc: attribute: smallint +NOTICE: CopyFromInFunc: attribute: integer +NOTICE: CopyFromInFunc: attribute: bigint +NOTICE: CopyFromStart: the number of attributes: 3 +NOTICE: CopyFromOneRow +NOTICE: CopyFromOneRow +NOTICE: CopyFromOneRow +ERROR: too much lines: 3 +CONTEXT: COPY copy_data, line 3 COPY copy_data TO stdout WITH (FORMAT 'test_copy_format'); NOTICE: test_copy_format: is_from=false NOTICE: CopyToOutFunc: attribute: smallint @@ -21,7 +66,12 @@ NOTICE: CopyToStart: the number of attributes: 3 NOTICE: CopyToOneRow: the number of valid values: 3 NOTICE: CopyToOneRow: the number of valid values: 3 NOTICE: CopyToOneRow: the number of valid values: 3 +NOTICE: CopyToOneRow: the number of valid values: 3 +NOTICE: CopyToOneRow: the number of valid values: 3 NOTICE: CopyToEnd +-- Reset data. +TRUNCATE copy_data; +INSERT INTO copy_data VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789); DROP EXTENSION test_copy_format; -- Install custom COPY handlers to a schema that isn't included in -- search_path. diff --git a/src/test/modules/test_copy_format/sql/test_copy_format.sql b/src/test/modules/test_copy_format/sql/test_copy_format.sql index b262794f878..c7beb2fb8ae 100644 --- a/src/test/modules/test_copy_format/sql/test_copy_format.sql +++ b/src/test/modules/test_copy_format/sql/test_copy_format.sql @@ -4,10 +4,45 @@ INSERT INTO copy_data VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789); -- No WITH SCHEMA. It installs custom COPY handlers to the current -- schema. CREATE EXTENSION test_copy_format; + -- We can find a custom COPY handler without schema. + +-- 987 is accepted. +-- 654 is a hard error because ON_ERROR is stop by default. COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format'); +987 +654 \. + +-- 987 is accepted. +-- 654 is a soft error because ON_ERROR is ignore. +COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format', ON_ERROR ignore); +987 +654 +\. + +-- 987 is accepted. +-- 654 is a soft error because ON_ERROR is ignore. +COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format', ON_ERROR ignore, LOG_VERBOSITY verbose); +987 +654 +\. + +-- 987 is accepted. +-- 654 is a soft error because ON_ERROR is ignore. +-- 321 is a hard error. +COPY copy_data FROM stdin WITH (FORMAT 'test_copy_format', ON_ERROR ignore); +987 +654 +321 +\. + COPY copy_data TO stdout WITH (FORMAT 'test_copy_format'); + +-- Reset data. +TRUNCATE copy_data; +INSERT INTO copy_data VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789); + DROP EXTENSION test_copy_format; diff --git a/src/test/modules/test_copy_format/test_copy_format.c b/src/test/modules/test_copy_format/test_copy_format.c index 1d754201336..34ec693a7ec 100644 --- a/src/test/modules/test_copy_format/test_copy_format.c +++ b/src/test/modules/test_copy_format/test_copy_format.c @@ -14,6 +14,7 @@ #include "postgres.h" #include "commands/copyapi.h" +#include "commands/copyfrom_internal.h" #include "commands/defrem.h" #include "utils/builtins.h" @@ -35,8 +36,85 @@ TestCopyFromStart(CopyFromState cstate, TupleDesc tupDesc) static bool TestCopyFromOneRow(CopyFromState cstate, ExprContext *econtext, Datum *values, bool *nulls) { + int n_attributes = list_length(cstate->attnumlist); + char *line; + int line_size = n_attributes + 1; /* +1 is for new line */ + int read_bytes; + ereport(NOTICE, (errmsg("CopyFromOneRow"))); - return false; + + cstate->cur_lineno++; + line = palloc(line_size); + read_bytes = CopyFromStateGetData(cstate, line, line_size, line_size); + if (read_bytes == 0) + return false; + if (read_bytes != line_size) + ereport(ERROR, + (errcode(ERRCODE_BAD_COPY_FILE_FORMAT), + errmsg("one line must be %d bytes: %d", + line_size, read_bytes))); + + if (cstate->cur_lineno == 1) + { + /* Success */ + TupleDesc tupDesc = RelationGetDescr(cstate->rel); + ListCell *cur; + int i = 0; + + foreach(cur, cstate->attnumlist) + { + int attnum = lfirst_int(cur); + int m = attnum - 1; + Form_pg_attribute att = TupleDescAttr(tupDesc, m); + + if (att->atttypid == INT2OID) + { + values[i] = Int16GetDatum(line[i] - '0'); + } + else if (att->atttypid == INT4OID) + { + values[i] = Int32GetDatum(line[i] - '0'); + } + else if (att->atttypid == INT8OID) + { + values[i] = Int64GetDatum(line[i] - '0'); + } + nulls[i] = false; + i++; + } + } + else if (cstate->cur_lineno == 2) + { + /* Soft error */ + TupleDesc tupDesc = RelationGetDescr(cstate->rel); + int attnum = lfirst_int(list_head(cstate->attnumlist)); + int m = attnum - 1; + Form_pg_attribute att = TupleDescAttr(tupDesc, m); + char value[2]; + + cstate->cur_attname = NameStr(att->attname); + value[0] = line[0]; + value[1] = '\0'; + cstate->cur_attval = value; + errsave((Node *) cstate->escontext, + ( + errcode(ERRCODE_INVALID_TEXT_REPRESENTATION), + errmsg("invalid value: \"%c\"", line[0]))); + CopyFromSkipErrorRow(cstate); + cstate->cur_attname = NULL; + cstate->cur_attval = NULL; + return true; + } + else + { + /* Hard error */ + ereport(ERROR, + (errcode(ERRCODE_BAD_COPY_FILE_FORMAT), + errmsg("too much lines: %llu", + (unsigned long long) cstate->cur_lineno))); + } + + return true; } static void -- 2.47.2 [text/x-patch] v40-0004-Use-copy-handlers-for-built-in-formats.patch (17.8K, ../../[email protected]/5-v40-0004-Use-copy-handlers-for-built-in-formats.patch) download | inline diff: From ed454fd1998bca012182b977c227b4a0caa3ccd6 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei <[email protected]> Date: Thu, 27 Mar 2025 11:56:45 +0900 Subject: [PATCH v40 4/6] Use copy handlers for built-in formats This adds copy handlers for text, csv and binary. We can simplify Copy{To,From}GetRoutine() by this. We'll be able to remove CopyFormatOptions::{binary,csv_mode} when we add more callbacks to Copy{To,From}Routine and move format specific routines to Copy{To,From}Routine::*. --- src/backend/commands/copy.c | 101 ++++++++++++------ src/backend/commands/copyfrom.c | 42 ++++---- src/backend/commands/copyto.c | 42 ++++---- src/include/catalog/pg_proc.dat | 11 ++ src/include/commands/copy.h | 2 +- src/include/commands/copyfrom_internal.h | 6 +- src/include/commands/copyto_internal.h | 6 +- .../expected/test_copy_format.out | 35 ++++++ .../test_copy_format/sql/test_copy_format.sql | 32 ++++++ .../test_copy_format--1.0.sql | 15 +++ 10 files changed, 211 insertions(+), 81 deletions(-) diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 9515c4d5786..38ed8bccacd 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -22,7 +22,9 @@ #include "access/table.h" #include "access/xact.h" #include "catalog/pg_authid.h" -#include "commands/copy.h" +#include "commands/copyapi.h" +#include "commands/copyto_internal.h" +#include "commands/copyfrom_internal.h" #include "commands/defrem.h" #include "executor/executor.h" #include "mb/pg_wchar.h" @@ -521,43 +523,45 @@ ProcessCopyOptions(ParseState *pstate, if (strcmp(defel->defname, "format") == 0) { - char *fmt = defGetString(defel); + char *format = defGetString(defel); + List *qualified_format; + char *schema; + char *fmt; + Oid arg_types[1]; + Oid handler = InvalidOid; if (format_specified) errorConflictingDefElem(defel, pstate); format_specified = true; - if (strcmp(fmt, "text") == 0) - /* default format */ ; - else if (strcmp(fmt, "csv") == 0) - opts_out->csv_mode = true; - else if (strcmp(fmt, "binary") == 0) - opts_out->binary = true; - else + + qualified_format = stringToQualifiedNameList(format, NULL); + DeconstructQualifiedName(qualified_format, &schema, &fmt); + if (!schema || strcmp(schema, "pg_catalog") == 0) { - List *qualified_format; - Oid arg_types[1]; - Oid handler = InvalidOid; - - qualified_format = stringToQualifiedNameList(fmt, NULL); - arg_types[0] = INTERNALOID; - handler = LookupFuncName(qualified_format, 1, - arg_types, true); - if (!OidIsValid(handler)) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("COPY format \"%s\" not recognized", fmt), - parser_errposition(pstate, defel->location))); - - /* check that handler has correct return type */ - if (get_func_rettype(handler) != COPY_HANDLEROID) - ereport(ERROR, - (errcode(ERRCODE_WRONG_OBJECT_TYPE), - errmsg("function %s must return type %s", - fmt, "copy_handler"), - parser_errposition(pstate, defel->location))); - - opts_out->handler = handler; + if (strcmp(fmt, "csv") == 0) + opts_out->csv_mode = true; + else if (strcmp(fmt, "binary") == 0) + opts_out->binary = true; } + + arg_types[0] = INTERNALOID; + handler = LookupFuncName(qualified_format, 1, + arg_types, true); + if (!OidIsValid(handler)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY format \"%s\" not recognized", format), + parser_errposition(pstate, defel->location))); + + /* check that handler has correct return type */ + if (get_func_rettype(handler) != COPY_HANDLEROID) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("function %s must return type %s", + format, "copy_handler"), + parser_errposition(pstate, defel->location))); + + opts_out->handler = handler; } else if (strcmp(defel->defname, "freeze") == 0) { @@ -1040,3 +1044,36 @@ CopyGetAttnums(TupleDesc tupDesc, Relation rel, List *attnamelist) return attnums; } + +Datum +copy_text_handler(PG_FUNCTION_ARGS) +{ + bool is_from = PG_GETARG_BOOL(0); + + if (is_from) + PG_RETURN_POINTER(&CopyFromRoutineText); + else + PG_RETURN_POINTER(&CopyToRoutineText); +} + +Datum +copy_csv_handler(PG_FUNCTION_ARGS) +{ + bool is_from = PG_GETARG_BOOL(0); + + if (is_from) + PG_RETURN_POINTER(&CopyFromRoutineCSV); + else + PG_RETURN_POINTER(&CopyToRoutineCSV); +} + +Datum +copy_binary_handler(PG_FUNCTION_ARGS) +{ + bool is_from = PG_GETARG_BOOL(0); + + if (is_from) + PG_RETURN_POINTER(&CopyFromRoutineBinary); + else + PG_RETURN_POINTER(&CopyToRoutineBinary); +} diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 3d86e8a8328..74a8051c24c 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -45,6 +45,7 @@ #include "rewrite/rewriteHandler.h" #include "storage/fd.h" #include "tcop/tcopprot.h" +#include "utils/fmgroids.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/portal.h" @@ -128,7 +129,7 @@ static void CopyFromBinaryEnd(CopyFromState cstate); */ /* text format */ -static const CopyFromRoutine CopyFromRoutineText = { +const CopyFromRoutine CopyFromRoutineText = { .type = T_CopyFromRoutine, .CopyFromInFunc = CopyFromTextLikeInFunc, .CopyFromStart = CopyFromTextLikeStart, @@ -137,7 +138,7 @@ static const CopyFromRoutine CopyFromRoutineText = { }; /* CSV format */ -static const CopyFromRoutine CopyFromRoutineCSV = { +const CopyFromRoutine CopyFromRoutineCSV = { .type = T_CopyFromRoutine, .CopyFromInFunc = CopyFromTextLikeInFunc, .CopyFromStart = CopyFromTextLikeStart, @@ -146,7 +147,7 @@ static const CopyFromRoutine CopyFromRoutineCSV = { }; /* binary format */ -static const CopyFromRoutine CopyFromRoutineBinary = { +const CopyFromRoutine CopyFromRoutineBinary = { .type = T_CopyFromRoutine, .CopyFromInFunc = CopyFromBinaryInFunc, .CopyFromStart = CopyFromBinaryStart, @@ -158,28 +159,23 @@ static const CopyFromRoutine CopyFromRoutineBinary = { static const CopyFromRoutine * CopyFromGetRoutine(const CopyFormatOptions *opts) { - if (OidIsValid(opts->handler)) - { - Datum datum; - Node *routine; - - datum = OidFunctionCall1(opts->handler, BoolGetDatum(true)); - routine = (Node *) DatumGetPointer(datum); - if (routine == NULL || !IsA(routine, CopyFromRoutine)) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("COPY handler function %s.%s did not return CopyFromRoutine struct", - get_namespace_name(get_func_namespace(opts->handler)), - get_func_name(opts->handler)))); - return castNode(CopyFromRoutine, routine); - } - else if (opts->csv_mode) - return &CopyFromRoutineCSV; - else if (opts->binary) - return &CopyFromRoutineBinary; + Oid handler = opts->handler; + Datum datum; + Node *routine; /* default is text */ - return &CopyFromRoutineText; + if (!OidIsValid(handler)) + handler = F_TEXT_INTERNAL; + + datum = OidFunctionCall1(handler, BoolGetDatum(true)); + routine = (Node *) DatumGetPointer(datum); + if (routine == NULL || !IsA(routine, CopyFromRoutine)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY handler function %s.%s did not return CopyFromRoutine struct", + get_namespace_name(get_func_namespace(handler)), + get_func_name(handler)))); + return castNode(CopyFromRoutine, routine); } /* Implementation of the start callback for text and CSV formats */ diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index d6fcfdfb9b1..4e1b154cad2 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -30,6 +30,7 @@ #include "pgstat.h" #include "storage/fd.h" #include "tcop/tcopprot.h" +#include "utils/fmgroids.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -87,7 +88,7 @@ static void CopySendInt16(CopyToState cstate, int16 val); */ /* text format */ -static const CopyToRoutine CopyToRoutineText = { +const CopyToRoutine CopyToRoutineText = { .type = T_CopyToRoutine, .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, @@ -96,7 +97,7 @@ static const CopyToRoutine CopyToRoutineText = { }; /* CSV format */ -static const CopyToRoutine CopyToRoutineCSV = { +const CopyToRoutine CopyToRoutineCSV = { .type = T_CopyToRoutine, .CopyToStart = CopyToTextLikeStart, .CopyToOutFunc = CopyToTextLikeOutFunc, @@ -105,7 +106,7 @@ static const CopyToRoutine CopyToRoutineCSV = { }; /* binary format */ -static const CopyToRoutine CopyToRoutineBinary = { +const CopyToRoutine CopyToRoutineBinary = { .type = T_CopyToRoutine, .CopyToStart = CopyToBinaryStart, .CopyToOutFunc = CopyToBinaryOutFunc, @@ -117,28 +118,23 @@ static const CopyToRoutine CopyToRoutineBinary = { static const CopyToRoutine * CopyToGetRoutine(const CopyFormatOptions *opts) { - if (OidIsValid(opts->handler)) - { - Datum datum; - Node *routine; - - datum = OidFunctionCall1(opts->handler, BoolGetDatum(false)); - routine = (Node *) DatumGetPointer(datum); - if (routine == NULL || !IsA(routine, CopyToRoutine)) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("COPY handler function %s.%s did not return CopyToRoutine struct", - get_namespace_name(get_func_namespace(opts->handler)), - get_func_name(opts->handler)))); - return castNode(CopyToRoutine, routine); - } - else if (opts->csv_mode) - return &CopyToRoutineCSV; - else if (opts->binary) - return &CopyToRoutineBinary; + Oid handler = opts->handler; + Datum datum; + Node *routine; /* default is text */ - return &CopyToRoutineText; + if (!OidIsValid(handler)) + handler = F_TEXT_INTERNAL; + + datum = OidFunctionCall1(handler, BoolGetDatum(false)); + routine = (Node *) DatumGetPointer(datum); + if (routine == NULL || !IsA(routine, CopyToRoutine)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("COPY handler function %s.%s did not return CopyToRoutine struct", + get_namespace_name(get_func_namespace(handler)), + get_func_name(handler)))); + return castNode(CopyToRoutine, routine); } /* Implementation of the start callback for text and CSV formats */ diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index ba46bfa48a8..e038157eb74 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12572,4 +12572,15 @@ proargnames => '{pid,io_id,io_generation,state,operation,off,length,target,handle_data_len,raw_result,result,target_desc,f_sync,f_localmem,f_buffered}', prosrc => 'pg_get_aios' }, +# COPY handlers +{ oid => '8100', descr => 'text COPY FORMAT handler', + proname => 'text', provolatile => 'i', prorettype => 'copy_handler', + proargtypes => 'internal', prosrc => 'copy_text_handler' }, +{ oid => '8101', descr => 'csv COPY FORMAT handler', + proname => 'csv', provolatile => 'i', prorettype => 'copy_handler', + proargtypes => 'internal', prosrc => 'copy_csv_handler' }, +{ oid => '8102', descr => 'binary COPY FORMAT handler', + proname => 'binary', provolatile => 'i', prorettype => 'copy_handler', + proargtypes => 'internal', prosrc => 'copy_binary_handler' }, + ] diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 6df1f8a3b9b..4525261fcc4 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -87,7 +87,7 @@ typedef struct CopyFormatOptions CopyLogVerbosityChoice log_verbosity; /* verbosity of logged messages */ int64 reject_limit; /* maximum tolerable number of errors */ List *convert_select; /* list of column names (can be NIL) */ - Oid handler; /* handler function for custom format routine */ + Oid handler; /* handler function */ } CopyFormatOptions; /* These are private in commands/copy[from|to]_internal.h */ diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index f9e27152313..51d181c3ab4 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -14,7 +14,7 @@ #ifndef COPYFROM_INTERNAL_H #define COPYFROM_INTERNAL_H -#include "commands/copy.h" +#include "commands/copyapi.h" #include "commands/trigger.h" #include "nodes/miscnodes.h" @@ -197,4 +197,8 @@ extern bool CopyFromCSVOneRow(CopyFromState cstate, ExprContext *econtext, extern bool CopyFromBinaryOneRow(CopyFromState cstate, ExprContext *econtext, Datum *values, bool *nulls); +extern PGDLLIMPORT const CopyFromRoutine CopyFromRoutineText; +extern PGDLLIMPORT const CopyFromRoutine CopyFromRoutineCSV; +extern PGDLLIMPORT const CopyFromRoutine CopyFromRoutineBinary; + #endif /* COPYFROM_INTERNAL_H */ diff --git a/src/include/commands/copyto_internal.h b/src/include/commands/copyto_internal.h index 3bd9d702bf0..9faf97c718a 100644 --- a/src/include/commands/copyto_internal.h +++ b/src/include/commands/copyto_internal.h @@ -14,7 +14,7 @@ #ifndef COPYTO_INTERNAL_H #define COPYTO_INTERNAL_H -#include "commands/copy.h" +#include "commands/copyapi.h" #include "executor/execdesc.h" #include "executor/tuptable.h" #include "nodes/execnodes.h" @@ -83,4 +83,8 @@ typedef struct CopyToStateData void *opaque; /* private space */ } CopyToStateData; +extern PGDLLIMPORT const CopyToRoutine CopyToRoutineText; +extern PGDLLIMPORT const CopyToRoutine CopyToRoutineCSV; +extern PGDLLIMPORT const CopyToRoutine CopyToRoutineBinary; + #endif /* COPYTO_INTERNAL_H */ diff --git a/src/test/modules/test_copy_format/expected/test_copy_format.out b/src/test/modules/test_copy_format/expected/test_copy_format.out index 47a875f0ab1..aa51e480b1d 100644 --- a/src/test/modules/test_copy_format/expected/test_copy_format.out +++ b/src/test/modules/test_copy_format/expected/test_copy_format.out @@ -72,6 +72,41 @@ NOTICE: CopyToEnd -- Reset data. TRUNCATE copy_data; INSERT INTO copy_data VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789); +-- test_copy_format extension installs text, csv and binary custom +-- COPY handlers to the public schema but they must not be +-- used. Builtin COPY handlers must be used. +-- public.text must not be used +COPY copy_data FROM stdin WITH (FORMAT text); +COPY copy_data TO stdout WITH (FORMAT text); +1 2 3 +12 34 56 +123 456 789 +COPY copy_data FROM stdin WITH (FORMAT 'pg_catalog.text'); +COPY copy_data TO stdout WITH (FORMAT 'pg_catalog.text'); +1 2 3 +12 34 56 +123 456 789 +-- public.csv must not be used +COPY copy_data FROM stdin WITH (FORMAT csv); +COPY copy_data TO stdout WITH (FORMAT csv); +1,2,3 +12,34,56 +123,456,789 +COPY copy_data FROM stdin WITH (FORMAT 'pg_catalog.csv'); +COPY copy_data TO stdout WITH (FORMAT 'pg_catalog.csv'); +1,2,3 +12,34,56 +123,456,789 +-- public.binary must not be used +\getenv abs_builddir PG_ABS_BUILDDIR +\set filename :abs_builddir '/results/binary.data' +COPY copy_data TO :'filename' WITH (FORMAT binary); +COPY copy_data FROM :'filename' WITH (FORMAT binary); +COPY copy_data TO :'filename' WITH (FORMAT 'pg_catalog.binary'); +COPY copy_data FROM :'filename' WITH (FORMAT 'pg_catalog.binary'); +-- Reset data. +TRUNCATE copy_data; +INSERT INTO copy_data VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789); DROP EXTENSION test_copy_format; -- Install custom COPY handlers to a schema that isn't included in -- search_path. diff --git a/src/test/modules/test_copy_format/sql/test_copy_format.sql b/src/test/modules/test_copy_format/sql/test_copy_format.sql index c7beb2fb8ae..3b7f6e72e13 100644 --- a/src/test/modules/test_copy_format/sql/test_copy_format.sql +++ b/src/test/modules/test_copy_format/sql/test_copy_format.sql @@ -43,6 +43,38 @@ COPY copy_data TO stdout WITH (FORMAT 'test_copy_format'); TRUNCATE copy_data; INSERT INTO copy_data VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789); +-- test_copy_format extension installs text, csv and binary custom +-- COPY handlers to the public schema but they must not be +-- used. Builtin COPY handlers must be used. + +-- public.text must not be used +COPY copy_data FROM stdin WITH (FORMAT text); +\. +COPY copy_data TO stdout WITH (FORMAT text); +COPY copy_data FROM stdin WITH (FORMAT 'pg_catalog.text'); +\. +COPY copy_data TO stdout WITH (FORMAT 'pg_catalog.text'); + +-- public.csv must not be used +COPY copy_data FROM stdin WITH (FORMAT csv); +\. +COPY copy_data TO stdout WITH (FORMAT csv); +COPY copy_data FROM stdin WITH (FORMAT 'pg_catalog.csv'); +\. +COPY copy_data TO stdout WITH (FORMAT 'pg_catalog.csv'); + +-- public.binary must not be used +\getenv abs_builddir PG_ABS_BUILDDIR +\set filename :abs_builddir '/results/binary.data' +COPY copy_data TO :'filename' WITH (FORMAT binary); +COPY copy_data FROM :'filename' WITH (FORMAT binary); +COPY copy_data TO :'filename' WITH (FORMAT 'pg_catalog.binary'); +COPY copy_data FROM :'filename' WITH (FORMAT 'pg_catalog.binary'); + +-- Reset data. +TRUNCATE copy_data; +INSERT INTO copy_data VALUES (1, 2, 3), (12, 34, 56), (123, 456, 789); + DROP EXTENSION test_copy_format; diff --git a/src/test/modules/test_copy_format/test_copy_format--1.0.sql b/src/test/modules/test_copy_format/test_copy_format--1.0.sql index c1a137181f8..bfa1900e828 100644 --- a/src/test/modules/test_copy_format/test_copy_format--1.0.sql +++ b/src/test/modules/test_copy_format/test_copy_format--1.0.sql @@ -22,3 +22,18 @@ CREATE FUNCTION test_copy_format_wrong_return_value(internal) RETURNS copy_handler AS 'MODULE_PATHNAME', 'test_copy_format_wrong_return_value' LANGUAGE C; + +CREATE FUNCTION text(internal) + RETURNS copy_handler + AS 'MODULE_PATHNAME', 'test_copy_format' + LANGUAGE C; + +CREATE FUNCTION csv(internal) + RETURNS copy_handler + AS 'MODULE_PATHNAME', 'test_copy_format' + LANGUAGE C; + +CREATE FUNCTION binary(internal) + RETURNS copy_handler + AS 'MODULE_PATHNAME', 'test_copy_format' + LANGUAGE C; -- 2.47.2 [text/x-patch] v40-0005-Remove-CopyFormatOptions-binary-csv_mode.patch (11.1K, ../../[email protected]/6-v40-0005-Remove-CopyFormatOptions-binary-csv_mode.patch) download | inline diff: From 6e014bf226713a2c9f37da4c4f337128c4392212 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei <[email protected]> Date: Fri, 25 Apr 2025 18:49:41 +0900 Subject: [PATCH v40 5/6] Remove CopyFormatOptions::{binary,csv_mode} Because we can compute them from CopyFormatOptions::handler. --- src/backend/commands/copy.c | 61 +++++++++++++--------------- src/backend/commands/copyfrom.c | 2 +- src/backend/commands/copyfromparse.c | 7 ++-- src/backend/commands/copyto.c | 4 +- src/include/commands/copy.h | 2 - 5 files changed, 36 insertions(+), 40 deletions(-) diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 38ed8bccacd..21db5e964cf 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -38,6 +38,7 @@ #include "parser/parse_relation.h" #include "utils/acl.h" #include "utils/builtins.h" +#include "utils/fmgroids.h" #include "utils/lsyscache.h" #include "utils/regproc.h" #include "utils/rel.h" @@ -508,6 +509,8 @@ ProcessCopyOptions(ParseState *pstate, bool on_error_specified = false; bool log_verbosity_specified = false; bool reject_limit_specified = false; + bool binary = false; + bool csv_mode = false; ListCell *option; /* Support external use for option sanity checking */ @@ -525,8 +528,6 @@ ProcessCopyOptions(ParseState *pstate, { char *format = defGetString(defel); List *qualified_format; - char *schema; - char *fmt; Oid arg_types[1]; Oid handler = InvalidOid; @@ -535,15 +536,6 @@ ProcessCopyOptions(ParseState *pstate, format_specified = true; qualified_format = stringToQualifiedNameList(format, NULL); - DeconstructQualifiedName(qualified_format, &schema, &fmt); - if (!schema || strcmp(schema, "pg_catalog") == 0) - { - if (strcmp(fmt, "csv") == 0) - opts_out->csv_mode = true; - else if (strcmp(fmt, "binary") == 0) - opts_out->binary = true; - } - arg_types[0] = INTERNALOID; handler = LookupFuncName(qualified_format, 1, arg_types, true); @@ -562,6 +554,11 @@ ProcessCopyOptions(ParseState *pstate, parser_errposition(pstate, defel->location))); opts_out->handler = handler; + if (opts_out->handler == F_CSV) + csv_mode = true; + else if (opts_out->handler == F_BINARY) + binary = true; + } else if (strcmp(defel->defname, "freeze") == 0) { @@ -716,31 +713,31 @@ ProcessCopyOptions(ParseState *pstate, * Check for incompatible options (must do these three before inserting * defaults) */ - if (opts_out->binary && opts_out->delim) + if (binary && opts_out->delim) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "DELIMITER"))); - if (opts_out->binary && opts_out->null_print) + if (binary && opts_out->null_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "NULL"))); - if (opts_out->binary && opts_out->default_print) + if (binary && opts_out->default_print) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("cannot specify %s in BINARY mode", "DEFAULT"))); /* Set defaults for omitted options */ if (!opts_out->delim) - opts_out->delim = opts_out->csv_mode ? "," : "\t"; + opts_out->delim = csv_mode ? "," : "\t"; if (!opts_out->null_print) - opts_out->null_print = opts_out->csv_mode ? "" : "\\N"; + opts_out->null_print = csv_mode ? "" : "\\N"; opts_out->null_print_len = strlen(opts_out->null_print); - if (opts_out->csv_mode) + if (csv_mode) { if (!opts_out->quote) opts_out->quote = "\""; @@ -788,7 +785,7 @@ ProcessCopyOptions(ParseState *pstate, * future-proofing. Likewise we disallow all digits though only octal * digits are actually dangerous. */ - if (!opts_out->csv_mode && + if (!csv_mode && strchr("\\.abcdefghijklmnopqrstuvwxyz0123456789", opts_out->delim[0]) != NULL) ereport(ERROR, @@ -796,43 +793,43 @@ ProcessCopyOptions(ParseState *pstate, errmsg("COPY delimiter cannot be \"%s\"", opts_out->delim))); /* Check header */ - if (opts_out->binary && opts_out->header_line) + if (binary && opts_out->header_line) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("cannot specify %s in BINARY mode", "HEADER"))); /* Check quote */ - if (!opts_out->csv_mode && opts_out->quote != NULL) + if (!csv_mode && opts_out->quote != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "QUOTE"))); - if (opts_out->csv_mode && strlen(opts_out->quote) != 1) + if (csv_mode && strlen(opts_out->quote) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY quote must be a single one-byte character"))); - if (opts_out->csv_mode && opts_out->delim[0] == opts_out->quote[0]) + if (csv_mode && opts_out->delim[0] == opts_out->quote[0]) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("COPY delimiter and quote must be different"))); /* Check escape */ - if (!opts_out->csv_mode && opts_out->escape != NULL) + if (!csv_mode && opts_out->escape != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ errmsg("COPY %s requires CSV mode", "ESCAPE"))); - if (opts_out->csv_mode && strlen(opts_out->escape) != 1) + if (csv_mode && strlen(opts_out->escape) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("COPY escape must be a single one-byte character"))); /* Check force_quote */ - if (!opts_out->csv_mode && (opts_out->force_quote || opts_out->force_quote_all)) + if (!csv_mode && (opts_out->force_quote || opts_out->force_quote_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -846,8 +843,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY FROM"))); /* Check force_notnull */ - if (!opts_out->csv_mode && (opts_out->force_notnull != NIL || - opts_out->force_notnull_all)) + if (!csv_mode && (opts_out->force_notnull != NIL || + opts_out->force_notnull_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -862,8 +859,8 @@ ProcessCopyOptions(ParseState *pstate, "COPY TO"))); /* Check force_null */ - if (!opts_out->csv_mode && (opts_out->force_null != NIL || - opts_out->force_null_all)) + if (!csv_mode && (opts_out->force_null != NIL || + opts_out->force_null_all)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), /*- translator: %s is the name of a COPY option, e.g. ON_ERROR */ @@ -887,7 +884,7 @@ ProcessCopyOptions(ParseState *pstate, "NULL"))); /* Don't allow the CSV quote char to appear in the null string. */ - if (opts_out->csv_mode && + if (csv_mode && strchr(opts_out->null_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), @@ -923,7 +920,7 @@ ProcessCopyOptions(ParseState *pstate, "DEFAULT"))); /* Don't allow the CSV quote char to appear in the default string. */ - if (opts_out->csv_mode && + if (csv_mode && strchr(opts_out->default_print, opts_out->quote[0]) != NULL) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), @@ -940,7 +937,7 @@ ProcessCopyOptions(ParseState *pstate, errmsg("NULL specification and DEFAULT specification cannot be the same"))); } /* Check on_error */ - if (opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP) + if (binary && opts_out->on_error != COPY_ON_ERROR_STOP) ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), errmsg("only ON_ERROR STOP is allowed in BINARY mode"))); diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 74a8051c24c..b09b6b3e101 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -275,7 +275,7 @@ CopyFromErrorCallback(void *arg) cstate->cur_relname); return; } - if (cstate->opts.binary) + if (cstate->opts.handler == F_BINARY) { /* can't usefully display the data */ if (cstate->cur_attname) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index de68b53b000..148fa1f2062 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -73,6 +73,7 @@ #include "pgstat.h" #include "port/pg_bswap.h" #include "utils/builtins.h" +#include "utils/fmgroids.h" #include "utils/rel.h" #define ISOCTAL(c) (((c) >= '0') && ((c) <= '7')) @@ -171,7 +172,7 @@ ReceiveCopyBegin(CopyFromState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.handler == F_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyInResponse); @@ -758,7 +759,7 @@ bool NextCopyFromRawFields(CopyFromState cstate, char ***fields, int *nfields) { return NextCopyFromRawFieldsInternal(cstate, fields, nfields, - cstate->opts.csv_mode); + cstate->opts.handler == F_CSV); } /* @@ -785,7 +786,7 @@ NextCopyFromRawFieldsInternal(CopyFromState cstate, char ***fields, int *nfields bool done; /* only available for text or csv input */ - Assert(!cstate->opts.binary); + Assert(cstate->opts.handler != F_BINARY); /* on input check that the header line is correct if needed */ if (cstate->cur_lineno == 0 && cstate->opts.header_line) diff --git a/src/backend/commands/copyto.c b/src/backend/commands/copyto.c index 4e1b154cad2..4f8f5813172 100644 --- a/src/backend/commands/copyto.c +++ b/src/backend/commands/copyto.c @@ -167,7 +167,7 @@ CopyToTextLikeStart(CopyToState cstate, TupleDesc tupDesc) colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname); - if (cstate->opts.csv_mode) + if (cstate->opts.handler == F_CSV) CopyAttributeOutCSV(cstate, colname, false); else CopyAttributeOutText(cstate, colname); @@ -344,7 +344,7 @@ SendCopyBegin(CopyToState cstate) { StringInfoData buf; int natts = list_length(cstate->attnumlist); - int16 format = (cstate->opts.binary ? 1 : 0); + int16 format = (cstate->opts.handler == F_BINARY ? 1 : 0); int i; pq_beginmessage(&buf, PqMsg_CopyOutResponse); diff --git a/src/include/commands/copy.h b/src/include/commands/copy.h index 4525261fcc4..04f8f5ef1b2 100644 --- a/src/include/commands/copy.h +++ b/src/include/commands/copy.h @@ -61,9 +61,7 @@ typedef struct CopyFormatOptions /* parameters from the COPY command */ int file_encoding; /* file or remote side's character encoding, * -1 if not specified */ - bool binary; /* binary format? */ bool freeze; /* freeze rows on loading? */ - bool csv_mode; /* Comma Separated Value format? */ CopyHeaderChoice header_line; /* header line? */ char *null_print; /* NULL marker string (server encoding!) */ int null_print_len; /* length of same */ -- 2.47.2 [text/x-patch] v40-0006-Add-document-how-to-write-a-COPY-handler.patch (14.7K, ../../[email protected]/7-v40-0006-Add-document-how-to-write-a-COPY-handler.patch) download | inline diff: From 421c34b76a5e9fe45b49bdbe52ecda4d0f638617 Mon Sep 17 00:00:00 2001 From: Sutou Kouhei <[email protected]> Date: Wed, 19 Mar 2025 11:46:34 +0900 Subject: [PATCH v40 6/6] Add document how to write a COPY handler This is WIP because we haven't decided our API yet. Co-authored-by: David G. Johnston <[email protected]> --- doc/src/sgml/copy-handler.sgml | 394 +++++++++++++++++++++++++++++++++ doc/src/sgml/filelist.sgml | 1 + doc/src/sgml/postgres.sgml | 1 + src/include/commands/copyapi.h | 9 +- 4 files changed, 401 insertions(+), 4 deletions(-) create mode 100644 doc/src/sgml/copy-handler.sgml diff --git a/doc/src/sgml/copy-handler.sgml b/doc/src/sgml/copy-handler.sgml new file mode 100644 index 00000000000..5bc87d16662 --- /dev/null +++ b/doc/src/sgml/copy-handler.sgml @@ -0,0 +1,394 @@ +<!-- doc/src/sgml/copy-handler.sgml --> + +<chapter id="copy-handler"> + <title>Writing a Copy Handler</title> + + <indexterm zone="copy-handler"> + <primary><literal>COPY</literal> handler</primary> + </indexterm> + + <para> + <productname>PostgreSQL</productname> supports + custom <link linkend="sql-copy"><literal>COPY</literal></link> handlers; + adding additional <replaceable>format_name</replaceable> options to + the <literal>FORMAT</literal> clause. + </para> + + <para> + At the SQL level, a copy handler method is represented by a single SQL + function (see <xref linkend="sql-createfunction"/>), typically implemented in + C, having the signature +<synopsis> +<replaceable>format_name</replaceable>(internal) RETURNS <literal>copy_handler</literal> +</synopsis> + The function's name is then accepted as a + valid <replaceable>format_name</replaceable>. The return + pseudo-type <literal>copy_handler</literal> informs the system that this + function needs to be registered as a copy handler. + The <type>internal</type> argument is a dummy that prevents this function + from being called directly from an SQL command. As the handler + implementation must be server-lifetime immutable; this SQL function's + volatility should be marked immutable. The <literal>link_symbol</literal> + for this function is the name of the implementation function, described + next. + </para> + + <para> + The implementation function signature expected for the function named + in the <literal>link_symbol</literal> is: +<synopsis> +Datum +<replaceable>copy_format_handler</replaceable>(PG_FUNCTION_ARGS) +</synopsis> + The convention for the name is to replace the word + <replaceable>format</replaceable> in the placeholder above with the value given + to <replaceable>format_name</replaceable> in the SQL function. + The first argument is a <type>boolean</type> that indicates whether the handler + must provide a pointer to its implementation for <literal>COPY FROM</literal> + (a <type>CopyFromRoutine *</type>). If <literal>false</literal>, the handler + must provide a pointer to its implementation of <literal>COPY TO</literal> + (a <type>CopyToRoutine *</type>). These structs are declared in + <filename>src/include/commands/copyapi.h</filename>. + </para> + + <para> + The structs hold pointers to implementation functions for initializing, + starting, processing rows, and ending a copy operation. The specific + structures vary a bit between <literal>COPY FROM</literal> and + <literal>COPY TO</literal> so the next two sections describes each + in detail. + </para> + + <sect1 id="copy-handler-from"> + <title>Copy From Handler</title> + + <para> + The opening to this chapter describes how the executor will call the main + handler function with, in this case, + a <type>boolean</type> <literal>true</literal>, and expect to receive a + <type>CopyFromRoutine *</type> <type>Datum</type>. This section describes + the components of the <type>CopyFromRoutine</type> struct. + </para> + + <para> +<programlisting> +void +CopyFromInFunc(CopyFromState cstate, + Oid atttypid, + FmgrInfo *finfo, + Oid *typioparam); +</programlisting> + + This sets input function information for the + given <literal>atttypid</literal> attribute. This function is called once + at the beginning of <literal>COPY FROM</literal>. If + this <literal>COPY</literal> handler doesn't use any input functions, this + function doesn't need to do anything. + + <variablelist> + <varlistentry> + <term><literal>CopyFromState *cstate</literal></term> + <listitem> + <para> + This is an internal struct that contains all the state variables used + throughout a <literal>COPY FROM</literal> operation. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>Oid atttypid</literal></term> + <listitem> + <para> + This is the OID of data type used by the relation's attribute. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>FmgrInfo *finfo</literal></term> + <listitem> + <para> + This can be optionally filled to provide the catalog information of + the input function. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>Oid *typioparam</literal></term> + <listitem> + <para> + This can be optionally filled to define the OID of the type to + pass to the input function. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> +<programlisting> +void +CopyFromStart(CopyFromState cstate, + TupleDesc tupDesc); +</programlisting> + + This starts a <literal>COPY FROM</literal>. This function is called once at + the beginning of <literal>COPY FROM</literal>. + + <variablelist> + <varlistentry> + <term><literal>CopyFromState *cstate</literal></term> + <listitem> + <para> + This is an internal struct that contains all the state variables used + throughout a <literal>COPY FROM</literal> operation. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>TupleDesc tupDesc</literal></term> + <listitem> + <para> + This is the tuple descriptor of the relation where the data needs to be + copied. This can be used for any initialization steps required by a + format. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> +<programlisting> +bool +CopyFromOneRow(CopyFromState cstate, + ExprContext *econtext, + Datum *values, + bool *nulls); +</programlisting> + + This reads one row from the source and fill <literal>values</literal> + and <literal>nulls</literal>. If there is one or more tuples to be read, + this must return <literal>true</literal>. If there are no more tuples to + read, this must return <literal>false</literal>. + + <variablelist> + <varlistentry> + <term><literal>CopyFromState *cstate</literal></term> + <listitem> + <para> + This is an internal struct that contains all the state variables used + throughout a <literal>COPY FROM</literal> operation. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>ExprContext *econtext</literal></term> + <listitem> + <para> + This is used to evaluate default expression for each column that is + either not read from the file or is using + the <literal>DEFAULT</literal> option of <literal>COPY + FROM</literal>. It is <literal>NULL</literal> if no default values are + used. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>Datum *values</literal></term> + <listitem> + <para> + This is an output variable to store read tuples. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>bool *nulls</literal></term> + <listitem> + <para> + This is an output variable to store whether the read columns + are <literal>NULL</literal> or not. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> +<programlisting> +void +CopyFromEnd(CopyFromState cstate); +</programlisting> + + This ends a <literal>COPY FROM</literal>. This function is called once at + the end of <literal>COPY FROM</literal>. + + <variablelist> + <varlistentry> + <term><literal>CopyFromState *cstate</literal></term> + <listitem> + <para> + This is an internal struct that contains all the state variables used + throughout a <literal>COPY FROM</literal> operation. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + TODO: Add CopyFromStateGetData() and CopyFromSkipErrowRow()? + </para> + </sect1> + + <sect1 id="copy-handler-to"> + <title>Copy To Handler</title> + + <para> + The <literal>COPY</literal> handler function for <literal>COPY + TO</literal> returns a <type>CopyToRoutine</type> struct containing + pointers to the functions described below. All functions are required. + </para> + + <para> +<programlisting> +void +CopyToOutFunc(CopyToState cstate, + Oid atttypid, + FmgrInfo *finfo); +</programlisting> + + This sets output function information for the + given <literal>atttypid</literal> attribute. This function is called once + at the beginning of <literal>COPY TO</literal>. If + this <literal>COPY</literal> handler doesn't use any output functions, this + function doesn't need to do anything. + + <variablelist> + <varlistentry> + <term><literal>CopyToState *cstate</literal></term> + <listitem> + <para> + This is an internal struct that contains all the state variables used + throughout a <literal>COPY TO</literal> operation. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>Oid atttypid</literal></term> + <listitem> + <para> + This is the OID of data type used by the relation's attribute. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>FmgrInfo *finfo</literal></term> + <listitem> + <para> + This can be optionally filled to provide the catalog information of + the output function. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> +<programlisting> +void +CopyToStart(CopyToState cstate, + TupleDesc tupDesc); +</programlisting> + + This starts a <literal>COPY TO</literal>. This function is called once at + the beginning of <literal>COPY TO</literal>. + + <variablelist> + <varlistentry> + <term><literal>CopyToState *cstate</literal></term> + <listitem> + <para> + This is an internal struct that contains all the state variables used + throughout a <literal>COPY TO</literal> operation. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>TupleDesc tupDesc</literal></term> + <listitem> + <para> + This is the tuple descriptor of the relation where the data is read. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> +<programlisting> +bool +CopyToOneRow(CopyToState cstate, + TupleTableSlot *slot); +</programlisting> + + This writes one row stored in <literal>slot</literal> to the destination. + + <variablelist> + <varlistentry> + <term><literal>CopyToState *cstate</literal></term> + <listitem> + <para> + This is an internal struct that contains all the state variables used + throughout a <literal>COPY TO</literal> operation. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term><literal>TupleTableSlot *slot</literal></term> + <listitem> + <para> + This is used to get row to be written. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> +<programlisting> +void +CopyToEnd(CopyToState cstate); +</programlisting> + + This ends a <literal>COPY TO</literal>. This function is called once at + the end of <literal>COPY TO</literal>. + + <variablelist> + <varlistentry> + <term><literal>CopyToState *cstate</literal></term> + <listitem> + <para> + This is an internal struct that contains all the state variables used + throughout a <literal>COPY TO</literal> operation. + </para> + </listitem> + </varlistentry> + </variablelist> + </para> + + <para> + TODO: Add CopyToStateFlush()? + </para> + </sect1> +</chapter> diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml index fef9584f908..700cf22b502 100644 --- a/doc/src/sgml/filelist.sgml +++ b/doc/src/sgml/filelist.sgml @@ -107,6 +107,7 @@ <!ENTITY storage SYSTEM "storage.sgml"> <!ENTITY transaction SYSTEM "xact.sgml"> <!ENTITY tablesample-method SYSTEM "tablesample-method.sgml"> +<!ENTITY copy-handler SYSTEM "copy-handler.sgml"> <!ENTITY wal-for-extensions SYSTEM "wal-for-extensions.sgml"> <!ENTITY generic-wal SYSTEM "generic-wal.sgml"> <!ENTITY custom-rmgr SYSTEM "custom-rmgr.sgml"> diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml index af476c82fcc..8ba319ae2df 100644 --- a/doc/src/sgml/postgres.sgml +++ b/doc/src/sgml/postgres.sgml @@ -254,6 +254,7 @@ break is not needed in a wider output rendering. &plhandler; &fdwhandler; &tablesample-method; + ©-handler; &custom-scan; &geqo; &tableam; diff --git a/src/include/commands/copyapi.h b/src/include/commands/copyapi.h index 500ece7d5bb..24710cb667a 100644 --- a/src/include/commands/copyapi.h +++ b/src/include/commands/copyapi.h @@ -28,10 +28,10 @@ typedef struct CopyToRoutine * Set output function information. This callback is called once at the * beginning of COPY TO. * + * 'atttypid' is the OID of data type used by the relation's attribute. + * * 'finfo' can be optionally filled to provide the catalog information of * the output function. - * - * 'atttypid' is the OID of data type used by the relation's attribute. */ void (*CopyToOutFunc) (CopyToState cstate, Oid atttypid, FmgrInfo *finfo); @@ -70,12 +70,13 @@ typedef struct CopyFromRoutine * Set input function information. This callback is called once at the * beginning of COPY FROM. * + * 'atttypid' is the OID of data type used by the relation's attribute. + * * 'finfo' can be optionally filled to provide the catalog information of * the input function. * * 'typioparam' can be optionally filled to define the OID of the type to - * pass to the input function.'atttypid' is the OID of data type used by - * the relation's attribute. + * pass to the input function. */ void (*CopyFromInFunc) (CopyFromState cstate, Oid atttypid, FmgrInfo *finfo, Oid *typioparam); -- 2.47.2 ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> @ 2025-05-01 19:15 ` Masahiko Sawada <[email protected]> 2025-05-01 23:03 ` Re: Make COPY format extendable: Extract COPY TO format implementations Michael Paquier <[email protected]> 2025-05-03 02:19 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 05:36 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 0 siblings, 3 replies; 41+ messages in thread From: Masahiko Sawada @ 2025-05-01 19:15 UTC (permalink / raw) To: Sutou Kouhei <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers On Fri, Apr 25, 2025 at 5:45 AM Sutou Kouhei <[email protected]> wrote: > > Hi, > > I've updated the patch set. See the attached v40 patch set. > > In <CAD21AoAXzwPC7jjPMTcT80hnzmPa2SUJkiqdYHweEY8sZscEMA@mail.gmail.com> > "Re: Make COPY format extendable: Extract COPY TO format implementations" on Wed, 23 Apr 2025 23:44:55 -0700, > Masahiko Sawada <[email protected]> wrote: > > >> Are the followings correct? > >> > >> 1. Move invalid input patterns in > >> src/test/modules/test_copy_format/sql/invalid.sql to > >> src/test/regress/sql/copy.sql as much as possible. > >> 2. Create > >> src/test/modules/test_copy_format/sql/test_copy_format.sql > >> and move all contents in existing *.sql to the file. > >> 3. Add comments what the tests expect to > >> src/test/modules/test_copy_format/sql/test_copy_format.sql. > >> 4. Remove CopyFormatOptions::{binary,csv_mode}. > > > > Agreed with the above items. > > Done except 1. because 1. is removed by 3. in the following > list: > > ---- > >> There are 3 unconfirmed suggested changes for tests in: > >> https://www.postgresql.org/message-id/20250330.113126.433742864258096312.kou%40clear-code.com > >> > >> Here are my opinions for them: > >> > >> > 1.: There is no difference between single-quoting and > >> > double-quoting here. Because the information what quote > >> > was used for the given FORMAT value isn't remained > >> > here. Should we update gram.y? > >> > > >> > 2.: I don't have a strong opinion for it. If nobody objects > >> > it, I'll remove them. > >> > > >> > 3.: I don't have a strong opinion for it. If nobody objects > >> > it, I'll remove them. > ---- > > 0005 is added for 4. Could you squash 0004 ("Use copy > handler for bult-in formats") and 0005 ("Remove > CopyFormatOptions::{binary,csv_mode}") if needed when you > push? > > >> 6. Use handler OID for detecting the default built-in format > >> instead of comparing the given format as string. > > Done. > > >> 7. Update documentation. > > Could someone help this? 0007 is the draft commit for this. > > >> There are 3 unconfirmed suggested changes for tests in: > >> https://www.postgresql.org/message-id/20250330.113126.433742864258096312.kou%40clear-code.com > >> > >> Here are my opinions for them: > >> > >> > 1.: There is no difference between single-quoting and > >> > double-quoting here. Because the information what quote > >> > was used for the given FORMAT value isn't remained > >> > here. Should we update gram.y? > >> > > >> > 2.: I don't have a strong opinion for it. If nobody objects > >> > it, I'll remove them. > >> > > >> > 3.: I don't have a strong opinion for it. If nobody objects > >> > it, I'll remove them. > >> > >> Is the 1. required for "ready for merge"? If so, is there > >> any suggestion? I don't have a strong opinion for it. > >> > >> If there are no more opinions for 2. and 3., I'll remove > >> them. > > > > Agreed. > > 1.: I didn't do anything. Because there is no suggestion. > > 2., 3.: Done. Thank you for updating the patches. One of the primary considerations we need to address is the treatment of the specified format name. The current patch set utilizes built-in formats (namely 'csv', 'text', and 'binary') when the format name is either unqualified or explicitly specified with 'pg_catalog' as the schema. In all other cases, we search for custom format handler functions based on the search_path. To be frank, I have reservations about this interface design, as the dependence of the specified custom format name on the search_path could potentially confuse users. In light of these concerns, I've been contemplating alternative interface designs. One promising approach would involve registering custom copy formats via a C function during module loading (specifically, in _PG_init()). This method would require extension authors to invoke a registration function, say RegisterCustomCopyFormat(), in _PG_init() as follows: JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines", &JsonLinesCopyToRoutine, &JsonLinesCopyFromRoutine); The registration function would validate the format name and store it in TopMemoryContext. It would then return a unique identifier that can be used subsequently to reference the custom copy format extension. Custom copy format modules could be loaded through shared_preload_libraries, session_preload_libraries, or the LOAD command. Extensions could register their own options within this framework, for example: RegisterCustomCopyFormatOption(JsonLinesFormatId, "custom_option", custom_option_handler); This approach offers several advantages: it would eliminate the search_path issue, provide greater flexibility, and potentially simplify the overall interface for users and developers alike. We might be able to provide a view showing the registered custom COPY format in the future. Also, these interfaces align with other customizable functionalities such as custom rmgr, custom lwlock, custom waitevent, and custom EXPLAIN option etc. Feedback is very welcome. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> @ 2025-05-01 23:03 ` Michael Paquier <[email protected]> 2025-05-02 22:52 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2 siblings, 1 reply; 41+ messages in thread From: Michael Paquier @ 2025-05-01 23:03 UTC (permalink / raw) To: Masahiko Sawada <[email protected]>; +Cc: Sutou Kouhei <[email protected]>; [email protected]; [email protected]; [email protected]; pgsql-hackers On Thu, May 01, 2025 at 12:15:30PM -0700, Masahiko Sawada wrote: > In light of these concerns, I've been contemplating alternative > interface designs. One promising approach would involve registering > custom copy formats via a C function during module loading > (specifically, in _PG_init()). This method would require extension > authors to invoke a registration function, say > RegisterCustomCopyFormat(), in _PG_init() as follows: > > JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines", > &JsonLinesCopyToRoutine, > &JsonLinesCopyFromRoutine); > > The registration function would validate the format name and store it > in TopMemoryContext. It would then return a unique identifier that can > be used subsequently to reference the custom copy format extension. Hmm. How much should we care about the observability of the COPY format used by a given backend? Storing this information in a backend's TopMemoryContext is OK to get the extensibility basics to work, but could it make sense to use some shmem state to allocate a uint32 ID that could be shared by all backends. Contrary to EXPLAIN, COPY commands usually run for a very long time, so I am wondering if these APIs should be designed so as it would be possible to monitor the format used. One layer where the format information could be made available is the progress reporting view for COPY, for example. I can also imagine a pgstats kind where we do COPY stats aggregates, with a per-format pgstats kind, and sharing a fixed ID across multiple backends is relevant (when flushing the stats at shutdown, we would use a name/ID mapping like replication slots). I don't think that this needs to be relevant for the option part, just for the format where, I suspect, we should store in a shmem array based on the ID allocated the name of the format, the library of the callback and the function name fed to load_external_function(). Note that custom LWLock and wait events use a shmem state for monitoring purposes, where we are able to do ID->format name lookups as much as format->ID lookups. Perhaps it's OK not to do that for COPY, but I am wondering if we'd better design things from scratch with states in shmem state knowing that COPY is a long-running operation, and that if one mixes multiple formats they would most likely want to know which formats are bottlenecks, through SQL. Cloud providers would love that. > This approach offers several advantages: it would eliminate the > search_path issue, provide greater flexibility, and potentially > simplify the overall interface for users and developers alike. We > might be able to provide a view showing the registered custom COPY > format in the future. Also, these interfaces align with other > customizable functionalities such as custom rmgr, custom lwlock, > custom waitevent, and custom EXPLAIN option etc. Yeah, agreed with the search_path concerns. We are getting better at making areas of Postgres more pluggable lately, having a loading path where we don't have any of these potential issues by design matters. -- Michael Attachments: [application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc) download ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-01 23:03 ` Re: Make COPY format extendable: Extract COPY TO format implementations Michael Paquier <[email protected]> @ 2025-05-02 22:52 ` Masahiko Sawada <[email protected]> 2025-05-03 02:24 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Masahiko Sawada @ 2025-05-02 22:52 UTC (permalink / raw) To: Michael Paquier <[email protected]>; +Cc: Sutou Kouhei <[email protected]>; [email protected]; [email protected]; [email protected]; pgsql-hackers On Thu, May 1, 2025 at 4:04 PM Michael Paquier <[email protected]> wrote: > > On Thu, May 01, 2025 at 12:15:30PM -0700, Masahiko Sawada wrote: > > In light of these concerns, I've been contemplating alternative > > interface designs. One promising approach would involve registering > > custom copy formats via a C function during module loading > > (specifically, in _PG_init()). This method would require extension > > authors to invoke a registration function, say > > RegisterCustomCopyFormat(), in _PG_init() as follows: > > > > JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines", > > &JsonLinesCopyToRoutine, > > &JsonLinesCopyFromRoutine); > > > > The registration function would validate the format name and store it > > in TopMemoryContext. It would then return a unique identifier that can > > be used subsequently to reference the custom copy format extension. > > Hmm. How much should we care about the observability of the COPY > format used by a given backend? Storing this information in a > backend's TopMemoryContext is OK to get the extensibility basics to > work, but could it make sense to use some shmem state to allocate a > uint32 ID that could be shared by all backends. Contrary to EXPLAIN, > COPY commands usually run for a very long time, so I am wondering if > these APIs should be designed so as it would be possible to monitor > the format used. One layer where the format information could be made > available is the progress reporting view for COPY, for example. I can > also imagine a pgstats kind where we do COPY stats aggregates, with a > per-format pgstats kind, and sharing a fixed ID across multiple > backends is relevant (when flushing the stats at shutdown, we would > use a name/ID mapping like replication slots). > > I don't think that this needs to be relevant for the option part, just > for the format where, I suspect, we should store in a shmem array > based on the ID allocated the name of the format, the library of the > callback and the function name fed to load_external_function(). > > Note that custom LWLock and wait events use a shmem state for > monitoring purposes, where we are able to do ID->format name lookups > as much as format->ID lookups. Perhaps it's OK not to do that for > COPY, but I am wondering if we'd better design things from scratch > with states in shmem state knowing that COPY is a long-running > operation, and that if one mixes multiple formats they would most > likely want to know which formats are bottlenecks, through SQL. Cloud > providers would love that. Good point. It would make sense to have such information as a map on shmem. It might be better to use dshash here since a custom copy format module can be loaded at runtime. Or we can use dynahash with large enough elements. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-01 23:03 ` Re: Make COPY format extendable: Extract COPY TO format implementations Michael Paquier <[email protected]> 2025-05-02 22:52 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> @ 2025-05-03 02:24 ` Sutou Kouhei <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Sutou Kouhei @ 2025-05-03 02:24 UTC (permalink / raw) To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers Hi, In <CAD21AoB82+MoP_RJ=zzhO9KaHK4LbfGjORkre34C7g-xsCdegQ@mail.gmail.com> "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 15:52:49 -0700, Masahiko Sawada <[email protected]> wrote: >> Hmm. How much should we care about the observability of the COPY >> format used by a given backend? Storing this information in a >> backend's TopMemoryContext is OK to get the extensibility basics to >> work, but could it make sense to use some shmem state to allocate a >> uint32 ID that could be shared by all backends. Contrary to EXPLAIN, >> COPY commands usually run for a very long time, so I am wondering if >> these APIs should be designed so as it would be possible to monitor >> the format used. One layer where the format information could be made >> available is the progress reporting view for COPY, for example. I can >> also imagine a pgstats kind where we do COPY stats aggregates, with a >> per-format pgstats kind, and sharing a fixed ID across multiple >> backends is relevant (when flushing the stats at shutdown, we would >> use a name/ID mapping like replication slots). >> >> I don't think that this needs to be relevant for the option part, just >> for the format where, I suspect, we should store in a shmem array >> based on the ID allocated the name of the format, the library of the >> callback and the function name fed to load_external_function(). >> >> Note that custom LWLock and wait events use a shmem state for >> monitoring purposes, where we are able to do ID->format name lookups >> as much as format->ID lookups. Perhaps it's OK not to do that for >> COPY, but I am wondering if we'd better design things from scratch >> with states in shmem state knowing that COPY is a long-running >> operation, and that if one mixes multiple formats they would most >> likely want to know which formats are bottlenecks, through SQL. Cloud >> providers would love that. > > Good point. It would make sense to have such information as a map on > shmem. It might be better to use dshash here since a custom copy > format module can be loaded at runtime. Or we can use dynahash with > large enough elements. If we don't need to assign an ID for each format, can we avoid it? If we implement it, is this approach more complex than the current table sampling method like approach? Thanks, -- kou ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> @ 2025-05-03 02:19 ` Sutou Kouhei <[email protected]> 2025-05-03 04:38 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2 siblings, 1 reply; 41+ messages in thread From: Sutou Kouhei @ 2025-05-03 02:19 UTC (permalink / raw) To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers Hi, In <CAD21AoBuEqcz2_+dpA3WTiDUF=FgudPBKwM+nvH+qHT-k4p5mA@mail.gmail.com> "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 1 May 2025 12:15:30 -0700, Masahiko Sawada <[email protected]> wrote: > One of the primary considerations we need to address is the treatment > of the specified format name. The current patch set utilizes built-in > formats (namely 'csv', 'text', and 'binary') when the format name is > either unqualified or explicitly specified with 'pg_catalog' as the > schema. In all other cases, we search for custom format handler > functions based on the search_path. To be frank, I have reservations > about this interface design, as the dependence of the specified custom > format name on the search_path could potentially confuse users. How about requiring schema for all custom formats? Valid: COPY ... TO ... (FORMAT 'text'); COPY ... TO ... (FORMAT 'my_schema.jsonlines'); Invalid: COPY ... TO ... (FORMAT 'jsonlines'); -- no schema COPY ... TO ... (FORMAT 'pg_catalog.text'); -- needless schema If we require "schema" for all custom formats, we don't need to depend on search_path. > In light of these concerns, I've been contemplating alternative > interface designs. One promising approach would involve registering > custom copy formats via a C function during module loading > (specifically, in _PG_init()). This method would require extension > authors to invoke a registration function, say > RegisterCustomCopyFormat(), in _PG_init() as follows: > > JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines", > &JsonLinesCopyToRoutine, > &JsonLinesCopyFromRoutine); > > The registration function would validate the format name and store it > in TopMemoryContext. It would then return a unique identifier that can > be used subsequently to reference the custom copy format extension. I don't object the suggested interface because I don't have a strong opinion how to implement this feature. Why do we need to assign a unique ID? For performance? For RegisterCustomCopyFormatOption()? I think that we don't need to use it so much in COPY. We don't need to use format name and assigned ID after we retrieve a corresponding Copy{To,From}Routine. Because all needed information are in Copy{To,From}Routine. > Extensions could register their own options within this > framework, for example: > > RegisterCustomCopyFormatOption(JsonLinesFormatId, > "custom_option", > custom_option_handler); Can we defer to discuss how to add support for custom options while we focus on the first implementation? Earlier patch sets with the current approach had custom options support but it's removed in the first implementation. (BTW, I think that it's not a good API because we want COPY FROM only options and COPY TO only options something like "compression level".) > This approach offers several advantages: it would eliminate the > search_path issue, provide greater flexibility, and potentially > simplify the overall interface for users and developers alike. What contributes to the "flexibility"? Developers can call multiple Register* functions in _PG_Init(), right? Thanks, -- kou ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 02:19 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> @ 2025-05-03 04:38 ` Masahiko Sawada <[email protected]> 2025-05-03 04:56 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Masahiko Sawada @ 2025-05-03 04:38 UTC (permalink / raw) To: Sutou Kouhei <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers On Fri, May 2, 2025 at 7:20 PM Sutou Kouhei <[email protected]> wrote: > > Hi, > > In <CAD21AoBuEqcz2_+dpA3WTiDUF=FgudPBKwM+nvH+qHT-k4p5mA@mail.gmail.com> > "Re: Make COPY format extendable: Extract COPY TO format implementations" on Thu, 1 May 2025 12:15:30 -0700, > Masahiko Sawada <[email protected]> wrote: > > > One of the primary considerations we need to address is the treatment > > of the specified format name. The current patch set utilizes built-in > > formats (namely 'csv', 'text', and 'binary') when the format name is > > either unqualified or explicitly specified with 'pg_catalog' as the > > schema. In all other cases, we search for custom format handler > > functions based on the search_path. To be frank, I have reservations > > about this interface design, as the dependence of the specified custom > > format name on the search_path could potentially confuse users. > > How about requiring schema for all custom formats? > > Valid: > > COPY ... TO ... (FORMAT 'text'); > COPY ... TO ... (FORMAT 'my_schema.jsonlines'); > > Invalid: > > COPY ... TO ... (FORMAT 'jsonlines'); -- no schema > COPY ... TO ... (FORMAT 'pg_catalog.text'); -- needless schema > > If we require "schema" for all custom formats, we don't need > to depend on search_path. I'm concerned that users cannot use the same format name in the FORMAT option depending on which schema the handler function is created. > > > In light of these concerns, I've been contemplating alternative > > interface designs. One promising approach would involve registering > > custom copy formats via a C function during module loading > > (specifically, in _PG_init()). This method would require extension > > authors to invoke a registration function, say > > RegisterCustomCopyFormat(), in _PG_init() as follows: > > > > JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines", > > &JsonLinesCopyToRoutine, > > &JsonLinesCopyFromRoutine); > > > > The registration function would validate the format name and store it > > in TopMemoryContext. It would then return a unique identifier that can > > be used subsequently to reference the custom copy format extension. > > I don't object the suggested interface because I don't have > a strong opinion how to implement this feature. > > Why do we need to assign a unique ID? For performance? For > RegisterCustomCopyFormatOption()? I think it's required for monitoring purposes for example. For instance, we can set the format ID in the progress information and the progress view can fetch the format name by the ID so that users can see what format is being used in the COPY command. > > > Extensions could register their own options within this > > framework, for example: > > > > RegisterCustomCopyFormatOption(JsonLinesFormatId, > > "custom_option", > > custom_option_handler); > > Can we defer to discuss how to add support for custom > options while we focus on the first implementation? Earlier > patch sets with the current approach had custom options > support but it's removed in the first implementation. I think we can skip the custom option patch for the first implementation but still need to discuss how we will be able to implement it to understand the big picture of this feature. Otherwise we could end up going the wrong direction. > > (BTW, I think that it's not a good API because we want COPY > FROM only options and COPY TO only options something like > "compression level".) Why does this matter in terms of API? I think that even with this API we can pass is_from to the option handler function so that it validates the option based on it. > > > This approach offers several advantages: it would eliminate the > > search_path issue, provide greater flexibility, and potentially > > simplify the overall interface for users and developers alike. > > What contributes to the "flexibility"? Developers can call > multiple Register* functions in _PG_Init(), right? I think that with a tablesample-like approach we need to do everything based on one handler function and callbacks returned from it whereas there is no such limitation with C API style. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 02:19 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 04:38 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> @ 2025-05-03 04:56 ` Sutou Kouhei <[email protected]> 2025-05-03 06:02 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Sutou Kouhei @ 2025-05-03 04:56 UTC (permalink / raw) To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers Hi, In <CAD21AoBGRFStdVbHUcxL0QB8wn92J3Sn-6x=RhsSMuhepRH0NQ@mail.gmail.com> "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 21:38:32 -0700, Masahiko Sawada <[email protected]> wrote: >> How about requiring schema for all custom formats? >> >> Valid: >> >> COPY ... TO ... (FORMAT 'text'); >> COPY ... TO ... (FORMAT 'my_schema.jsonlines'); >> >> Invalid: >> >> COPY ... TO ... (FORMAT 'jsonlines'); -- no schema >> COPY ... TO ... (FORMAT 'pg_catalog.text'); -- needless schema >> >> If we require "schema" for all custom formats, we don't need >> to depend on search_path. > > I'm concerned that users cannot use the same format name in the FORMAT > option depending on which schema the handler function is created. I'm not sure that it's a problem or not. If users want to use the same format name, they can install the handler function to the same schema. >> Why do we need to assign a unique ID? For performance? For >> RegisterCustomCopyFormatOption()? > > I think it's required for monitoring purposes for example. For > instance, we can set the format ID in the progress information and the > progress view can fetch the format name by the ID so that users can > see what format is being used in the COPY command. How about setting the format name instead of the format ID in the progress information? > I think we can skip the custom option patch for the first > implementation but still need to discuss how we will be able to > implement it to understand the big picture of this feature. Otherwise > we could end up going the wrong direction. I think that we don't need to discuss it deeply because we have many options with this approach. We can call C functions in _PG_Init(). I think that this feature will not be a blocker of this approach. >> (BTW, I think that it's not a good API because we want COPY >> FROM only options and COPY TO only options something like >> "compression level".) > > Why does this matter in terms of API? I think that even with this API > we can pass is_from to the option handler function so that it > validates the option based on it. If we choose the API, each custom format developer needs to handle the case in handler function. For example, if we pass information whether this option is only for TO to PostgreSQL, ProcessCopyOptions() not handler functions can handle it. Anyway, I think that we don't need to discuss this deeply for now. >> What contributes to the "flexibility"? Developers can call >> multiple Register* functions in _PG_Init(), right? > > I think that with a tablesample-like approach we need to do everything > based on one handler function and callbacks returned from it whereas > there is no such limitation with C API style. Thanks for clarifying it. It seems that my understanding is correct. I hope that the flexibility is needed flexibility and too much flexibility doesn't introduce too much complexity. Thanks, -- kou ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 02:19 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 04:38 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 04:56 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> @ 2025-05-03 06:02 ` Masahiko Sawada <[email protected]> 2025-05-03 06:20 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Masahiko Sawada @ 2025-05-03 06:02 UTC (permalink / raw) To: Sutou Kouhei <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers On Fri, May 2, 2025 at 9:56 PM Sutou Kouhei <[email protected]> wrote: > > Hi, > > In <CAD21AoBGRFStdVbHUcxL0QB8wn92J3Sn-6x=RhsSMuhepRH0NQ@mail.gmail.com> > "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 21:38:32 -0700, > Masahiko Sawada <[email protected]> wrote: > > >> How about requiring schema for all custom formats? > >> > >> Valid: > >> > >> COPY ... TO ... (FORMAT 'text'); > >> COPY ... TO ... (FORMAT 'my_schema.jsonlines'); > >> > >> Invalid: > >> > >> COPY ... TO ... (FORMAT 'jsonlines'); -- no schema > >> COPY ... TO ... (FORMAT 'pg_catalog.text'); -- needless schema > >> > >> If we require "schema" for all custom formats, we don't need > >> to depend on search_path. > > > > I'm concerned that users cannot use the same format name in the FORMAT > > option depending on which schema the handler function is created. > > I'm not sure that it's a problem or not. If users want to > use the same format name, they can install the handler > function to the same schema. > > >> Why do we need to assign a unique ID? For performance? For > >> RegisterCustomCopyFormatOption()? > > > > I think it's required for monitoring purposes for example. For > > instance, we can set the format ID in the progress information and the > > progress view can fetch the format name by the ID so that users can > > see what format is being used in the COPY command. > > How about setting the format name instead of the format ID > in the progress information? The progress view can know only numbers. We need to extend the progress view infrastructure so that we can pass other data types. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 02:19 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 04:38 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 04:56 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 06:02 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> @ 2025-05-03 06:20 ` Sutou Kouhei <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Sutou Kouhei @ 2025-05-03 06:20 UTC (permalink / raw) To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers Hi, In <CAD21AoDnY2fhC7tp7jpn24AuwkeW-0YjFEtZbEfPwg8YcH6bAw@mail.gmail.com> "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 23:02:25 -0700, Masahiko Sawada <[email protected]> wrote: > The progress view can know only numbers. We need to extend the > progress view infrastructure so that we can pass other data types. Sorry. Could you tell me what APIs referred here? pgstat_progress_*() functions in src/include/utils/backend_progress.h? Thanks, -- kou ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 02:19 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 04:38 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 04:56 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 06:02 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 06:20 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> @ 2025-05-03 06:37 ` Masahiko Sawada <[email protected]> 2025-05-09 08:51 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Masahiko Sawada @ 2025-05-03 06:37 UTC (permalink / raw) To: Sutou Kouhei <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers On Fri, May 2, 2025 at 11:20 PM Sutou Kouhei <[email protected]> wrote: > > Hi, > > In <CAD21AoDnY2fhC7tp7jpn24AuwkeW-0YjFEtZbEfPwg8YcH6bAw@mail.gmail.com> > "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 23:02:25 -0700, > Masahiko Sawada <[email protected]> wrote: > > > The progress view can know only numbers. We need to extend the > > progress view infrastructure so that we can pass other data types. > > Sorry. Could you tell me what APIs referred here? > pgstat_progress_*() functions in > src/include/utils/backend_progress.h? The progress information is stored in PgBackendStatus defined in backend_status.h: /* * Command progress reporting. Any command which wishes can advertise * that it is running by setting st_progress_command, * st_progress_command_target, and st_progress_param[]. * st_progress_command_target should be the OID of the relation which the * command targets (we assume there's just one, as this is meant for * utility commands), but the meaning of each element in the * st_progress_param array is command-specific. */ ProgressCommandType st_progress_command; Oid st_progress_command_target; int64 st_progress_param[PGSTAT_NUM_PROGRESS_PARAM]; Then the progress view maps the numbers to the corresponding strings: CREATE VIEW pg_stat_progress_copy AS SELECT S.pid AS pid, S.datid AS datid, D.datname AS datname, S.relid AS relid, CASE S.param5 WHEN 1 THEN 'COPY FROM' WHEN 2 THEN 'COPY TO' END AS command, CASE S.param6 WHEN 1 THEN 'FILE' WHEN 2 THEN 'PROGRAM' WHEN 3 THEN 'PIPE' WHEN 4 THEN 'CALLBACK' END AS "type", S.param1 AS bytes_processed, S.param2 AS bytes_total, S.param3 AS tuples_processed, S.param4 AS tuples_excluded, S.param7 AS tuples_skipped FROM pg_stat_get_progress_info('COPY') AS S LEFT JOIN pg_database D ON S.datid = D.oid; So the idea is that the backend process sets the format ID somewhere in st_progress_param, and then the progress view calls a SQL function, say pg_stat_get_copy_format_name(), with the format ID that returns the corresponding format name. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 02:19 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 04:38 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 04:56 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 06:02 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 06:20 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> @ 2025-05-09 08:51 ` Sutou Kouhei <[email protected]> 2025-05-10 04:29 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Sutou Kouhei @ 2025-05-09 08:51 UTC (permalink / raw) To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers Hi, In <CAD21AoD9CBjh4u6jdiE0tG-jvejw-GJN8fUPoQSVhKh36HW2NQ@mail.gmail.com> "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 23:37:46 -0700, Masahiko Sawada <[email protected]> wrote: > The progress information is stored in PgBackendStatus defined in > backend_status.h: > > /* > * Command progress reporting. Any command which wishes can advertise > * that it is running by setting st_progress_command, > * st_progress_command_target, and st_progress_param[]. > * st_progress_command_target should be the OID of the relation which the > * command targets (we assume there's just one, as this is meant for > * utility commands), but the meaning of each element in the > * st_progress_param array is command-specific. > */ > ProgressCommandType st_progress_command; > Oid st_progress_command_target; > int64 st_progress_param[PGSTAT_NUM_PROGRESS_PARAM]; > > Then the progress view maps the numbers to the corresponding strings: > > CREATE VIEW pg_stat_progress_copy AS > SELECT > S.pid AS pid, S.datid AS datid, D.datname AS datname, > S.relid AS relid, > CASE S.param5 WHEN 1 THEN 'COPY FROM' > WHEN 2 THEN 'COPY TO' > END AS command, > CASE S.param6 WHEN 1 THEN 'FILE' > WHEN 2 THEN 'PROGRAM' > WHEN 3 THEN 'PIPE' > WHEN 4 THEN 'CALLBACK' > END AS "type", > S.param1 AS bytes_processed, > S.param2 AS bytes_total, > S.param3 AS tuples_processed, > S.param4 AS tuples_excluded, > S.param7 AS tuples_skipped > FROM pg_stat_get_progress_info('COPY') AS S > LEFT JOIN pg_database D ON S.datid = D.oid; Thanks. I didn't know about how to implement pg_stat_progress_copy. > So the idea is that the backend process sets the format ID somewhere > in st_progress_param, and then the progress view calls a SQL function, > say pg_stat_get_copy_format_name(), with the format ID that returns > the corresponding format name. Does it work when we use session_preload_libraries or the LOAD command? If we have 2 sessions and both of them load "jsonlines" COPY FORMAT extensions, what will be happened? For example: 1. Session 1: Register "jsonlines" 2. Session 2: Register "jsonlines" (Should global format ID <-> format name mapping be updated?) 3. Session 2: Close this session. Unregister "jsonlines". (Can we unregister COPY FORMAT extension?) (Should global format ID <-> format name mapping be updated?) 4. Session 1: Close this session. Unregister "jsonlines". (Can we unregister COPY FORMAT extension?) (Should global format ID <-> format name mapping be updated?) Thanks, -- kou ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 02:19 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 04:38 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 04:56 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 06:02 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 06:20 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-09 08:51 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> @ 2025-05-10 04:29 ` Masahiko Sawada <[email protected]> 2025-05-26 01:27 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Masahiko Sawada @ 2025-05-10 04:29 UTC (permalink / raw) To: Sutou Kouhei <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers On Fri, May 9, 2025 at 1:51 AM Sutou Kouhei <[email protected]> wrote: > > Hi, > > In <CAD21AoD9CBjh4u6jdiE0tG-jvejw-GJN8fUPoQSVhKh36HW2NQ@mail.gmail.com> > "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 2 May 2025 23:37:46 -0700, > Masahiko Sawada <[email protected]> wrote: > > > The progress information is stored in PgBackendStatus defined in > > backend_status.h: > > > > /* > > * Command progress reporting. Any command which wishes can advertise > > * that it is running by setting st_progress_command, > > * st_progress_command_target, and st_progress_param[]. > > * st_progress_command_target should be the OID of the relation which the > > * command targets (we assume there's just one, as this is meant for > > * utility commands), but the meaning of each element in the > > * st_progress_param array is command-specific. > > */ > > ProgressCommandType st_progress_command; > > Oid st_progress_command_target; > > int64 st_progress_param[PGSTAT_NUM_PROGRESS_PARAM]; > > > > Then the progress view maps the numbers to the corresponding strings: > > > > CREATE VIEW pg_stat_progress_copy AS > > SELECT > > S.pid AS pid, S.datid AS datid, D.datname AS datname, > > S.relid AS relid, > > CASE S.param5 WHEN 1 THEN 'COPY FROM' > > WHEN 2 THEN 'COPY TO' > > END AS command, > > CASE S.param6 WHEN 1 THEN 'FILE' > > WHEN 2 THEN 'PROGRAM' > > WHEN 3 THEN 'PIPE' > > WHEN 4 THEN 'CALLBACK' > > END AS "type", > > S.param1 AS bytes_processed, > > S.param2 AS bytes_total, > > S.param3 AS tuples_processed, > > S.param4 AS tuples_excluded, > > S.param7 AS tuples_skipped > > FROM pg_stat_get_progress_info('COPY') AS S > > LEFT JOIN pg_database D ON S.datid = D.oid; > > Thanks. I didn't know about how to implement > pg_stat_progress_copy. > > > So the idea is that the backend process sets the format ID somewhere > > in st_progress_param, and then the progress view calls a SQL function, > > say pg_stat_get_copy_format_name(), with the format ID that returns > > the corresponding format name. > > Does it work when we use session_preload_libraries or the > LOAD command? If we have 2 sessions and both of them load > "jsonlines" COPY FORMAT extensions, what will be happened? > > For example: > > 1. Session 1: Register "jsonlines" > 2. Session 2: Register "jsonlines" > (Should global format ID <-> format name mapping > be updated?) > 3. Session 2: Close this session. > Unregister "jsonlines". > (Can we unregister COPY FORMAT extension?) > (Should global format ID <-> format name mapping > be updated?) > 4. Session 1: Close this session. > Unregister "jsonlines". > (Can we unregister COPY FORMAT extension?) > (Should global format ID <-> format name mapping > be updated?) I imagine that only for progress reporting purposes, I think session 1 and 2 can have different format IDs for the same 'jsonlines' if they load it by LOAD command. They can advertise the format IDs on the shmem and we can also provide a SQL function for the progress view that can get the format name by the format ID. Considering the possibility that we might want to use the format ID also in the cumulative statistics, we might want to strictly provide the unique format ID for each custom format as the format IDs are serialized to the pgstat file. One possible way to implement it is that we manage the custom format IDs in a wiki page like we do for custom cumulative statistics and custom RMGR[1][2]. That is, a custom format extension registers the format name along with the format ID that is pre-registered in the wiki page or the format ID (e.g. 128) indicating under development. If either the format name or format ID conflict with an already registered custom format extension, the registration function raises an error. And we preallocate enough format IDs for built-in formats. As for unregistration, I think that even if we provide an unregisteration API, it ultimately depends on whether or not custom format extensions call it in _PG_fini(). Regards, [1] https://wiki.postgresql.org/wiki/CustomCumulativeStats [2] https://wiki.postgresql.org/wiki/CustomWALResourceManagers Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 02:19 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 04:38 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 04:56 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 06:02 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 06:20 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-09 08:51 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-10 04:29 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> @ 2025-05-26 01:27 ` Sutou Kouhei <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Sutou Kouhei @ 2025-05-26 01:27 UTC (permalink / raw) To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers Hi, In <CAD21AoAY_h-9nuhs14e3cyO_A2rH7==zuq+NPHkn9ggwyaXnPQ@mail.gmail.com> "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 9 May 2025 21:29:23 -0700, Masahiko Sawada <[email protected]> wrote: >> > So the idea is that the backend process sets the format ID somewhere >> > in st_progress_param, and then the progress view calls a SQL function, >> > say pg_stat_get_copy_format_name(), with the format ID that returns >> > the corresponding format name. >> >> Does it work when we use session_preload_libraries or the >> LOAD command? If we have 2 sessions and both of them load >> "jsonlines" COPY FORMAT extensions, what will be happened? >> >> For example: >> >> 1. Session 1: Register "jsonlines" >> 2. Session 2: Register "jsonlines" >> (Should global format ID <-> format name mapping >> be updated?) >> 3. Session 2: Close this session. >> Unregister "jsonlines". >> (Can we unregister COPY FORMAT extension?) >> (Should global format ID <-> format name mapping >> be updated?) >> 4. Session 1: Close this session. >> Unregister "jsonlines". >> (Can we unregister COPY FORMAT extension?) >> (Should global format ID <-> format name mapping >> be updated?) > > I imagine that only for progress reporting purposes, I think session 1 > and 2 can have different format IDs for the same 'jsonlines' if they > load it by LOAD command. They can advertise the format IDs on the > shmem and we can also provide a SQL function for the progress view > that can get the format name by the format ID. > > Considering the possibility that we might want to use the format ID > also in the cumulative statistics, we might want to strictly provide > the unique format ID for each custom format as the format IDs are > serialized to the pgstat file. One possible way to implement it is > that we manage the custom format IDs in a wiki page like we do for > custom cumulative statistics and custom RMGR[1][2]. That is, a custom > format extension registers the format name along with the format ID > that is pre-registered in the wiki page or the format ID (e.g. 128) > indicating under development. If either the format name or format ID > conflict with an already registered custom format extension, the > registration function raises an error. And we preallocate enough > format IDs for built-in formats. > > As for unregistration, I think that even if we provide an > unregisteration API, it ultimately depends on whether or not custom > format extensions call it in _PG_fini(). Thanks for sharing your idea. With the former ID issuing approach, it seems that we need a global format ID <-> name mapping and a per session registered format name list. The custom COPY FORMAT register function rejects the same format name, right? If we support both of shared_preload_libraries and session_preload_libraries/LOAD, we have different life time custom formats. It may introduce a complexity with the ID issuing approach. With the latter static ID approach, how to implement a function that converts format ID to format name? PostgreSQL itself doesn't know ID <-> name mapping in the Wiki page. It seems that custom COPY FORMAT implementation needs to register its name to PostgreSQL by itself. Thanks, -- kou ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> @ 2025-05-03 05:36 ` David G. Johnston <[email protected]> 2025-05-03 05:57 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2 siblings, 1 reply; 41+ messages in thread From: David G. Johnston @ 2025-05-03 05:36 UTC (permalink / raw) To: Masahiko Sawada <[email protected]>; +Cc: Sutou Kouhei <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers On Thursday, May 1, 2025, Masahiko Sawada <[email protected]> wrote: > > In light of these concerns, I've been contemplating alternative > interface designs. One promising approach would involve registering > custom copy formats via a C function during module loading > (specifically, in _PG_init()). This method would require extension > authors to invoke a registration function, say > RegisterCustomCopyFormat(), in _PG_init() as follows: > > JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines", > &JsonLinesCopyToRoutine, > &JsonLinesCopyFromRoutine); > > The registration function would validate the format name and store it > in TopMemoryContext. It would then return a unique identifier that can > be used subsequently to reference the custom copy format extension. > How does this fix the search_path concern? Are query writers supposed to put JsonLinesFormatId into their queries? Or are you just prohibiting a DBA from ever installing an extension that wants to register a format name that is already registered so that no namespace is ever required? ISTM accommodating a namespace for formats is required just like we do for virtually every other named object in the system. At least, if we want to play nice with extension authors. It doesn’t have to be within the existing pg_proc scope, we can create a new scope if desired, but abolishing it seems unwise. It would be more consistent with established policy if we didn’t make exceptions for text/csv/binary - if the DBA permits a text format to exist in a different schema and that schema appears first in the search_path, unqualified references to text would resolve to the non-core handler. We already protect ourselves with safe search_paths. This is really no different than if someone wanted to implement a now() function and people are putting pg_catalog from of existing usage. It’s the DBAs problem, not ours. David J. ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 05:36 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> @ 2025-05-03 05:57 ` Masahiko Sawada <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Masahiko Sawada @ 2025-05-03 05:57 UTC (permalink / raw) To: David G. Johnston <[email protected]>; +Cc: Sutou Kouhei <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers On Fri, May 2, 2025 at 10:36 PM David G. Johnston <[email protected]> wrote: > > On Thursday, May 1, 2025, Masahiko Sawada <[email protected]> wrote: >> >> >> In light of these concerns, I've been contemplating alternative >> interface designs. One promising approach would involve registering >> custom copy formats via a C function during module loading >> (specifically, in _PG_init()). This method would require extension >> authors to invoke a registration function, say >> RegisterCustomCopyFormat(), in _PG_init() as follows: >> >> JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines", >> &JsonLinesCopyToRoutine, >> &JsonLinesCopyFromRoutine); >> >> The registration function would validate the format name and store it >> in TopMemoryContext. It would then return a unique identifier that can >> be used subsequently to reference the custom copy format extension. > > > How does this fix the search_path concern? Are query writers supposed to put JsonLinesFormatId into their queries? Or are you just prohibiting a DBA from ever installing an extension that wants to register a format name that is already registered so that no namespace is ever required? Users can specify "jsonlines", passed in the first argument to the register function, to the COPY FORMAT option in this case. While JsonLinesFormatId is reserved for internal operations such as module processing and monitoring, any attempt to load another custom COPY format module named 'jsonlines' will result in an error. > ISTM accommodating a namespace for formats is required just like we do for virtually every other named object in the system. At least, if we want to play nice with extension authors. It doesn’t have to be within the existing pg_proc scope, we can create a new scope if desired, but abolishing it seems unwise. > > It would be more consistent with established policy if we didn’t make exceptions for text/csv/binary - if the DBA permits a text format to exist in a different schema and that schema appears first in the search_path, unqualified references to text would resolve to the non-core handler. We already protect ourselves with safe search_paths. This is really no different than if someone wanted to implement a now() function and people are putting pg_catalog from of existing usage. It’s the DBAs problem, not ours. I'm concerned about allowing multiple 'text' format implementations with identical names within the database, as this could lead to considerable confusion. When users specify 'text', it would be more logical to guarantee that the built-in 'text' format is consistently used. This principle aligns with other customizable components, such as custom resource managers, wait events, lightweight locks, and custom scans. These components maintain their built-in data/types and explicitly prevent the registration of duplicate names. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 05:36 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 05:57 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> @ 2025-05-03 06:37 ` David G. Johnston <[email protected]> 2025-05-03 07:54 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: David G. Johnston @ 2025-05-03 06:37 UTC (permalink / raw) To: Masahiko Sawada <[email protected]>; +Cc: Sutou Kouhei <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers On Friday, May 2, 2025, Masahiko Sawada <[email protected]> wrote: > > I'm concerned about allowing multiple 'text' format implementations > with identical names within the database, as this could lead to > considerable confusion. When users specify 'text', it would be more > logical to guarantee that the built-in 'text' format is consistently > used. Do you want to only give text/csv/binary this special treatment or also any future format name we ever decide to implement in core. If an extension takes up “xml” and we try to do that in core do we fail an upgrade because of the conflict, and make it impossible to actually use said extension? This principle aligns with other customizable components, such > as custom resource managers, wait events, lightweight locks, and > custom scans. These components maintain their built-in data/types and > explicitly prevent the registration of duplicate names. > I am totally lost on how any of those resemble this feature. I’m all for registration to enable additional options and features - but am against moving away from turning format into a namespaced identifier. This is a query-facing feature where namespaces are common and fundamentally required. I have some sympathy for the fact that until now one could not prefix text/binary/csv with pg_catalog to be fully safe, but in reality DBAs/query authors either put pg_catalog first in their search_path or make an informed decision when they deviate. That is the established precedent relevant to this feature. The power, and responsibility for education, lies with the user. David J. ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 05:36 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 05:57 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> @ 2025-05-03 07:54 ` Masahiko Sawada <[email protected]> 2025-05-03 14:42 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Masahiko Sawada @ 2025-05-03 07:54 UTC (permalink / raw) To: David G. Johnston <[email protected]>; +Cc: Sutou Kouhei <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers On Fri, May 2, 2025 at 11:37 PM David G. Johnston <[email protected]> wrote: > > On Friday, May 2, 2025, Masahiko Sawada <[email protected]> wrote: >> >> >> I'm concerned about allowing multiple 'text' format implementations >> with identical names within the database, as this could lead to >> considerable confusion. When users specify 'text', it would be more >> logical to guarantee that the built-in 'text' format is consistently >> used. > > > Do you want to only give text/csv/binary this special treatment or also any future format name we ever decide to implement in core. If an extension takes up “xml” and we try to do that in core do we fail an upgrade because of the conflict, and make it impossible to actually use said extension? I guess that's an extension author's responsibility to upgrade its extension so as to work with the new PostgreSQL version, or carefully choose the format name. They can even name '[extension_name].[format_name]' as a format name. Even with the current patch design (i.e., search_path affects handler function lookups), users would end up using the built-in 'xml' format without notice after upgrade, no? I guess that could introduce another problem. I think that we need to ensure that if users specify text/csv/binary the built-in formats are always used, to keep backward compatibility. > >> This principle aligns with other customizable components, such >> as custom resource managers, wait events, lightweight locks, and >> custom scans. These components maintain their built-in data/types and >> explicitly prevent the registration of duplicate names. > > > I am totally lost on how any of those resemble this feature. > > I’m all for registration to enable additional options and features - but am against moving away from turning format into a namespaced identifier. This is a query-facing feature where namespaces are common and fundamentally required. That's a fair concern. But isn't the format name ultimately just an option value, but not like a database object? As I mentioned above, I think we need to keep backward compatibility but treating the built-in formats special seems inconsistent with common name resolution behavior. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 05:36 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 05:57 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 07:54 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> @ 2025-05-03 14:42 ` David G. Johnston <[email protected]> 2025-05-04 05:02 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: David G. Johnston @ 2025-05-03 14:42 UTC (permalink / raw) To: Masahiko Sawada <[email protected]>; +Cc: Sutou Kouhei <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers On Saturday, May 3, 2025, Masahiko Sawada <[email protected]> wrote: > I think that we need to ensure that if users specify text/csv/binary > the built-in formats are always used, to keep backward compatibility. That was my original thinking, but it’s inconsistent with how functions behave today. We don’t promise that installing extensions won’t cause existing code to change. > > > > I’m all for registration to enable additional options and features - but > am against moving away from turning format into a namespaced identifier. > This is a query-facing feature where namespaces are common and > fundamentally required. > > That's a fair concern. But isn't the format name ultimately just an > option value, but not like a database object? We get to decide that. And deciding in favor of “extensible database object in a namespace’ makes more sense - leveraging all that pre-existing design to play more nicely with extensions and give DBAs control. The SQL command to add one is “create function” instead of “create copy format”. David J. ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 05:36 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 05:57 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 07:54 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 14:42 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> @ 2025-05-04 05:02 ` Masahiko Sawada <[email protected]> 2025-05-04 05:27 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Masahiko Sawada @ 2025-05-04 05:02 UTC (permalink / raw) To: David G. Johnston <[email protected]>; +Cc: Sutou Kouhei <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers On Sat, May 3, 2025 at 7:42 AM David G. Johnston <[email protected]> wrote: > > On Saturday, May 3, 2025, Masahiko Sawada <[email protected]> wrote: > >> >> I think that we need to ensure that if users specify text/csv/binary >> the built-in formats are always used, to keep backward compatibility. > > > That was my original thinking, but it’s inconsistent with how functions behave today. We don’t promise that installing extensions won’t cause existing code to change. I'm skeptical about whether that's an acceptable backward compatibility breakage. >> > I’m all for registration to enable additional options and features - but am against moving away from turning format into a namespaced identifier. This is a query-facing feature where namespaces are common and fundamentally required. >> >> That's a fair concern. But isn't the format name ultimately just an >> option value, but not like a database object? > > > We get to decide that. And deciding in favor of “extensible database object in a namespace’ makes more sense - leveraging all that pre-existing design to play more nicely with extensions and give DBAs control. The SQL command to add one is “create function” instead of “create copy format”. I still don't fully understand why the FORMAT value alone needs to be treated like a schema-qualified object. If the concern is about name conflict with future built-in formats, I would argue that the same concern applies to custom EXPLAIN options and logical decoding plugins. To me, the benefit of treating the COPY FORMAT value as a schema-qualified object seems limited. Meanwhile, the risk of not protecting built-in formats like 'text', 'csv', and 'binary' is significant. If those names can be shadowed by extension via search_patch, we lose backward compatibility. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 05:36 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 05:57 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 07:54 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 14:42 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-04 05:02 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> @ 2025-05-04 05:27 ` David G. Johnston <[email protected]> 2025-05-09 09:41 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: David G. Johnston @ 2025-05-04 05:27 UTC (permalink / raw) To: Masahiko Sawada <[email protected]>; +Cc: Sutou Kouhei <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers On Saturday, May 3, 2025, Masahiko Sawada <[email protected]> wrote: > On Sat, May 3, 2025 at 7:42 AM David G. Johnston > <[email protected]> wrote: > > > > On Saturday, May 3, 2025, Masahiko Sawada <[email protected]> wrote: > > > >> > >> I think that we need to ensure that if users specify text/csv/binary > >> the built-in formats are always used, to keep backward compatibility. > > > > > > That was my original thinking, but it’s inconsistent with how functions > behave today. We don’t promise that installing extensions won’t cause > existing code to change. > > I'm skeptical about whether that's an acceptable backward > compatibility breakage. I’m skeptical you are correctly defining what backward-compatibility requires. Well, the only potential breakage is that we are searching for a matching function by signature without first limiting the mandated return type. But that is solve-able should anyone else see the problem as well. The global format name has its merits but neither it nor the namespaced format option suffer from breaking compatibility or policy. > > I still don't fully understand why the FORMAT value alone needs to be > treated like a schema-qualified object. If the concern is about name > conflict with future built-in formats, I would argue that the same > concern applies to custom EXPLAIN options and logical decoding > plugins. > Then maybe we have the same “problem” in those places. > > To me, the benefit of treating the COPY FORMAT value as a > schema-qualified object seems limited. Meanwhile, the risk of not > protecting built-in formats like 'text', 'csv', and 'binary' is > significant. Really? You think lots of extensions are going to choose to use these values even if they are permitted? Or are you concerned about attack surfaces? > If those names can be shadowed by extension via > search_patch, we lose backward compatibility. > This is not a definition of backward-compatibility that I am familiar with. If anything the ability for a DBA to arrange for such shadowing would be a feature enhancement. They can drop-in a more efficient or desirable implementation without having to change query code. In any case, I’m doubtful either of us can make a convincing enough argument to sway the other fully. Both options are plausible, IMO. Others need to chime in. David J. ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 05:36 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 05:57 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 07:54 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 14:42 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-04 05:02 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-04 05:27 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> @ 2025-05-09 09:41 ` Sutou Kouhei <[email protected]> 2025-05-10 00:57 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Sutou Kouhei @ 2025-05-09 09:41 UTC (permalink / raw) To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers Hi, In <CAKFQuwaRDXANaL+QcT6LZRAem4rwkSwv9v+viv_mcR+Rex3quA@mail.gmail.com> "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 3 May 2025 22:27:36 -0700, "David G. Johnston" <[email protected]> wrote: > In any case, I’m doubtful either of us can make a convincing enough > argument to sway the other fully. Both options are plausible, IMO. Others > need to chime in. I may misunderstand but here is the current summary, right? Proposed approaches to register custom COPY formats: a. Create a function that has the same name of custom COPY format b. Call a register function from _PG_init() FYI: I proposed c. approach that uses a. but it always requires schema name for format name in other e-mail. Users can register the same format name: a. Yes * Users can distinct the same format name by schema name * If format name doesn't have schema name, the used format depends on search_path * Pros: * Using schema for it is consistent with other PostgreSQL mechanisms * Custom format never conflict with built-in format. For example, an extension register "xml" and PostgreSQL adds "xml" later, they are never conflicted because PostgreSQL's "xml" is registered to pg_catalog. * Cons: Different format may be used with the same input. For example, "jsonlines" may choose "jsonlines" implemented by extension X or implemented by extension Y when search_path is different. b. No * Users can use "${schema}.${name}" for format name that mimics PostgreSQL's builtin schema (but it's just a string) Built-in formats (text/csv/binary) should be able to overwritten by extensions: a. (The current patch is no but David's answer is) Yes * Pros: Users can use drop-in replacement faster implementation without changing input * Cons: Users may overwrite them accidentally. It may break pg_dump result. (This is called as "backward incompatibility.") b. No Are there any missing or wrong items? If we can summarize the current discussion here correctly, others will be able to chime in this discussion. (At least I can do it.) Thanks, -- kou ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 05:36 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 05:57 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 07:54 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 14:42 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-04 05:02 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-04 05:27 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-09 09:41 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> @ 2025-05-10 00:57 ` Masahiko Sawada <[email protected]> 2025-05-26 01:04 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 0 siblings, 1 reply; 41+ messages in thread From: Masahiko Sawada @ 2025-05-10 00:57 UTC (permalink / raw) To: Sutou Kouhei <[email protected]>; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers On Fri, May 9, 2025 at 2:41 AM Sutou Kouhei <[email protected]> wrote: > > Hi, > > In <CAKFQuwaRDXANaL+QcT6LZRAem4rwkSwv9v+viv_mcR+Rex3quA@mail.gmail.com> > "Re: Make COPY format extendable: Extract COPY TO format implementations" on Sat, 3 May 2025 22:27:36 -0700, > "David G. Johnston" <[email protected]> wrote: > > > In any case, I’m doubtful either of us can make a convincing enough > > argument to sway the other fully. Both options are plausible, IMO. Others > > need to chime in. > > I may misunderstand but here is the current summary, right? Thank you for summarizing the discussion. > > Proposed approaches to register custom COPY formats: > a. Create a function that has the same name of custom COPY > format > b. Call a register function from _PG_init() > > FYI: I proposed c. approach that uses a. but it always > requires schema name for format name in other e-mail. With approach (c), do you mean that we require users to change all FORMAT option values like from 'text' to 'pg_catalog.text' after the upgrade? Or are we exempt the built-in formats? > > Users can register the same format name: > a. Yes > * Users can distinct the same format name by schema name > * If format name doesn't have schema name, the used > format depends on search_path > * Pros: > * Using schema for it is consistent with other > PostgreSQL mechanisms > * Custom format never conflict with built-in > format. For example, an extension register "xml" and > PostgreSQL adds "xml" later, they are never > conflicted because PostgreSQL's "xml" is registered > to pg_catalog. > * Cons: Different format may be used with the same > input. For example, "jsonlines" may choose > "jsonlines" implemented by extension X or implemented > by extension Y when search_path is different. > b. No > * Users can use "${schema}.${name}" for format name > that mimics PostgreSQL's builtin schema (but it's just > a string) > > > Built-in formats (text/csv/binary) should be able to > overwritten by extensions: > a. (The current patch is no but David's answer is) Yes > * Pros: Users can use drop-in replacement faster > implementation without changing input > * Cons: Users may overwrite them accidentally. > It may break pg_dump result. > (This is called as "backward incompatibility.") > b. No The summary matches my understanding. I think the second point is important. If we go with a tablesample-like API, I agree with David's point that all FORMAT values including the built-in formats should depend on the search_path value. While it provides a similar user experience to other database objects, there is a possibility that a COPY with built-in format could work differently on v19 than v18 or earlier depending on the search_path value. > Are there any missing or wrong items? I think the approach (b) provides more flexibility than (a) in terms of API design as with (a) we need to do everything based on one handler function and callbacks. > If we can summarize > the current discussion here correctly, others will be able > to chime in this discussion. (At least I can do it.) +1 Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 41+ messages in thread
* Re: Make COPY format extendable: Extract COPY TO format implementations 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 05:36 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 05:57 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 07:54 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 14:42 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-04 05:02 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-04 05:27 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-09 09:41 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-10 00:57 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> @ 2025-05-26 01:04 ` Sutou Kouhei <[email protected]> 0 siblings, 0 replies; 41+ messages in thread From: Sutou Kouhei @ 2025-05-26 01:04 UTC (permalink / raw) To: [email protected]; +Cc: [email protected]; [email protected]; [email protected]; [email protected]; pgsql-hackers Hi, In <CAD21AoBrSTmPyDai_QVR-XOe7PL722Dazm70A+FpvGy2hfSV9g@mail.gmail.com> "Re: Make COPY format extendable: Extract COPY TO format implementations" on Fri, 9 May 2025 17:57:35 -0700, Masahiko Sawada <[email protected]> wrote: >> Proposed approaches to register custom COPY formats: >> a. Create a function that has the same name of custom COPY >> format >> b. Call a register function from _PG_init() >> >> FYI: I proposed c. approach that uses a. but it always >> requires schema name for format name in other e-mail. > > With approach (c), do you mean that we require users to change all > FORMAT option values like from 'text' to 'pg_catalog.text' after the > upgrade? Or are we exempt the built-in formats? The latter. 'text' must be accepted because existing pg_dump results use 'text'. If we reject 'text', it's a big incompatibility. (We can't dump on old PostgreSQL and restore to new PostgreSQL.) >> Users can register the same format name: >> a. Yes >> * Users can distinct the same format name by schema name >> * If format name doesn't have schema name, the used >> format depends on search_path >> * Pros: >> * Using schema for it is consistent with other >> PostgreSQL mechanisms >> * Custom format never conflict with built-in >> format. For example, an extension register "xml" and >> PostgreSQL adds "xml" later, they are never >> conflicted because PostgreSQL's "xml" is registered >> to pg_catalog. >> * Cons: Different format may be used with the same >> input. For example, "jsonlines" may choose >> "jsonlines" implemented by extension X or implemented >> by extension Y when search_path is different. >> b. No >> * Users can use "${schema}.${name}" for format name >> that mimics PostgreSQL's builtin schema (but it's just >> a string) >> >> >> Built-in formats (text/csv/binary) should be able to >> overwritten by extensions: >> a. (The current patch is no but David's answer is) Yes >> * Pros: Users can use drop-in replacement faster >> implementation without changing input >> * Cons: Users may overwrite them accidentally. >> It may break pg_dump result. >> (This is called as "backward incompatibility.") >> b. No > > The summary matches my understanding. I think the second point is > important. If we go with a tablesample-like API, I agree with David's > point that all FORMAT values including the built-in formats should > depend on the search_path value. While it provides a similar user > experience to other database objects, there is a possibility that a > COPY with built-in format could work differently on v19 than v18 or > earlier depending on the search_path value. Thanks for sharing additional points. David said that the additional point case is a responsibility or DBA not PostgreSQL, right? As I already said, I don't have a strong opinion on which approach is better. My opinion for the (important) second point is no. I feel that the pros of a. isn't realistic. If users want to improve text/csv/binary performance (or something), they should improve PostgreSQL itself instead of replacing it as an extension. (Or they should create another custom copy format such as "faster_text" not "text".) So I'm OK with the approach b. >> Are there any missing or wrong items? > > I think the approach (b) provides more flexibility than (a) in terms > of API design as with (a) we need to do everything based on one > handler function and callbacks. Thanks for sharing this missing point. I have a concern that the flexibility may introduce needless complexity. If it's not a real concern, I'm OK with the approach b. >> If we can summarize >> the current discussion here correctly, others will be able >> to chime in this discussion. (At least I can do it.) > > +1 Are there any more people who are interested in custom COPY FORMAT implementation design? If no more people, let's decide it by us. Thanks, -- kou ^ permalink raw reply [nested|flat] 41+ messages in thread
end of thread, other threads:[~2025-05-26 01:27 UTC | newest] Thread overview: 41+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2018-10-31 17:53 Re: Continue work on changes to recovery.conf API Robert Haas <[email protected]> 2022-02-21 07:45 [PATCH v7 1/2] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-02-21 07:45 [PATCH v2 3/4] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-02-21 07:45 [PATCH v5 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-02-21 07:45 [PATCH v1 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-02-21 07:45 [PATCH v3 3/4] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-02-21 07:45 [PATCH v6 1/2] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-02-21 07:45 [PATCH v7 1/2] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-02-21 07:45 [PATCH v7 1/2] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-02-21 07:45 [PATCH v4 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-05-30 03:15 [PATCH v8 5/6] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-05-30 03:15 [PATCH v10 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-05-30 03:15 [PATCH v11 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-05-30 03:15 [PATCH v9 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-05-30 03:15 [PATCH v10 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-05-30 03:15 [PATCH v12 2/3] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2022-10-25 06:26 [PATCH v14 4/5] Allow file inclusion in pg_hba and pg_ident files. Julien Rouhaud <[email protected]> 2025-04-25 12:45 Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-01 19:15 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-01 23:03 ` Re: Make COPY format extendable: Extract COPY TO format implementations Michael Paquier <[email protected]> 2025-05-02 22:52 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 02:24 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 02:19 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 04:38 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 04:56 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 06:02 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 06:20 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-09 08:51 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-10 04:29 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-26 01:27 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-03 05:36 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 05:57 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 06:37 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-03 07:54 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-03 14:42 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-04 05:02 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-04 05:27 ` Re: Make COPY format extendable: Extract COPY TO format implementations David G. Johnston <[email protected]> 2025-05-09 09:41 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]> 2025-05-10 00:57 ` Re: Make COPY format extendable: Extract COPY TO format implementations Masahiko Sawada <[email protected]> 2025-05-26 01:04 ` Re: Make COPY format extendable: Extract COPY TO format implementations Sutou Kouhei <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox