public inbox for [email protected]
help / color / mirror / Atom feedproposal: possibility to read dumped table's name from file
181+ messages / 18 participants
[nested] [flat]
* proposal: possibility to read dumped table's name from file
@ 2020-05-29 14:21 Pavel Stehule <[email protected]>
0 siblings, 3 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-05-29 14:21 UTC (permalink / raw)
To: pgsql-hackers
Hi
one my customer has to specify dumped tables name by name. After years and
increasing database size and table numbers he has problem with too short
command line. He need to read the list of tables from file (or from stdin).
I wrote simple PoC patch
Comments, notes, ideas?
Regards
Pavel
Attachments:
[text/x-patch] pg_dump-table-list-from-file.patch (2.6K, ../../CAFj8pRB10wvW0CC9Xq=1XDs=zCQxer3cbLcNZa+qiX4cUH-G_A@mail.gmail.com/3-pg_dump-table-list-from-file.patch)
download | inline diff:
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index bd14d38740..d8330ec32a 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -362,6 +362,7 @@ main(int argc, char **argv)
{"serializable-deferrable", no_argument, &dopt.serializable_deferrable, 1},
{"snapshot", required_argument, NULL, 6},
{"strict-names", no_argument, &strict_names, 1},
+ {"table-names-from-file", required_argument, NULL, 8},
{"use-set-session-authorization", no_argument, &dopt.use_setsessauth, 1},
{"no-publications", no_argument, &dopt.no_publications, 1},
{"no-security-labels", no_argument, &dopt.no_security_labels, 1},
@@ -553,6 +554,59 @@ main(int argc, char **argv)
dosync = false;
break;
+ case 8: /* read table names from file */
+ {
+ FILE *f;
+ char *line;
+ size_t line_size = 1024;
+ ssize_t chars;
+ bool use_stdin = false;
+
+ if (strcmp(optarg, "-") != 0)
+ {
+ f = fopen(optarg, "r");
+ if (!f)
+ {
+ fprintf(stderr, _("%s: could not open the input file \"%s\": %s\n"),
+ progname, optarg, strerror(errno));
+ exit_nicely(1);
+ }
+ }
+ else
+ {
+ f = stdin;
+ use_stdin = true;
+ }
+
+ line = malloc(line_size);
+
+ while ((chars = getline(&line, &line_size, f)) != -1)
+ {
+ if (line[chars - 1] == '\n')
+ line[chars - 1] = '\0';
+
+ /* ignore empty rows */
+ if (*line != '\0')
+ {
+ simple_string_list_append(&table_include_patterns, line);
+ dopt.include_everything = false;
+ }
+ }
+
+ if (ferror(f))
+ {
+ fprintf(stderr, _("%s: could not read from file \"%s\": %s\n"),
+ progname, use_stdin ? "stdin" : optarg, strerror(errno));
+ exit_nicely(1);
+ }
+
+ if (!use_stdin)
+ fclose(f);
+
+ free(line);
+ }
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -977,6 +1031,8 @@ help(const char *progname)
printf(_(" --snapshot=SNAPSHOT use given snapshot for the dump\n"));
printf(_(" --strict-names require table and/or schema include patterns to\n"
" match at least one entity each\n"));
+ printf(_(" --table-names-from-file=FILENAME\n"
+ " read table's names from file (one table name per line)\n"));
printf(_(" --use-set-session-authorization\n"
" use SET SESSION AUTHORIZATION commands instead of\n"
" ALTER OWNER commands to set ownership\n"));
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-05-29 14:28 Tom Lane <[email protected]>
parent: Pavel Stehule <[email protected]>
2 siblings, 1 reply; 181+ messages in thread
From: Tom Lane @ 2020-05-29 14:28 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers
Pavel Stehule <[email protected]> writes:
> one my customer has to specify dumped tables name by name. After years and
> increasing database size and table numbers he has problem with too short
> command line. He need to read the list of tables from file (or from stdin).
I guess the question is why. That seems like an enormously error-prone
approach. Can't they switch to selecting schemas? Or excluding the
hopefully-short list of tables they don't want?
regards, tom lane
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-05-29 14:47 Pavel Stehule <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-05-29 14:47 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
pá 29. 5. 2020 v 16:28 odesílatel Tom Lane <[email protected]> napsal:
> Pavel Stehule <[email protected]> writes:
> > one my customer has to specify dumped tables name by name. After years
> and
> > increasing database size and table numbers he has problem with too short
> > command line. He need to read the list of tables from file (or from
> stdin).
>
> I guess the question is why. That seems like an enormously error-prone
> approach. Can't they switch to selecting schemas? Or excluding the
> hopefully-short list of tables they don't want?
>
It is not typical application. It is a analytic application when the schema
of database is based on dynamic specification of end user (end user can do
customization every time). So schema is very dynamic.
For example - typical server has about four thousand databases and every
database has some between 1K .. 10K tables.
Another specific are different versions of data in different tables. A user
can work with one set of data (one set of tables) and a application
prepares new set of data (new set of tables). Load can be slow, because
sometimes bigger tables are filled (about forty GB). pg_dump backups one
set of tables (little bit like snapshot of data). So it is strange OLAP
(but successfull) application.
Regards
Pavel
> regards, tom lane
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-05-29 15:28 David Fetter <[email protected]>
parent: Pavel Stehule <[email protected]>
2 siblings, 1 reply; 181+ messages in thread
From: David Fetter @ 2020-05-29 15:28 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers
On Fri, May 29, 2020 at 04:21:00PM +0200, Pavel Stehule wrote:
> Hi
>
> one my customer has to specify dumped tables name by name. After years and
> increasing database size and table numbers he has problem with too short
> command line. He need to read the list of tables from file (or from stdin).
>
> I wrote simple PoC patch
>
> Comments, notes, ideas?
This seems like a handy addition. What I've done in cases similar to
this was to use `grep -f` on the output of `pg_dump -l` to create a
file suitable for `pg_dump -L`, or mash them together like this:
pg_restore -L <(pg_dump -l /path/to/dumpfile | grep -f /path/to/listfile) -d new_db /path/to/dumpfile
That's a lot of shell magic and obscure corners of commands to expect
people to use.
Would it make sense to expand this patch to handle other objects?
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-05-29 16:03 Tom Lane <[email protected]>
parent: David Fetter <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Tom Lane @ 2020-05-29 16:03 UTC (permalink / raw)
To: David Fetter <[email protected]>; +Cc: Pavel Stehule <[email protected]>; pgsql-hackers
David Fetter <[email protected]> writes:
> Would it make sense to expand this patch to handle other objects?
If we're gonna do something like this, +1 for being more general.
The fact that pg_dump only has selection switches for tables and
schemas has always struck me as an omission.
regards, tom lane
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-05-29 16:14 Pavel Stehule <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-05-29 16:14 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: David Fetter <[email protected]>; pgsql-hackers
pá 29. 5. 2020 v 18:03 odesílatel Tom Lane <[email protected]> napsal:
> David Fetter <[email protected]> writes:
> > Would it make sense to expand this patch to handle other objects?
>
Sure. Just we should to design system (and names of options).
>
> If we're gonna do something like this, +1 for being more general.
> The fact that pg_dump only has selection switches for tables and
> schemas has always struck me as an omission.
>
a implementation is trivial, hard is good design of names for these options.
Pavel
>
> regards, tom lane
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-05-29 18:25 Justin Pryzby <[email protected]>
parent: Pavel Stehule <[email protected]>
2 siblings, 1 reply; 181+ messages in thread
From: Justin Pryzby @ 2020-05-29 18:25 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers
On Fri, May 29, 2020 at 04:21:00PM +0200, Pavel Stehule wrote:
> one my customer has to specify dumped tables name by name. After years and
> increasing database size and table numbers he has problem with too short
> command line. He need to read the list of tables from file (or from stdin).
+1 - we would use this.
We put a regex (actually a pg_dump pattern) of tables to skip (timeseries
partitions which are older than a few days and which are also dumped once not
expected to change, and typically not redumped). We're nowhere near the
execve() limit, but it'd be nice if the command was primarily a list of options
and not a long regex.
Please also support reading from file for --exclude-table=pattern.
I'm drawing a parallel between this and rsync --include/--exclude and --filter.
We'd be implementing a new --filter, which might have similar syntax to rsync
(which I always forget).
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-06-08 17:18 Pavel Stehule <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-06-08 17:18 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: pgsql-hackers
Hi
pá 29. 5. 2020 v 20:25 odesílatel Justin Pryzby <[email protected]>
napsal:
> On Fri, May 29, 2020 at 04:21:00PM +0200, Pavel Stehule wrote:
> > one my customer has to specify dumped tables name by name. After years
> and
> > increasing database size and table numbers he has problem with too short
> > command line. He need to read the list of tables from file (or from
> stdin).
>
> +1 - we would use this.
>
> We put a regex (actually a pg_dump pattern) of tables to skip (timeseries
> partitions which are older than a few days and which are also dumped once
> not
> expected to change, and typically not redumped). We're nowhere near the
> execve() limit, but it'd be nice if the command was primarily a list of
> options
> and not a long regex.
>
> Please also support reading from file for --exclude-table=pattern.
>
> I'm drawing a parallel between this and rsync --include/--exclude and
> --filter.
>
> We'd be implementing a new --filter, which might have similar syntax to
> rsync
> (which I always forget).
>
I implemented support for all "repeated" pg_dump options.
--exclude-schemas-file=FILENAME
--exclude-tables-data-file=FILENAME
--exclude-tables-file=FILENAME
--include-foreign-data-file=FILENAME
--include-schemas-file=FILENAME
--include-tables-file=FILENAME
Regards
Pavel
I invite any help with doc. There is just very raw text
> --
> Justin
>
Attachments:
[text/x-patch] pg_dump-read-options-from-file.patch (8.3K, ../../CAFj8pRDGTeb9OtXqFiE-tnYHCV=HyZQ2T_7S5v_50eJqAnLDwg@mail.gmail.com/3-pg_dump-read-options-from-file.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 2f0807e912..5e39268d51 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -725,6 +725,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-schemas-file=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ This option allows to specify file with schemas that will not be
+ dumped. Any row of is one schema's name.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--exclude-table-data=<replaceable class="parameter">pattern</replaceable></option></term>
<listitem>
@@ -743,6 +753,24 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-tables-data-file=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not dump data of tables spefified in file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--exclude-tables-file=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not dump tables spefified in file.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--extra-float-digits=<replaceable class="parameter">ndigits</replaceable></option></term>
<listitem>
@@ -795,6 +823,33 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--include-foreign-data-file=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Include data of foreign servers specified in file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--include-schemas-file=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Dump schema(s) specified in file.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--include-tables-file=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Dump table(s) specified in file.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--inserts</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index dfe43968b8..db9fb10f68 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -290,6 +290,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static bool read_options_from_file(SimpleStringList *slist, char *filename);
int
@@ -362,8 +363,13 @@ main(int argc, char **argv)
{"disable-dollar-quoting", no_argument, &dopt.disable_dollar_quoting, 1},
{"disable-triggers", no_argument, &dopt.disable_triggers, 1},
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
+ {"exclude-schemas-file", required_argument, NULL, 12},
{"exclude-table-data", required_argument, NULL, 4},
+ {"exclude-tables-data-file", required_argument, NULL, 14},
+ {"exclude-tables-file", required_argument, NULL, 13},
{"extra-float-digits", required_argument, NULL, 8},
+ {"include-schemas-file", required_argument, NULL, 15},
+ {"include-tables-file", required_argument, NULL, 16},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -386,6 +392,7 @@ main(int argc, char **argv)
{"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1},
{"rows-per-insert", required_argument, NULL, 10},
{"include-foreign-data", required_argument, NULL, 11},
+ {"include-foreign-data-file", required_argument, NULL, 17},
{NULL, 0, NULL, 0}
};
@@ -603,6 +610,32 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* read exclude schama names from file */
+ (void) read_options_from_file(&schema_exclude_patterns, optarg);
+ break;
+
+ case 13: /* read exclude table names from file */
+ (void) read_options_from_file(&table_exclude_patterns, optarg);
+ break;
+
+ case 14: /* read exclude table data names from file */
+ (void) read_options_from_file(&tabledata_exclude_patterns, optarg);
+ break;
+
+ case 15: /* read table names from file */
+ if (read_options_from_file(&schema_include_patterns, optarg))
+ dopt.include_everything = false;
+ break;
+
+ case 16: /* read table names from file */
+ if (read_options_from_file(&table_include_patterns, optarg))
+ dopt.include_everything = false;
+ break;
+
+ case 17: /* read exclude table data names from file */
+ (void) read_options_from_file(&foreign_servers_include_patterns, optarg);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1020,12 +1053,24 @@ help(const char *progname)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security (dump only content user has\n"
" access to)\n"));
+ printf(_(" --exclude-schemas-file=FILENAME\n"
+ " do NOT dump schema specified in file\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
+ printf(_(" --exclude-tables-data-file=FILENAME\n"
+ " do NOT dump data for tables specified in file\n"));
+ printf(_(" --exclude-tables-file=FILENAME\n"
+ " do NOT dump tables specified in file\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
" servers matching PATTERN\n"));
+ printf(_(" --include-foreign-data-file=FILENAME\n"
+ " include data of foreign servers specified by file\n"));
+ printf(_(" --include-schemas-file=FILENAME\n"
+ " dump schema(s) specified in file\n"));
+ printf(_(" --include-tables-file=FILENAME\n"
+ " dump table(s) specified in file\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
printf(_(" --no-comments do not dump comments\n"));
@@ -18647,3 +18692,70 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Read list of values from file specified by name. Returns true when
+ * at least one value was read.
+ */
+static bool
+read_options_from_file(SimpleStringList *slist, char *filename)
+{
+ FILE *f;
+ char *line;
+ ssize_t chars;
+ size_t line_size = 1024;
+ bool use_stdin = false;
+ bool result = false;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ f = fopen(optarg, "r");
+ if (!f)
+ {
+ fprintf(stderr,
+ _("%s: could not open the input file \"%s\": %s\n"),
+ progname,
+ optarg,
+ strerror(errno));
+ exit_nicely(1);
+ }
+ }
+ else
+ {
+ f = stdin;
+ use_stdin = true;
+ }
+
+ line = malloc(line_size);
+
+ while ((chars = getline(&line, &line_size, f)) != -1)
+ {
+ if (line[chars - 1] == '\n')
+ line[chars - 1] = '\0';
+
+ /* ignore empty rows */
+ if (*line != '\0')
+ {
+ simple_string_list_append(slist, line);
+ result = true;
+ }
+ }
+
+ if (ferror(f))
+ {
+ fprintf(stderr,
+ _("%s: could not read from file \"%s\": %s\n"),
+ progname,
+ use_stdin ? "stdin" : optarg,
+ strerror(errno));
+ exit_nicely(1);
+ }
+
+ if (!use_stdin)
+ fclose(f);
+
+ free(line);
+
+ return result;
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-06-08 21:29 Justin Pryzby <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Justin Pryzby @ 2020-06-08 21:29 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers
On Mon, Jun 08, 2020 at 07:18:49PM +0200, Pavel Stehule wrote:
> pá 29. 5. 2020 v 20:25 odesílatel Justin Pryzby <[email protected]> napsal:
>
> > On Fri, May 29, 2020 at 04:21:00PM +0200, Pavel Stehule wrote:
> > > one my customer has to specify dumped tables name by name. After years and
> > > increasing database size and table numbers he has problem with too short
> > > command line. He need to read the list of tables from file (or from stdin).
> >
> > +1 - we would use this.
> >
> > We put a regex (actually a pg_dump pattern) of tables to skip (timeseries
> > partitions which are older than a few days and which are also dumped once not
> > expected to change, and typically not redumped). We're nowhere near the
> > execve() limit, but it'd be nice if the command was primarily a list of options
> > and not a long regex.
> >
> > Please also support reading from file for --exclude-table=pattern.
> >
> > I'm drawing a parallel between this and rsync --include/--exclude and
> > --filter.
> >
> > We'd be implementing a new --filter, which might have similar syntax to rsync
> > (which I always forget).
>
> I implemented support for all "repeated" pg_dump options.
>
> I invite any help with doc. There is just very raw text
>
> + Do not dump data of tables spefified in file.
*specified
I still wonder if a better syntax would use a unified --filter option, whose
argument would allow including/excluding any type of object:
+[tn] include (t)table/(n)namespace/...
-[tn] exclude (t)table/(n)namespace/...
In the past, I looked for a way to exclude extended stats objects, and ended up
using a separate schema. An "extensible" syntax might be better (although
reading a file of just patterns has the advantage that the function can just be
called once for each option for each type of object).
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-06-09 09:46 Pavel Stehule <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-06-09 09:46 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: pgsql-hackers
po 8. 6. 2020 v 23:30 odesílatel Justin Pryzby <[email protected]>
napsal:
> On Mon, Jun 08, 2020 at 07:18:49PM +0200, Pavel Stehule wrote:
> > pá 29. 5. 2020 v 20:25 odesílatel Justin Pryzby <[email protected]>
> napsal:
> >
> > > On Fri, May 29, 2020 at 04:21:00PM +0200, Pavel Stehule wrote:
> > > > one my customer has to specify dumped tables name by name. After
> years and
> > > > increasing database size and table numbers he has problem with too
> short
> > > > command line. He need to read the list of tables from file (or from
> stdin).
> > >
> > > +1 - we would use this.
> > >
> > > We put a regex (actually a pg_dump pattern) of tables to skip
> (timeseries
> > > partitions which are older than a few days and which are also dumped
> once not
> > > expected to change, and typically not redumped). We're nowhere near
> the
> > > execve() limit, but it'd be nice if the command was primarily a list
> of options
> > > and not a long regex.
> > >
> > > Please also support reading from file for --exclude-table=pattern.
> > >
> > > I'm drawing a parallel between this and rsync --include/--exclude and
> > > --filter.
> > >
> > > We'd be implementing a new --filter, which might have similar syntax
> to rsync
> > > (which I always forget).
> >
> > I implemented support for all "repeated" pg_dump options.
> >
> > I invite any help with doc. There is just very raw text
> >
> > + Do not dump data of tables spefified in file.
>
> *specified
>
>
I am sending updated version - now with own implementation GNU (not POSIX)
function getline
I still wonder if a better syntax would use a unified --filter option, whose
> argument would allow including/excluding any type of object:
>
> +[tn] include (t)table/(n)namespace/...
> -[tn] exclude (t)table/(n)namespace/...
>
> In the past, I looked for a way to exclude extended stats objects, and
> ended up
> using a separate schema. An "extensible" syntax might be better (although
> reading a file of just patterns has the advantage that the function can
> just be
> called once for each option for each type of object).
>
I tried to implement simple format "[+-][tndf] objectname"
please, check attached patch
Regards
Pavel
>
> --
> Justin
>
Attachments:
[text/x-patch] pg_dump-filter.patch (6.4K, ../../CAFj8pRBP105Qa=2q9MEy_PWmCZGaWd4nt4nB2Gv8WgBO7UbkwA@mail.gmail.com/3-pg_dump-filter.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 2f0807e912..332f0c312f 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -813,6 +813,15 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read filters from file. Format "(+|-)(tnfd) objectname:
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--load-via-partition-root</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index dfe43968b8..acc936ac7b 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -290,6 +290,8 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static size_t pg_getline(char **lineptr, size_t *n, FILE *fp);
+static void invalid_filter_format(char *filename, char *line, int lineno);
int
@@ -364,6 +366,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -603,6 +606,116 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ {
+ FILE *f;
+ char *line;
+ ssize_t chars;
+ size_t line_size = 1024;
+ int lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(optarg, "-") != 0)
+ {
+ f = fopen(optarg, "r");
+ if (!f)
+ fatal("could not open the input file \"%s\": %s",
+ optarg,
+ strerror(errno));
+ }
+ else
+ f = stdin;
+
+ line = pg_malloc(line_size);
+
+ while ((chars = pg_getline(&line, &line_size, f)) != -1)
+ {
+ lineno += 1;
+
+ if (line[chars - 1] == '\n')
+ line[chars - 1] = '\0';
+
+ /* ignore empty rows */
+ if (*line != '\0')
+ {
+ bool include_filter = false;
+ bool exclude_filter = false;
+ char objecttype;
+ char *objectname;
+
+ if (chars < 4)
+ invalid_filter_format(optarg, line, lineno);
+
+ if (line[0] == '+')
+ include_filter = true;
+ else if (line[0] == '-')
+ exclude_filter = true;
+ else
+ invalid_filter_format(optarg, line, lineno);
+
+ objecttype = line[1];
+
+ if (objecttype != 't' &&
+ objecttype != 'n' &&
+ objecttype != 'd' &&
+ objecttype != 'f')
+ invalid_filter_format(optarg, line, lineno);
+
+ objectname = &line[3];
+
+ /* skip initial spaces */
+ while (*objectname == ' ')
+ objectname++;
+
+ if (objecttype == 't')
+ {
+ if (include_filter)
+ {
+ simple_string_list_append(&table_include_patterns, objectname);
+ dopt.include_everything = false;
+ }
+ else if (exclude_filter)
+ simple_string_list_append(&table_exclude_patterns, objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (include_filter)
+ {
+ simple_string_list_append(&schema_include_patterns, objectname);
+ dopt.include_everything = false;
+ }
+ else if (exclude_filter)
+ simple_string_list_append(&schema_exclude_patterns, objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (include_filter)
+ invalid_filter_format(optarg, line, lineno);
+ else if (exclude_filter)
+ simple_string_list_append(&tabledata_exclude_patterns, objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (include_filter)
+ simple_string_list_append(&foreign_servers_include_patterns, objectname);
+ else if (exclude_filter)
+ invalid_filter_format(optarg, line, lineno);
+ }
+ }
+ }
+
+ if (ferror(f))
+ fatal("could not read from file \"%s\": %s",
+ f == stdin ? "stdin" : optarg,
+ strerror(errno));
+
+ if (f != stdin)
+ fclose(f);
+
+ pg_free(line);
+ }
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1022,6 +1135,7 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME read object names from file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18647,3 +18761,61 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * getline is originaly GNU function, and should not be everywhere still.
+ * Use own reduced implementation.
+ */
+static size_t
+pg_getline(char **lineptr, size_t *n, FILE *fp)
+{
+ size_t total_chars = 0;
+
+ while (!feof(fp) && !ferror(fp))
+ {
+ char *str;
+ size_t chars;
+
+ str = fgets(*lineptr + total_chars,
+ *n - total_chars,
+ fp);
+
+ if (ferror(fp))
+ return -1;
+
+ if (str)
+ {
+ chars = strlen(str);
+ total_chars += chars;
+
+ if (chars > 0 && str[chars - 1] == '\n')
+ return total_chars;
+
+ /* check, if there is good enough space for next content */
+ if (*n - total_chars < 2)
+ {
+ *n += 1024;
+ *lineptr = pg_realloc(*lineptr, *n);
+ }
+ }
+ else
+ break;
+ }
+
+ if (ferror(fp))
+ return -1;
+
+ return total_chars > 0 ? total_chars : -1;
+}
+
+/*
+ * Print error message and exit.
+ */
+static void
+invalid_filter_format(char *filename, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\"",
+ *filename == '-' ? "stdin" : filename);
+ fprintf(stderr, "%d: %s\n", lineno, line);
+ exit_nicely(1);
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-06-09 22:30 Justin Pryzby <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Justin Pryzby @ 2020-06-09 22:30 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers
On Tue, Jun 09, 2020 at 11:46:24AM +0200, Pavel Stehule wrote:
> po 8. 6. 2020 v 23:30 odesílatel Justin Pryzby <[email protected]> napsal:
> I still wonder if a better syntax would use a unified --filter option, whose
> > argument would allow including/excluding any type of object:
> I tried to implement simple format "[+-][tndf] objectname"
Thanks.
> + /* ignore empty rows */
> + if (*line != '\0')
Maybe: if line=='\0': continue
We should also support comments.
> + bool include_filter = false;
> + bool exclude_filter = false;
I think we only need one bool.
You could call it: bool is_exclude = false
> +
> + if (chars < 4)
> + invalid_filter_format(optarg, line, lineno);
I think that check is too lax.
I think it's ok if we require the first char to be [-+] and the 2nd char to be
[dntf]
> + objecttype = line[1];
... but I think this is inadequately "liberal in what it accepts"; I think it
should skip spaces. In my proposed scheme, someone might reasonably write:
> +
> + objectname = &line[3];
> +
> + /* skip initial spaces */
> + while (*objectname == ' ')
> + objectname++;
I suggest to use isspace()
I think we should check that *objectname != '\0', rather than chars>=4, above.
> + if (include_filter)
> + {
> + simple_string_list_append(&table_include_patterns, objectname);
> + dopt.include_everything = false;
> + }
> + else if (exclude_filter)
> + simple_string_list_append(&table_exclude_patterns, objectname);
If you use bool is_exclude, then this becomes "else" and you don't need to
think about checking if (!include && !exclude).
> + else if (objecttype == 'f')
> + {
> + if (include_filter)
> + simple_string_list_append(&foreign_servers_include_patterns, objectname);
> + else if (exclude_filter)
> + invalid_filter_format(optarg, line, lineno);
> + }
I would handle invalid object types as "else: invalid_filter_format()" here,
rather than duplicating above as: !=ALL('d','n','t','f')
> +
> + if (ferror(f))
> + fatal("could not read from file \"%s\": %s",
> + f == stdin ? "stdin" : optarg,
> + strerror(errno));
I think we're allowed to use %m here ?
> + printf(_(" --filter=FILENAME read object names from file\n"));
Object name filter expression, or something..
> + * getline is originaly GNU function, and should not be everywhere still.
originally
> + * Use own reduced implementation.
Did you "reduce" this from another implementation? Where?
What is its license ?
Maybe a line-reader already exists in the frontend (?) .. or maybe it should.
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-06-10 03:03 Pavel Stehule <[email protected]>
parent: Justin Pryzby <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-06-10 03:03 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: pgsql-hackers
st 10. 6. 2020 v 0:30 odesílatel Justin Pryzby <[email protected]>
napsal:
> On Tue, Jun 09, 2020 at 11:46:24AM +0200, Pavel Stehule wrote:
> > po 8. 6. 2020 v 23:30 odesílatel Justin Pryzby <[email protected]>
> napsal:
>
> > I still wonder if a better syntax would use a unified --filter option,
> whose
> > > argument would allow including/excluding any type of object:
> > I tried to implement simple format "[+-][tndf] objectname"
>
I had another idea about format - instead using +-, we can use case
sensitive options same to pg_dump command line (with extending Df -
because these options doesn't exists in short form)
So format can looks like
[tTnNDf] {objectname}
What do you think about this? This format is simpler, and it can work. What
do you think about it?
> Did you "reduce" this from another implementation? Where?
> What is its license ?
The code is 100% mine. It is not copy from gnulib and everybody can simply
check it
https://code.woboq.org/userspace/glibc/stdio-common/getline.c.html
https://code.woboq.org/userspace/glibc/libio/iogetdelim.c.html#_IO_getdelim
Reduced in functionality sense. There is no full argument check that is
necessary for glibc functions. There are no memory checks because
pg_malloc, pg_realloc are used.
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-06-10 03:57 Justin Pryzby <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Justin Pryzby @ 2020-06-10 03:57 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers
On Wed, Jun 10, 2020 at 05:03:49AM +0200, Pavel Stehule wrote:
> st 10. 6. 2020 v 0:30 odesílatel Justin Pryzby <[email protected]> napsal:
>
> > On Tue, Jun 09, 2020 at 11:46:24AM +0200, Pavel Stehule wrote:
> > > po 8. 6. 2020 v 23:30 odesílatel Justin Pryzby <[email protected]> napsal:
> >
> > > I still wonder if a better syntax would use a unified --filter option, whose
> > > > argument would allow including/excluding any type of object:
> > > I tried to implement simple format "[+-][tndf] objectname"
>
> I had another idea about format - instead using +-, we can use case
> sensitive options same to pg_dump command line (with extending Df -
> because these options doesn't exists in short form)
>
> So format can looks like
>
> [tTnNDf] {objectname}
>
> What do you think about this? This format is simpler, and it can work. What
> do you think about it?
I prefer [-+][dtnf], which is similar to rsync --filter, and clear what it's
doing. I wouldn't put much weight on what the short options are.
I wonder if some people would want to be able to use *long* or short options:
-table foo
+schema baz
Or maybe:
exclude-table=foo
schema=bar
Some tools use "long options without leading dashes" as their configuration
file format. Examples: openvpn, mysql. So that could be a good option.
OTOH, there's only a few "keys", so I'm not sure how many people would want to
repeat them, if there's enough to bother putting them in the file rather than
the cmdline.
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-06-11 07:36 Pavel Stehule <[email protected]>
parent: Justin Pryzby <[email protected]>
1 sibling, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-06-11 07:36 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: pgsql-hackers
st 10. 6. 2020 v 0:30 odesílatel Justin Pryzby <[email protected]>
napsal:
> On Tue, Jun 09, 2020 at 11:46:24AM +0200, Pavel Stehule wrote:
> > po 8. 6. 2020 v 23:30 odesílatel Justin Pryzby <[email protected]>
> napsal:
>
> > I still wonder if a better syntax would use a unified --filter option,
> whose
> > > argument would allow including/excluding any type of object:
> > I tried to implement simple format "[+-][tndf] objectname"
>
> Thanks.
>
> > + /* ignore empty rows */
> > + if (*line != '\0')
>
> Maybe: if line=='\0': continue
>
ok
> We should also support comments.
>
> > + bool
> include_filter = false;
> > + bool
> exclude_filter = false;
>
> I think we only need one bool.
> You could call it: bool is_exclude = false
>
>
ok
> +
> > + if (chars < 4)
> > +
> invalid_filter_format(optarg, line, lineno);
>
> I think that check is too lax.
> I think it's ok if we require the first char to be [-+] and the 2nd char
> to be
> [dntf]
>
> > + objecttype =
> line[1];
>
> ... but I think this is inadequately "liberal in what it accepts"; I think
> it
> should skip spaces. In my proposed scheme, someone might reasonably write:
>
> > +
> > + objectname =
> &line[3];
> > +
> > + /* skip initial
> spaces */
> > + while (*objectname
> == ' ')
> > +
> objectname++;
>
> I suggest to use isspace()
>
ok
> I think we should check that *objectname != '\0', rather than chars>=4,
> above.
>
done
> > + if
> (include_filter)
> > + {
> > +
> simple_string_list_append(&table_include_patterns, objectname);
> > +
> dopt.include_everything = false;
> > + }
> > + else if
> (exclude_filter)
> > +
> simple_string_list_append(&table_exclude_patterns, objectname);
>
> If you use bool is_exclude, then this becomes "else" and you don't need to
> think about checking if (!include && !exclude).
>
> > + else if
> (objecttype == 'f')
> > + {
> > + if
> (include_filter)
> > +
> simple_string_list_append(&foreign_servers_include_patterns, objectname);
> > + else if
> (exclude_filter)
> > +
> invalid_filter_format(optarg, line, lineno);
> > + }
>
> I would handle invalid object types as "else: invalid_filter_format()"
> here,
> rather than duplicating above as: !=ALL('d','n','t','f')
>
good idea
> > +
> > + if (ferror(f))
> > + fatal("could not read from
> file \"%s\": %s",
> > + f == stdin ?
> "stdin" : optarg,
> > + strerror(errno));
>
> I think we're allowed to use %m here ?
>
changed
> > + printf(_(" --filter=FILENAME read object names from
> file\n"));
>
> Object name filter expression, or something..
>
yes, it is not object names now
> > + * getline is originaly GNU function, and should not be everywhere
> still.
> originally
>
> > + * Use own reduced implementation.
>
> Did you "reduce" this from another implementation? Where?
> What is its license ?
>
> Maybe a line-reader already exists in the frontend (?) .. or maybe it
> should.
>
everywhere else is used a function fgets. Currently pg_getline is used just
on only one place, so I don't think so moving it to some common part is
maybe premature.
Maybe it can be used as replacement of some fgets calls, but then it is
different topic, I think.
Thank you for comments, attached updated patch
Regards
Pavel
> --
> Justin
>
Attachments:
[text/x-patch] pg_dump-filter-20200611.patch (6.9K, ../../CAFj8pRDk6EQGpbVO5tOsDvNnw41wmYWS7-w5Ym8gyRjvfoZ64g@mail.gmail.com/3-pg_dump-filter-20200611.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 2f0807e912..332f0c312f 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -813,6 +813,15 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read filters from file. Format "(+|-)(tnfd) objectname:
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--load-via-partition-root</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index dfe43968b8..9e76189635 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -290,6 +290,8 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static size_t pg_getline(char **lineptr, size_t *n, FILE *fp);
+static void invalid_filter_format(char *message, char *filename, char *line, int lineno);
int
@@ -364,6 +366,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -603,6 +606,135 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ {
+ FILE *f;
+ char *line;
+ ssize_t chars;
+ size_t line_size = 1024;
+ int lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(optarg, "-") != 0)
+ {
+ f = fopen(optarg, "r");
+ if (!f)
+ fatal("could not open the input file \"%s\": %m",
+ optarg);
+ }
+ else
+ f = stdin;
+
+ line = pg_malloc(line_size);
+
+ while ((chars = pg_getline(&line, &line_size, f)) != -1)
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+
+ lineno += 1;
+
+ if (line[chars - 1] == '\n')
+ line[chars - 1] = '\0';
+
+ /* ignore empty rows */
+ if (*line == '\0')
+ continue;
+
+ if (chars < 2)
+ invalid_filter_format("too short line",
+ optarg,
+ line,
+ lineno);
+
+ if (line[0] == '+')
+ is_include = true;
+ else if (line[0] == '-')
+ is_include = false;
+ else
+ invalid_filter_format("invalid option type (use [+-]",
+ optarg,
+ line,
+ lineno);
+
+ objecttype = line[1];
+ objectname = &line[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ invalid_filter_format("missing object name",
+ optarg,
+ line,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt.include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt.include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ invalid_filter_format("include filter is not supported for this type of object",
+ optarg,
+ line,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ invalid_filter_format("exclude filter is not supported for this type of object",
+ optarg,
+ line,
+ lineno);
+ }
+ else
+ invalid_filter_format("invalid object type (use [tndf])",
+ optarg,
+ line,
+ lineno);
+ }
+
+ if (ferror(f))
+ fatal("could not read from file \"%s\": %m",
+ f == stdin ? "stdin" : optarg);
+
+ if (f != stdin)
+ fclose(f);
+
+ pg_free(line);
+ }
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1022,6 +1154,7 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME read object name filter expressions from file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18647,3 +18780,67 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * getline is originaly GNU function, and should not be everywhere still.
+ * Use own reduced implementation.
+ */
+static size_t
+pg_getline(char **lineptr, size_t *n, FILE *fp)
+{
+ size_t total_chars = 0;
+
+ while (!feof(fp) && !ferror(fp))
+ {
+ char *str;
+ size_t chars;
+
+ str = fgets(*lineptr + total_chars,
+ *n - total_chars,
+ fp);
+
+ if (ferror(fp))
+ return -1;
+
+ if (str)
+ {
+ chars = strlen(str);
+ total_chars += chars;
+
+ if (chars > 0 && str[chars - 1] == '\n')
+ return total_chars;
+
+ /* check, if there is good enough space for next content */
+ if (*n - total_chars < 2)
+ {
+ *n += 1024;
+ *lineptr = pg_realloc(*lineptr, *n);
+ }
+ }
+ else
+ break;
+ }
+
+ if (ferror(fp))
+ return -1;
+
+ return total_chars > 0 ? total_chars : -1;
+}
+
+/*
+ * Print error message and exit.
+ */
+static void
+invalid_filter_format(char *message, char *filename, char *line, int lineno)
+{
+ char *displayname;
+
+ displayname = *filename == '-' ? "stdin" : filename;
+
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ displayname,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+ exit_nicely(1);
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-06-27 12:55 vignesh C <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: vignesh C @ 2020-06-27 12:55 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Justin Pryzby <[email protected]>; pgsql-hackers
On Thu, Jun 11, 2020 at 1:07 PM Pavel Stehule <[email protected]> wrote:
>
> Thank you for comments, attached updated patch
>
Few comments:
+invalid_filter_format(char *message, char *filename, char *line, int lineno)
+{
+ char *displayname;
+
+ displayname = *filename == '-' ? "stdin" : filename;
+
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ displayname,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+ exit_nicely(1);
+}
I think fclose is missing here.
+ if (line[chars - 1] == '\n')
+ line[chars - 1] = '\0';
Should we check for '\r' also to avoid failures in some platforms.
+ <varlistentry>
+ <term><option>--filter=<replaceable
class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read filters from file. Format "(+|-)(tnfd) objectname:
+ </para>
+ </listitem>
+ </varlistentry>
I felt some documentation is missing here. We could include,
options tnfd is for controlling table, schema, foreign server data &
table exclude patterns.
Instead of using tnfd, if we could use the same options as existing
pg_dump options it will be less confusing.
Regards,
Vignesh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-01 21:24 Justin Pryzby <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 2 replies; 181+ messages in thread
From: Justin Pryzby @ 2020-07-01 21:24 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers
On Thu, Jun 11, 2020 at 09:36:18AM +0200, Pavel Stehule wrote:
> st 10. 6. 2020 v 0:30 odesílatel Justin Pryzby <[email protected]> napsal:
> > > + /* ignore empty rows */
> > > + if (*line != '\0')
> >
> > Maybe: if line=='\0': continue
> > We should also support comments.
Comment support is still missing but easily added :)
I tried this patch and it works for my purposes.
Also, your getline is dynamically re-allocating lines of arbitrary length.
Possibly that's not needed. We'll typically read "+t schema.relname", which is
132 chars. Maybe it's sufficient to do
char buf[1024];
fgets(buf);
if strchr(buf, '\n') == NULL: error();
ret = pstrdup(buf);
In any case, you could have getline return a char* and (rather than following
GNU) no need to take char**, int* parameters to conflate inputs and outputs.
I realized that --filter has an advantage over the previous implementation
(with multiple --exclude-* and --include-*) in that it's possible to use stdin
for includes *and* excludes.
By chance, I had the opportunity yesterday to re-use with rsync a regex that
I'd previously been using with pg_dump and grep. What this patch calls
"--filter" in rsync is called "--filter-from". rsync's --filter-from rejects
filters of length longer than max filename, so I had to split it up into
multiple lines instead of using regex alternation ("|"). This option is a
close parallel in pg_dump.
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-05 19:50 Pavel Stehule <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-07-05 19:50 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Justin Pryzby <[email protected]>; pgsql-hackers
so 27. 6. 2020 v 14:55 odesílatel vignesh C <[email protected]> napsal:
> On Thu, Jun 11, 2020 at 1:07 PM Pavel Stehule <[email protected]>
> wrote:
> >
> > Thank you for comments, attached updated patch
> >
>
> Few comments:
> +invalid_filter_format(char *message, char *filename, char *line, int
> lineno)
> +{
> + char *displayname;
> +
> + displayname = *filename == '-' ? "stdin" : filename;
> +
> + pg_log_error("invalid format of filter file \"%s\": %s",
> + displayname,
> + message);
> +
> + fprintf(stderr, "%d: %s\n", lineno, line);
> + exit_nicely(1);
> +}
>
I think fclose is missing here.
>
done
>
> + if (line[chars - 1] ==
> '\n')
> + line[chars - 1] =
> '\0';
> Should we check for '\r' also to avoid failures in some platforms.
>
I checked other usage of fgets in Postgres source code, and everywhere is
used test on \n
When I did some fast research, then
https://stackoverflow.com/questions/12769289/carriage-return-by-fgets \r in
this case should be thrown by libc on Microsoft
https://stackoverflow.com/questions/2061334/fgets-linux-vs-mac
\n should be on Mac OS X .. 2001 year .. I am not sure if Mac OS 9 should
be supported.
>
> + <varlistentry>
> + <term><option>--filter=<replaceable
> class="parameter">filename</replaceable></option></term>
> + <listitem>
> + <para>
> + Read filters from file. Format "(+|-)(tnfd) objectname:
> + </para>
> + </listitem>
> + </varlistentry>
>
> I felt some documentation is missing here. We could include,
> options tnfd is for controlling table, schema, foreign server data &
> table exclude patterns.
>
I have a plan to completate doc when the design is completed. It was not
clear if people prefer long or short forms of option names.
> Instead of using tnfd, if we could use the same options as existing
> pg_dump options it will be less confusing.
>
it almost same
+-t .. tables
+-n schema
-d exclude data .. there is not short option for --exclude-table-data
+f include foreign table .. there is not short option for
--include-foreign-data
So still, there is a opened question if use +-tnfd system, or system based
on long option
table foo
exclude-table foo
schema xx
exclude-schema xx
include-foreign-data yyy
exclude-table-data zzz
Typically these files will be generated by scripts and processed via pipe,
so there I see just two arguments for and aginst:
short format - there is less probability to do typo error (but there is not
full consistency with pg_dump options)
long format - it is self documented (and there is full consistency with
pg_dump)
In this case I prefer short form .. it is more comfortable for users, and
there are only a few variants, so it is not necessary to use too verbose
language (design). But my opinion is not aggressively strong and I'll
accept any common agreement.
Regards
Updated patch attached
> Regards,
> Vignesh
> EnterpriseDB: http://www.enterprisedb.com
>
Attachments:
[text/x-patch] pg_dump-filter-20200705.patch (7.1K, ../../CAFj8pRB4g-rS+ZhHN8QFbZNpZ3M2MF-2_FxTYLYK=KckzapBeg@mail.gmail.com/3-pg_dump-filter-20200705.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 2f0807e912..332f0c312f 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -813,6 +813,15 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read filters from file. Format "(+|-)(tnfd) objectname:
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--load-via-partition-root</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a41a3db876..a3f8bebf52 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -290,6 +290,8 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static size_t pg_getline(char **lineptr, size_t *n, FILE *fp);
+static void exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno);
int
@@ -364,6 +366,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -603,6 +606,141 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ {
+ FILE *f;
+ char *line;
+ ssize_t chars;
+ size_t line_size = 1024;
+ int lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(optarg, "-") != 0)
+ {
+ f = fopen(optarg, "r");
+ if (!f)
+ fatal("could not open the input file \"%s\": %m",
+ optarg);
+ }
+ else
+ f = stdin;
+
+ line = pg_malloc(line_size);
+
+ while ((chars = pg_getline(&line, &line_size, f)) != -1)
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+
+ lineno += 1;
+
+ if (line[chars - 1] == '\n')
+ line[chars - 1] = '\0';
+
+ /* ignore empty rows */
+ if (*line == '\0')
+ continue;
+
+ if (chars < 2)
+ exit_invalid_filter_format(f,
+ optarg,
+ "too short line",
+ line,
+ lineno);
+
+ if (line[0] == '+')
+ is_include = true;
+ else if (line[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(f,
+ optarg,
+ "invalid option type (use [+-]",
+ line,
+ lineno);
+
+ objecttype = line[1];
+ objectname = &line[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(f,
+ optarg,
+ "missing object name",
+ line,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt.include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt.include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(f,
+ optarg,
+ "include filter is not supported for this type of object",
+ line,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(f,
+ optarg,
+ "exclude filter is not supported for this type of object",
+ line,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(f,
+ optarg,
+ "invalid object type (use [tndf])",
+ line,
+ lineno);
+ }
+
+ if (ferror(f))
+ fatal("could not read from file \"%s\": %m",
+ f == stdin ? "stdin" : optarg);
+
+ if (f != stdin)
+ fclose(f);
+
+ pg_free(line);
+ }
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1022,6 +1160,7 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME read object name filter expressions from file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18624,3 +18763,67 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * getline is originaly GNU function, and should not be everywhere still.
+ * Use own reduced implementation.
+ */
+static size_t
+pg_getline(char **lineptr, size_t *n, FILE *fp)
+{
+ size_t total_chars = 0;
+
+ while (!feof(fp) && !ferror(fp))
+ {
+ char *str;
+ size_t chars;
+
+ str = fgets(*lineptr + total_chars,
+ *n - total_chars,
+ fp);
+
+ if (ferror(fp))
+ return -1;
+
+ if (str)
+ {
+ chars = strlen(str);
+ total_chars += chars;
+
+ if (chars > 0 && str[chars - 1] == '\n')
+ return total_chars;
+
+ /* check, if there is good enough space for next content */
+ if (*n - total_chars < 2)
+ {
+ *n += 1024;
+ *lineptr = pg_realloc(*lineptr, *n);
+ }
+ }
+ else
+ break;
+ }
+
+ if (ferror(fp))
+ return -1;
+
+ return total_chars > 0 ? total_chars : -1;
+}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ *filename == '-' ? "stdin" : filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-05 20:08 Pavel Stehule <[email protected]>
parent: Justin Pryzby <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-07-05 20:08 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: pgsql-hackers
st 1. 7. 2020 v 23:24 odesílatel Justin Pryzby <[email protected]>
napsal:
> On Thu, Jun 11, 2020 at 09:36:18AM +0200, Pavel Stehule wrote:
> > st 10. 6. 2020 v 0:30 odesílatel Justin Pryzby <[email protected]>
> napsal:
> > > > + /* ignore empty rows */
> > > > + if (*line != '\0')
> > >
> > > Maybe: if line=='\0': continue
> > > We should also support comments.
>
> Comment support is still missing but easily added :)
>
> I tried this patch and it works for my purposes.
>
> Also, your getline is dynamically re-allocating lines of arbitrary length.
> Possibly that's not needed. We'll typically read "+t schema.relname",
> which is
> 132 chars. Maybe it's sufficient to do
> char buf[1024];
> fgets(buf);
> if strchr(buf, '\n') == NULL: error();
> ret = pstrdup(buf);
>
63 bytes is max effective identifier size, but it is not max size of
identifiers. It is very probably so buff with 1024 bytes will be enough for
all, but I do not want to increase any new magic limit. More when dynamic
implementation is not too hard.
Table name can be very long - sometimes the data names (table names) can be
stored in external storages with full length and should not be practical to
require truncating in filter file.
For this case it is very effective, because a resized (increased) buffer is
used for following rows, so realloc should not be often. So when I have to
choose between two implementations with similar complexity, I prefer more
dynamic code without hardcoded limits. This dynamic hasn't any overhead.
> In any case, you could have getline return a char* and (rather than
> following
> GNU) no need to take char**, int* parameters to conflate inputs and
> outputs.
>
no, it has a special benefit. It eliminates the short malloc/free cycle.
When some lines are longer, then the buffer is increased (and limits), and
for other rows with same or less size is not necessary realloc.
> I realized that --filter has an advantage over the previous implementation
> (with multiple --exclude-* and --include-*) in that it's possible to use
> stdin
> for includes *and* excludes.
>
yes, it looks like better choose
> By chance, I had the opportunity yesterday to re-use with rsync a regex
> that
> I'd previously been using with pg_dump and grep. What this patch calls
> "--filter" in rsync is called "--filter-from". rsync's --filter-from
> rejects
> filters of length longer than max filename, so I had to split it up into
> multiple lines instead of using regex alternation ("|"). This option is a
> close parallel in pg_dump.
>
we can talk about option name - maybe "--filter-from" is better than just
"--filter"
Regards
Pavel
>
> --
> Justin
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-05 20:31 Justin Pryzby <[email protected]>
parent: Justin Pryzby <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Justin Pryzby @ 2020-07-05 20:31 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers
On Wed, Jul 01, 2020 at 04:24:52PM -0500, Justin Pryzby wrote:
> On Thu, Jun 11, 2020 at 09:36:18AM +0200, Pavel Stehule wrote:
> > st 10. 6. 2020 v 0:30 odesílatel Justin Pryzby <[email protected]> napsal:
> > > > + /* ignore empty rows */
> > > > + if (*line != '\0')
> > >
> > > Maybe: if line=='\0': continue
> > > We should also support comments.
>
> Comment support is still missing but easily added :)
Still missing from the latest patch.
With some added documentation, I think this can be RfC.
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-05 20:37 Pavel Stehule <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-07-05 20:37 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: pgsql-hackers
ne 5. 7. 2020 v 22:31 odesílatel Justin Pryzby <[email protected]>
napsal:
> On Wed, Jul 01, 2020 at 04:24:52PM -0500, Justin Pryzby wrote:
> > On Thu, Jun 11, 2020 at 09:36:18AM +0200, Pavel Stehule wrote:
> > > st 10. 6. 2020 v 0:30 odesílatel Justin Pryzby <[email protected]>
> napsal:
> > > > > + /* ignore empty rows
> */
> > > > > + if (*line != '\0')
> > > >
> > > > Maybe: if line=='\0': continue
> > > > We should also support comments.
> >
> > Comment support is still missing but easily added :)
>
> Still missing from the latest patch.
>
I can implement a comment support. But I am not sure about the format. The
start can be "--" or classic #.
but "--" can be in this context messy
> With some added documentation, I think this can be RfC.
>
> --
> Justin
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-06 04:34 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-07-06 04:34 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: pgsql-hackers
ne 5. 7. 2020 v 22:37 odesílatel Pavel Stehule <[email protected]>
napsal:
>
>
> ne 5. 7. 2020 v 22:31 odesílatel Justin Pryzby <[email protected]>
> napsal:
>
>> On Wed, Jul 01, 2020 at 04:24:52PM -0500, Justin Pryzby wrote:
>> > On Thu, Jun 11, 2020 at 09:36:18AM +0200, Pavel Stehule wrote:
>> > > st 10. 6. 2020 v 0:30 odesílatel Justin Pryzby <[email protected]>
>> napsal:
>> > > > > + /* ignore empty
>> rows */
>> > > > > + if (*line != '\0')
>> > > >
>> > > > Maybe: if line=='\0': continue
>> > > > We should also support comments.
>> >
>> > Comment support is still missing but easily added :)
>>
>> Still missing from the latest patch.
>>
>
> I can implement a comment support. But I am not sure about the format. The
> start can be "--" or classic #.
>
> but "--" can be in this context messy
>
here is support for comment's line - first char should be #
Regards
Pavel
>
>
>> With some added documentation, I think this can be RfC.
>>
>> --
>> Justin
>>
>
Attachments:
[text/x-patch] pg_dump-filter-20200706.patch (7.2K, ../../CAFj8pRCRQOWLm2LrWcbkcm1Dx7yTuLks1_2VrTJO3PARDNXO3w@mail.gmail.com/3-pg_dump-filter-20200706.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 2f0807e912..332f0c312f 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -813,6 +813,15 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read filters from file. Format "(+|-)(tnfd) objectname:
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--load-via-partition-root</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a41a3db876..0b16d528c2 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -290,6 +290,8 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static size_t pg_getline(char **lineptr, size_t *n, FILE *fp);
+static void exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno);
int
@@ -364,6 +366,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -603,6 +606,145 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ {
+ FILE *f;
+ char *line;
+ ssize_t chars;
+ size_t line_size = 1024;
+ int lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(optarg, "-") != 0)
+ {
+ f = fopen(optarg, "r");
+ if (!f)
+ fatal("could not open the input file \"%s\": %m",
+ optarg);
+ }
+ else
+ f = stdin;
+
+ line = pg_malloc(line_size);
+
+ while ((chars = pg_getline(&line, &line_size, f)) != -1)
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+
+ lineno += 1;
+
+ if (line[chars - 1] == '\n')
+ line[chars - 1] = '\0';
+
+ /* ignore empty rows */
+ if (*line == '\0')
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*line == '#')
+ continue;
+
+ if (chars < 2)
+ exit_invalid_filter_format(f,
+ optarg,
+ "too short line",
+ line,
+ lineno);
+
+ if (line[0] == '+')
+ is_include = true;
+ else if (line[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(f,
+ optarg,
+ "invalid option type (use [+-]",
+ line,
+ lineno);
+
+ objecttype = line[1];
+ objectname = &line[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(f,
+ optarg,
+ "missing object name",
+ line,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt.include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt.include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(f,
+ optarg,
+ "include filter is not supported for this type of object",
+ line,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(f,
+ optarg,
+ "exclude filter is not supported for this type of object",
+ line,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(f,
+ optarg,
+ "invalid object type (use [tndf])",
+ line,
+ lineno);
+ }
+
+ if (ferror(f))
+ fatal("could not read from file \"%s\": %m",
+ f == stdin ? "stdin" : optarg);
+
+ if (f != stdin)
+ fclose(f);
+
+ pg_free(line);
+ }
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1022,6 +1164,7 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME read object name filter expressions from file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18624,3 +18767,67 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * getline is originaly GNU function, and should not be everywhere still.
+ * Use own reduced implementation.
+ */
+static size_t
+pg_getline(char **lineptr, size_t *n, FILE *fp)
+{
+ size_t total_chars = 0;
+
+ while (!feof(fp) && !ferror(fp))
+ {
+ char *str;
+ size_t chars;
+
+ str = fgets(*lineptr + total_chars,
+ *n - total_chars,
+ fp);
+
+ if (ferror(fp))
+ return -1;
+
+ if (str)
+ {
+ chars = strlen(str);
+ total_chars += chars;
+
+ if (chars > 0 && str[chars - 1] == '\n')
+ return total_chars;
+
+ /* check, if there is good enough space for next content */
+ if (*n - total_chars < 2)
+ {
+ *n += 1024;
+ *lineptr = pg_realloc(*lineptr, *n);
+ }
+ }
+ else
+ break;
+ }
+
+ if (ferror(fp))
+ return -1;
+
+ return total_chars > 0 ? total_chars : -1;
+}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ *filename == '-' ? "stdin" : filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-11 23:06 Justin Pryzby <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Justin Pryzby @ 2020-07-11 23:06 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers
On Mon, Jul 06, 2020 at 06:34:15AM +0200, Pavel Stehule wrote:
> >> > Comment support is still missing but easily added :)
> >> Still missing from the latest patch.
> >
> > I can implement a comment support. But I am not sure about the format. The
>
> here is support for comment's line - first char should be #
Thanks, that's how I assumed it would look.
> >> With some added documentation, I think this can be RfC.
Do you want to add any more documentation ?
Few more things:
> +exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
> +{
> + pg_log_error("invalid format of filter file \"%s\": %s",
> + *filename == '-' ? "stdin" : filename,
> + message);
You refer to as "stdin" any filename beginning with -.
I think you could just print "-" and not "stdin".
In any case, *filename=='-' is wrong since it only checks filename[0].
In a few places you compare ==stdin (which is right).
Also, I think "f" isn't as good a name as "fp".
You're adding 139 lines to a 600 line main(), and no other option is more than
15 lines. Would you put it in a separate function ?
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-12 01:43 vignesh C <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: vignesh C @ 2020-07-12 01:43 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Justin Pryzby <[email protected]>; pgsql-hackers
On Mon, Jul 6, 2020 at 10:05 AM Pavel Stehule <[email protected]> wrote:
>
> here is support for comment's line - first char should be #
>
Few comments:
+ str = fgets(*lineptr + total_chars,
+ *n - total_chars,
+ fp);
+
+ if (ferror(fp))
+ return -1;
Should we include any error message in the above case.
+ else
+ break;
+ }
+
+ if (ferror(fp))
+ return -1;
Similar to above.
+ /* check, if there is good enough space for
next content */
+ if (*n - total_chars < 2)
+ {
+ *n += 1024;
+ *lineptr = pg_realloc(*lineptr, *n);
+ }
We could use a macro in place of 1024.
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+
simple_string_list_append(&table_include_patterns,
+
objectname);
+
dopt.include_everything = false;
+ }
+ else
+
simple_string_list_append(&table_exclude_patterns,
+
objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+
simple_string_list_append(&schema_include_patterns,
+
objectname);
+
dopt.include_everything = false;
+ }
+ else
+
simple_string_list_append(&schema_exclude_patterns,
+
objectname);
+ }
Some of the above code is repetitive in above, can the common code be
made into a macro and called?
printf(_(" --extra-float-digits=NUM override default
setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME read object name
filter expressions from file\n"));
printf(_(" --if-exists use IF EXISTS when
dropping objects\n"));
Can this be changed to dump objects and data based on the filter
expressions from the filter file.
Regards,
Vignesh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-13 06:15 Pavel Stehule <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-07-13 06:15 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: pgsql-hackers
ne 12. 7. 2020 v 1:06 odesílatel Justin Pryzby <[email protected]>
napsal:
> On Mon, Jul 06, 2020 at 06:34:15AM +0200, Pavel Stehule wrote:
> > >> > Comment support is still missing but easily added :)
> > >> Still missing from the latest patch.
> > >
> > > I can implement a comment support. But I am not sure about the format.
> The
> >
> > here is support for comment's line - first char should be #
>
> Thanks, that's how I assumed it would look.
>
> > >> With some added documentation, I think this can be RfC.
>
> Do you want to add any more documentation ?
>
done
> Few more things:
>
> > +exit_invalid_filter_format(FILE *fp, char *filename, char *message,
> char *line, int lineno)
> > +{
> > + pg_log_error("invalid format of filter file \"%s\": %s",
> > + *filename == '-' ? "stdin" : filename,
> > + message);
>
> You refer to as "stdin" any filename beginning with -.
>
> I think you could just print "-" and not "stdin".
> In any case, *filename=='-' is wrong since it only checks filename[0].
> In a few places you compare ==stdin (which is right).
>
done
> Also, I think "f" isn't as good a name as "fp".
>
done
> You're adding 139 lines to a 600 line main(), and no other option is more
> than
> 15 lines. Would you put it in a separate function ?
>
done
please, check attached patch
Regards
Pavel
> --
> Justin
>
Attachments:
[text/x-patch] pg_dump-filter-20200713-1.patch (9.7K, ../../CAFj8pRAPjbjjZJ5t1-sp4X1o2fgz1n+nLV044hxJFA4uEfarfw@mail.gmail.com/3-pg_dump-filter-20200713-1.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..2f2bfb4dbf 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -755,6 +755,99 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ This option ensure reading object's filters from specified file.
+ If you use "-" as filename, then stdin is used as source. This file
+ has to have following line format:
+<synopsis>
+(+|-)[tnfd] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ Only one object name can be specified per one line:
+<programlisting>
++t mytable1
++t mytable2
++f some_foreign_table
+-d mytable3
+</programlisting>
+ With this file the dump ensures dump table <literal>mytable1</literal>,
+ <literal>mytable2</literal>. The data of foreign table
+ <literal>some_foreign_table</literal> will be dumped too. And the data
+ of <literal>mytable3</literal> will not be dumped.
+ </para>
+
+ <para>
+ The first char <literal>+</literal> or <literal>-</literal> specifies
+ if object name will be used as include or exclude filter.
+ </para>
+
+ <para>
+ The second char
+ <literal>t</literal>,
+ <literal>n</literal>,
+ <literal>f</literal>,
+ <literal>d</literal>
+ specifies a object type.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <listitem>
+ <para>
+ In inclusive form (<literal>+</literal>) it does same work like
+ <option>--table</option>. In exclusive form (<literal>-</literal>)
+ it is same like <option>--exclude-table</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>n</literal></term>
+ <listitem>
+ <para>
+ In inclusive form (<literal>+</literal>) it does same work like
+ <option>--schema</option>. In exclusive form (<literal>-</literal>)
+ it is same like <option>--exclude-schema</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>f</literal></term>
+ <listitem>
+ <para>
+ In inclusive form (<literal>+</literal>) it does same work like
+ <option>--include-foreign-data</option>. The exclusive form
+ (<literal>-</literal>) is not allowed.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <listitem>
+ <para>
+ The inclusive form (<literal>+</literal>) is not allowed.
+ In exclusive form (<literal>-</literal>) it is same like
+ <option>--exclude-table-data</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ The option <option>--filter</option> can be used together with options
+ <option>--table</option>, <option>--exclude-table</option>,
+ <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--include-foreign-data</option> and
+ <option>--exclude-table-data</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e758b5c50a..2fe6db9b05 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -290,6 +290,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
@@ -364,6 +365,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -603,6 +605,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1022,6 +1028,7 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME read object name filter expressions from file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18597,3 +18604,208 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * getline is originaly GNU function, and should not be everywhere still.
+ * Use own reduced implementation.
+ */
+static size_t
+pg_getline(char **lineptr, size_t *n, FILE *fp)
+{
+ size_t total_chars = 0;
+
+ while (!feof(fp) && !ferror(fp))
+ {
+ char *str;
+ size_t chars;
+
+ str = fgets(*lineptr + total_chars,
+ *n - total_chars,
+ fp);
+
+ if (ferror(fp))
+ return -1;
+
+ if (str)
+ {
+ chars = strlen(str);
+ total_chars += chars;
+
+ if (chars > 0 && str[chars - 1] == '\n')
+ return total_chars;
+
+ /* check, if there is good enough space for next content */
+ if (*n - total_chars < 2)
+ {
+ *n += 1024;
+ *lineptr = pg_realloc(*lineptr, *n);
+ }
+ }
+ else
+ break;
+ }
+
+ if (ferror(fp))
+ return -1;
+
+ return total_chars > 0 ? total_chars : -1;
+}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ char *line;
+ ssize_t chars;
+ size_t line_size = 1024;
+ int lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ line = pg_malloc(line_size);
+
+ while ((chars = pg_getline(&line, &line_size, fp)) != -1)
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+
+ lineno += 1;
+
+ if (line[chars - 1] == '\n')
+ line[chars - 1] = '\0';
+
+ /* ignore empty rows */
+ if (*line == '\0')
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*line == '#')
+ continue;
+
+ if (chars < 2)
+ exit_invalid_filter_format(fp,
+ filename,
+ "too short line",
+ line,
+ lineno);
+
+ if (line[0] == '+')
+ is_include = true;
+ else if (line[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid option type (use [+-]",
+ line,
+ lineno);
+
+ objecttype = line[1];
+ objectname = &line[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ line,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ line,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ line,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid object type (use [tndf])",
+ line,
+ lineno);
+ }
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ pg_free(line);
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* [PATCH 1/2] proposal: possibility to read dumped table's name from file
@ 2020-07-13 08:20 Pavel Stehule <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-07-13 08:20 UTC (permalink / raw)
---
doc/src/sgml/ref/pg_dump.sgml | 93 +++++++++++++++
src/bin/pg_dump/pg_dump.c | 215 ++++++++++++++++++++++++++++++++++
2 files changed, 308 insertions(+)
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..2f2bfb4dbf 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -755,6 +755,99 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ This option ensure reading object's filters from specified file.
+ If you use "-" as filename, then stdin is used as source. This file
+ has to have following line format:
+<synopsis>
+(+|-)[tnfd] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ Only one object name can be specified per one line:
+<programlisting>
++t mytable1
++t mytable2
++f some_foreign_table
+-d mytable3
+</programlisting>
+ With this file the dump ensures dump table <literal>mytable1</literal>,
+ <literal>mytable2</literal>. The data of foreign table
+ <literal>some_foreign_table</literal> will be dumped too. And the data
+ of <literal>mytable3</literal> will not be dumped.
+ </para>
+
+ <para>
+ The first char <literal>+</literal> or <literal>-</literal> specifies
+ if object name will be used as include or exclude filter.
+ </para>
+
+ <para>
+ The second char
+ <literal>t</literal>,
+ <literal>n</literal>,
+ <literal>f</literal>,
+ <literal>d</literal>
+ specifies a object type.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <listitem>
+ <para>
+ In inclusive form (<literal>+</literal>) it does same work like
+ <option>--table</option>. In exclusive form (<literal>-</literal>)
+ it is same like <option>--exclude-table</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>n</literal></term>
+ <listitem>
+ <para>
+ In inclusive form (<literal>+</literal>) it does same work like
+ <option>--schema</option>. In exclusive form (<literal>-</literal>)
+ it is same like <option>--exclude-schema</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>f</literal></term>
+ <listitem>
+ <para>
+ In inclusive form (<literal>+</literal>) it does same work like
+ <option>--include-foreign-data</option>. The exclusive form
+ (<literal>-</literal>) is not allowed.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <listitem>
+ <para>
+ The inclusive form (<literal>+</literal>) is not allowed.
+ In exclusive form (<literal>-</literal>) it is same like
+ <option>--exclude-table-data</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ The option <option>--filter</option> can be used together with options
+ <option>--table</option>, <option>--exclude-table</option>,
+ <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--include-foreign-data</option> and
+ <option>--exclude-table-data</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e758b5c50a..fd6b7a174a 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -290,6 +290,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
@@ -364,6 +365,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -603,6 +605,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1022,6 +1028,7 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME read object name filter expressions from file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18597,3 +18604,211 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+#define FILTER_INITIAL_LINE_SIZE 1024
+#define PG_GETLINE_EXTEND_LINE_SIZE 1024
+
+/*
+ * getline is originaly GNU function, and should not be everywhere still.
+ * Use own reduced implementation.
+ */
+static size_t
+pg_getline(char **lineptr, size_t *n, FILE *fp)
+{
+ size_t total_chars = 0;
+
+ while (!feof(fp) && !ferror(fp))
+ {
+ char *str;
+ size_t chars;
+
+ str = fgets(*lineptr + total_chars,
+ *n - total_chars,
+ fp);
+
+ if (ferror(fp))
+ return -1;
+
+ if (str)
+ {
+ chars = strlen(str);
+ total_chars += chars;
+
+ if (chars > 0 && str[chars - 1] == '\n')
+ return total_chars;
+
+ /* check, if there is good enough space for next content */
+ if (*n - total_chars < 2)
+ {
+ *n += PG_GETLINE_EXTEND_LINE_SIZE;
+ *lineptr = pg_realloc(*lineptr, *n);
+ }
+ }
+ else
+ break;
+ }
+
+ if (ferror(fp))
+ return -1;
+
+ return total_chars > 0 ? total_chars : -1;
+}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ char *line;
+ ssize_t chars;
+ size_t line_size = FILTER_INITIAL_LINE_SIZE;
+ int lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ line = pg_malloc(line_size);
+
+ while ((chars = pg_getline(&line, &line_size, fp)) != -1)
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+
+ lineno += 1;
+
+ if (line[chars - 1] == '\n')
+ line[chars - 1] = '\0';
+
+ /* ignore empty rows */
+ if (*line == '\0')
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*line == '#')
+ continue;
+
+ if (chars < 2)
+ exit_invalid_filter_format(fp,
+ filename,
+ "too short line",
+ line,
+ lineno);
+
+ if (line[0] == '+')
+ is_include = true;
+ else if (line[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid option type (use [+-]",
+ line,
+ lineno);
+
+ objecttype = line[1];
+ objectname = &line[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ line,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ line,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ line,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid object type (use [tndf])",
+ line,
+ lineno);
+ }
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ pg_free(line);
+}
--
2.17.0
--H4SyuGOnfnj3aJqJ
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="0002-fixen.patch"
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-13 08:20 Pavel Stehule <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-07-13 08:20 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Justin Pryzby <[email protected]>; pgsql-hackers
ne 12. 7. 2020 v 3:43 odesílatel vignesh C <[email protected]> napsal:
> On Mon, Jul 6, 2020 at 10:05 AM Pavel Stehule <[email protected]>
> wrote:
> >
> > here is support for comment's line - first char should be #
> >
>
> Few comments:
> + str = fgets(*lineptr + total_chars,
> + *n - total_chars,
> + fp);
> +
> + if (ferror(fp))
> + return -1;
>
> Should we include any error message in the above case.
>
> + else
> + break;
> + }
> +
> + if (ferror(fp))
> + return -1;
>
> Similar to above.
>
it should be ok, both variant finishing by
<-->if (ferror(fp))
<--><-->fatal("could not read from file \"%s\": %m", filename);
%m should to print related error message
>
> + /* check, if there is good enough space for
> next content */
> + if (*n - total_chars < 2)
> + {
> + *n += 1024;
> + *lineptr = pg_realloc(*lineptr, *n);
> + }
> We could use a macro in place of 1024.
>
done
> + if (objecttype == 't')
> + {
> + if (is_include)
> + {
> +
> simple_string_list_append(&table_include_patterns,
> +
> objectname);
> +
> dopt.include_everything = false;
> + }
> + else
> +
> simple_string_list_append(&table_exclude_patterns,
> +
> objectname);
> + }
> + else if (objecttype == 'n')
> + {
> + if (is_include)
> + {
> +
> simple_string_list_append(&schema_include_patterns,
> +
> objectname);
> +
> dopt.include_everything = false;
> + }
> + else
> +
> simple_string_list_append(&schema_exclude_patterns,
> +
> objectname);
> + }
> Some of the above code is repetitive in above, can the common code be
> made into a macro and called?
>
There are two same fragments and two different fragments. In this case I
don't think so using macro or auxiliary function can help with readability.
Current code is well structured and well readable.
>
> printf(_(" --extra-float-digits=NUM override default
> setting for extra_float_digits\n"));
> + printf(_(" --filter=FILENAME read object name
> filter expressions from file\n"));
> printf(_(" --if-exists use IF EXISTS when
> dropping objects\n"));
> Can this be changed to dump objects and data based on the filter
> expressions from the filter file.
>
I am sorry, I don't understand. This should work for data from specified by
filter without any modification.
attached updated patch
Regards
Pavel
> Regards,
> Vignesh
> EnterpriseDB: http://www.enterprisedb.com
>
Attachments:
[text/x-patch] pg_dump-filter-20200713-2.patch (9.8K, ../../CAFj8pRDU-CsvKjxEZ=SvuVjymFSEdS1pgLzxB7RTWSWnnqcLAA@mail.gmail.com/3-pg_dump-filter-20200713-2.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..2f2bfb4dbf 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -755,6 +755,99 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ This option ensure reading object's filters from specified file.
+ If you use "-" as filename, then stdin is used as source. This file
+ has to have following line format:
+<synopsis>
+(+|-)[tnfd] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ Only one object name can be specified per one line:
+<programlisting>
++t mytable1
++t mytable2
++f some_foreign_table
+-d mytable3
+</programlisting>
+ With this file the dump ensures dump table <literal>mytable1</literal>,
+ <literal>mytable2</literal>. The data of foreign table
+ <literal>some_foreign_table</literal> will be dumped too. And the data
+ of <literal>mytable3</literal> will not be dumped.
+ </para>
+
+ <para>
+ The first char <literal>+</literal> or <literal>-</literal> specifies
+ if object name will be used as include or exclude filter.
+ </para>
+
+ <para>
+ The second char
+ <literal>t</literal>,
+ <literal>n</literal>,
+ <literal>f</literal>,
+ <literal>d</literal>
+ specifies a object type.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <listitem>
+ <para>
+ In inclusive form (<literal>+</literal>) it does same work like
+ <option>--table</option>. In exclusive form (<literal>-</literal>)
+ it is same like <option>--exclude-table</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>n</literal></term>
+ <listitem>
+ <para>
+ In inclusive form (<literal>+</literal>) it does same work like
+ <option>--schema</option>. In exclusive form (<literal>-</literal>)
+ it is same like <option>--exclude-schema</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>f</literal></term>
+ <listitem>
+ <para>
+ In inclusive form (<literal>+</literal>) it does same work like
+ <option>--include-foreign-data</option>. The exclusive form
+ (<literal>-</literal>) is not allowed.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <listitem>
+ <para>
+ The inclusive form (<literal>+</literal>) is not allowed.
+ In exclusive form (<literal>-</literal>) it is same like
+ <option>--exclude-table-data</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ The option <option>--filter</option> can be used together with options
+ <option>--table</option>, <option>--exclude-table</option>,
+ <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--include-foreign-data</option> and
+ <option>--exclude-table-data</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e758b5c50a..fd6b7a174a 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -290,6 +290,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
@@ -364,6 +365,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -603,6 +605,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1022,6 +1028,7 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME read object name filter expressions from file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18597,3 +18604,211 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+#define FILTER_INITIAL_LINE_SIZE 1024
+#define PG_GETLINE_EXTEND_LINE_SIZE 1024
+
+/*
+ * getline is originaly GNU function, and should not be everywhere still.
+ * Use own reduced implementation.
+ */
+static size_t
+pg_getline(char **lineptr, size_t *n, FILE *fp)
+{
+ size_t total_chars = 0;
+
+ while (!feof(fp) && !ferror(fp))
+ {
+ char *str;
+ size_t chars;
+
+ str = fgets(*lineptr + total_chars,
+ *n - total_chars,
+ fp);
+
+ if (ferror(fp))
+ return -1;
+
+ if (str)
+ {
+ chars = strlen(str);
+ total_chars += chars;
+
+ if (chars > 0 && str[chars - 1] == '\n')
+ return total_chars;
+
+ /* check, if there is good enough space for next content */
+ if (*n - total_chars < 2)
+ {
+ *n += PG_GETLINE_EXTEND_LINE_SIZE;
+ *lineptr = pg_realloc(*lineptr, *n);
+ }
+ }
+ else
+ break;
+ }
+
+ if (ferror(fp))
+ return -1;
+
+ return total_chars > 0 ? total_chars : -1;
+}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ char *line;
+ ssize_t chars;
+ size_t line_size = FILTER_INITIAL_LINE_SIZE;
+ int lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ line = pg_malloc(line_size);
+
+ while ((chars = pg_getline(&line, &line_size, fp)) != -1)
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+
+ lineno += 1;
+
+ if (line[chars - 1] == '\n')
+ line[chars - 1] = '\0';
+
+ /* ignore empty rows */
+ if (*line == '\0')
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*line == '#')
+ continue;
+
+ if (chars < 2)
+ exit_invalid_filter_format(fp,
+ filename,
+ "too short line",
+ line,
+ lineno);
+
+ if (line[0] == '+')
+ is_include = true;
+ else if (line[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid option type (use [+-]",
+ line,
+ lineno);
+
+ objecttype = line[1];
+ objectname = &line[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ line,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ line,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ line,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid object type (use [tndf])",
+ line,
+ lineno);
+ }
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ pg_free(line);
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-13 10:04 Daniel Gustafsson <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 2 replies; 181+ messages in thread
From: Daniel Gustafsson @ 2020-07-13 10:04 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> On 13 Jul 2020, at 10:20, Pavel Stehule <[email protected]> wrote:
> attached updated patch
Sorry for jumping in late, but thinking about this extension to pg_dump:
doesn't it make more sense to use an existing file format like JSON for this,
given that virtually all devops/cd/etc tooling know about JSON already?
Considering its application and the problem space, I'd expect users to generate
this file rather than handcraft it with 10 rows of content, and standard file
formats help there. Creative users could even use the database itself to
easily manage its content and generate the file (which isn't limited to JSON of
course, but it would be easier). Also, we now have backup manifests in JSON
which IMO sets a bit of a precedent, even though thats a separate thing.
At the very least it seems limiting to not include a file format version
identifier since we'd otherwise risk running into backwards compat issues
should we want to expand on this in the future.
cheers ./daniel
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-13 11:02 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-07-13 11:02 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
po 13. 7. 2020 v 12:04 odesílatel Daniel Gustafsson <[email protected]>
napsal:
> > On 13 Jul 2020, at 10:20, Pavel Stehule <[email protected]> wrote:
>
> > attached updated patch
>
> Sorry for jumping in late, but thinking about this extension to pg_dump:
> doesn't it make more sense to use an existing file format like JSON for
> this,
> given that virtually all devops/cd/etc tooling know about JSON already?
>
> Considering its application and the problem space, I'd expect users to
> generate
> this file rather than handcraft it with 10 rows of content, and standard
> file
> formats help there. Creative users could even use the database itself to
> easily manage its content and generate the file (which isn't limited to
> JSON of
> course, but it would be easier). Also, we now have backup manifests in
> JSON
> which IMO sets a bit of a precedent, even though thats a separate thing.
>
> At the very least it seems limiting to not include a file format version
> identifier since we'd otherwise risk running into backwards compat issues
> should we want to expand on this in the future.
>
I like JSON format. But why here? For this purpose the JSON is over
engineered. This input file has no nested structure - it is just a stream
of lines.
I don't think so introducing JSON here can be a good idea. For this feature
typical usage can be used in pipe, and the most simple format (what is
possible) is ideal.
It is a really different case than pg_dump manifest file - in this case, in
this case pg_dump is consument.
Regards
Pavel
> cheers ./daniel
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-13 13:32 vignesh C <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: vignesh C @ 2020-07-13 13:32 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Justin Pryzby <[email protected]>; pgsql-hackers
On Mon, Jul 13, 2020 at 1:51 PM Pavel Stehule <[email protected]> wrote:
>> Can this be changed to dump objects and data based on the filter
>> expressions from the filter file.
>
>
> I am sorry, I don't understand. This should work for data from specified by filter without any modification.
>
I meant can this:
printf(_(" --filter=FILENAME read object name filter
expressions from file\n"));
be changed to:
printf(_(" --filter=FILENAME dump objects and data based
on the filter expressions from the filter file\n"));
Regards,
Vignesh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-13 14:32 Justin Pryzby <[email protected]>
parent: Daniel Gustafsson <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Justin Pryzby @ 2020-07-13 14:32 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Pavel Stehule <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Mon, Jul 13, 2020 at 12:04:09PM +0200, Daniel Gustafsson wrote:
> Sorry for jumping in late, but thinking about this extension to pg_dump:
> doesn't it make more sense to use an existing file format like JSON for this,
> given that virtually all devops/cd/etc tooling know about JSON already?
>
> Considering its application and the problem space, I'd expect users to generate
> this file rather than handcraft it with 10 rows of content, and standard file
> formats help there.
I mentioned having tested this patch as we would use it. But it's likely I
*wouldn't* use it if the format was something which required added complexity
to pipe in from an existing shell script.
> At the very least it seems limiting to not include a file format version
> identifier since we'd otherwise risk running into backwards compat issues
> should we want to expand on this in the future.
Maybe .. I'm not sure. The patch would of course be extended to handle
additional include/exclude options. Is there any other future behavior we
might reasonably anticipate ?
If at some point we wanted to support another file format, maybe it would look
like: --format=v2:filters.txt (or maybe the old one would be v1:filters.txt)
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-13 14:57 Daniel Gustafsson <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Daniel Gustafsson @ 2020-07-13 14:57 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
> On 13 Jul 2020, at 13:02, Pavel Stehule <[email protected]> wrote:
> I like JSON format. But why here? For this purpose the JSON is over engineered.
I respectfully disagree, JSON is a commonly used and known format in systems
administration and most importantly: we already have code to parse it in the
frontend.
> This input file has no nested structure - it is just a stream of lines.
Well, it has a set of object types which in turn have objects. There is more
structure than meets the eye.
Also, the current patch allows arbitrary whitespace before object names, but no
whitespace before comments etc. Using something where the rules of parsing are
known is rarely a bad thing.
> I don't think so introducing JSON here can be a good idea.
Quite possibly it isn't, but not discussing options seems like a worse idea so
I wanted to bring it up.
> It is a really different case than pg_dump manifest file - in this case, in this case pg_dump is consument.
Right, as I said these are two different, while tangentially related, things.
cheers ./daniel
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-13 15:33 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-07-13 15:33 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
po 13. 7. 2020 v 16:57 odesílatel Daniel Gustafsson <[email protected]>
napsal:
> > On 13 Jul 2020, at 13:02, Pavel Stehule <[email protected]> wrote:
>
> > I like JSON format. But why here? For this purpose the JSON is over
> engineered.
>
> I respectfully disagree, JSON is a commonly used and known format in
> systems
> administration and most importantly: we already have code to parse it in
> the
> frontend.
>
I disagree with the idea so if we have a client side JSON parser we have
to use it everywhere.
For this case, parsing JSON means more code, not less. I checked the
parse_manifest.c. More
the JSON API is DOM type. For this purpose the SAX type is better. But
still, things should be simple as possible.
There is not any necessity to use it.
JSON is good for a lot of purposes, and can be good if the document uses
more lexer types, numeric, ... But nothing is used there
> > This input file has no nested structure - it is just a stream of lines.
>
> Well, it has a set of object types which in turn have objects. There is
> more
> structure than meets the eye.
>
> Also, the current patch allows arbitrary whitespace before object names,
> but no
> whitespace before comments etc. Using something where the rules of
> parsing are
> known is rarely a bad thing.
>
if I know - JSON hasn't comments at all.
> > I don't think so introducing JSON here can be a good idea.
>
> Quite possibly it isn't, but not discussing options seems like a worse
> idea so
> I wanted to bring it up.
>
> > It is a really different case than pg_dump manifest file - in this case,
> in this case pg_dump is consument.
>
> Right, as I said these are two different, while tangentially related,
> things.
>
Backup manifest format has no trivial complexity - and using JSON has
sense. Input filter file is a trivial - +/- list of strings (and it will be
everytime).
In this case I don't see any benefits from JSON - on both sides (producent,
consuments). It is harder (little bit) to parse it, it is harder (little
bit) to generate it.
Regards
Pavel
> cheers ./daniel
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-13 17:59 Justin Pryzby <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Justin Pryzby @ 2020-07-13 17:59 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers
On Mon, Jul 13, 2020 at 08:15:42AM +0200, Pavel Stehule wrote:
> > Do you want to add any more documentation ?
> >
>
> done
Thanks - I think the documentation was maybe excessive. See attached.
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-14 06:31 Pavel Stehule <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-07-14 06:31 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; vignesh C <[email protected]>; +Cc: pgsql-hackers
po 13. 7. 2020 v 20:00 odesílatel Justin Pryzby <[email protected]>
napsal:
> On Mon, Jul 13, 2020 at 08:15:42AM +0200, Pavel Stehule wrote:
> > > Do you want to add any more documentation ?
> > >
> >
> > done
>
> Thanks - I think the documentation was maybe excessive. See attached.
>
I merged your patch - thank you
new patch with doc changes and text of help change requested by Vignesh
attached
Regards
Pavel
> --
> Justin
>
Attachments:
[text/x-patch] pg_dump-filter-20200714.patch (8.3K, ../../CAFj8pRAGNDe3QQKgYV7GM2PAPO53br60wbimdNAz30_OdhBR3A@mail.gmail.com/3-pg_dump-filter-20200714.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..c9502dca13 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -755,6 +755,50 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file.
+ If you use "-" as a filename, the filters are read from stdin.
+ The lines of this file must have the following format:
+<synopsis>
+(+|-)[tnfd] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first character specifies whether the object is to be included
+ (<literal>+</literal>) or excluded (<literal>-</literal>), and the
+ second character specifies the type of object to be filtered:
+ <literal>t</literal> (table),
+ <literal>n</literal> (schema),
+ <literal>f</literal> (foreign table),
+ <literal>d</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign table
+ <literal>some_foreign_table</literal>, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
++t mytable1
++f some_foreign_table
+-d mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e758b5c50a..778ae2d0fc 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -290,6 +290,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
@@ -364,6 +365,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -603,6 +605,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1022,6 +1028,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18597,3 +18605,211 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+#define FILTER_INITIAL_LINE_SIZE 1024
+#define PG_GETLINE_EXTEND_LINE_SIZE 1024
+
+/*
+ * getline is originally GNU function, and should not be everywhere still.
+ * Use own reduced implementation.
+ */
+static size_t
+pg_getline(char **lineptr, size_t *n, FILE *fp)
+{
+ size_t total_chars = 0;
+
+ while (!feof(fp) && !ferror(fp))
+ {
+ char *str;
+ size_t chars;
+
+ str = fgets(*lineptr + total_chars,
+ *n - total_chars,
+ fp);
+
+ if (ferror(fp))
+ return -1;
+
+ if (str)
+ {
+ chars = strlen(str);
+ total_chars += chars;
+
+ if (chars > 0 && str[chars - 1] == '\n')
+ return total_chars;
+
+ /* check, if there is good enough space for next content */
+ if (*n - total_chars < 2)
+ {
+ *n += PG_GETLINE_EXTEND_LINE_SIZE;
+ *lineptr = pg_realloc(*lineptr, *n);
+ }
+ }
+ else
+ break;
+ }
+
+ if (ferror(fp))
+ return -1;
+
+ return total_chars > 0 ? total_chars : -1;
+}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ char *line;
+ ssize_t chars;
+ size_t line_size = FILTER_INITIAL_LINE_SIZE;
+ int lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ line = pg_malloc(line_size);
+
+ while ((chars = pg_getline(&line, &line_size, fp)) != -1)
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+
+ lineno += 1;
+
+ if (line[chars - 1] == '\n')
+ line[chars - 1] = '\0';
+
+ /* ignore empty rows */
+ if (*line == '\0')
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*line == '#')
+ continue;
+
+ if (chars < 2)
+ exit_invalid_filter_format(fp,
+ filename,
+ "line too short",
+ line,
+ lineno);
+
+ if (line[0] == '+')
+ is_include = true;
+ else if (line[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid option type (use [+-]",
+ line,
+ lineno);
+
+ objecttype = line[1];
+ objectname = &line[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ line,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ line,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ line,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid object type (use [tndf])",
+ line,
+ lineno);
+ }
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ pg_free(line);
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-14 06:32 Pavel Stehule <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-07-14 06:32 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Justin Pryzby <[email protected]>; pgsql-hackers
po 13. 7. 2020 v 15:33 odesílatel vignesh C <[email protected]> napsal:
> On Mon, Jul 13, 2020 at 1:51 PM Pavel Stehule <[email protected]>
> wrote:
> >> Can this be changed to dump objects and data based on the filter
> >> expressions from the filter file.
> >
> >
> > I am sorry, I don't understand. This should work for data from specified
> by filter without any modification.
> >
> I meant can this:
> printf(_(" --filter=FILENAME read object name filter
> expressions from file\n"));
> be changed to:
> printf(_(" --filter=FILENAME dump objects and data based
> on the filter expressions from the filter file\n"));
>
done in today patch
Pavel
> Regards,
> Vignesh
> EnterpriseDB: http://www.enterprisedb.com
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-25 13:26 vignesh C <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: vignesh C @ 2020-07-25 13:26 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Justin Pryzby <[email protected]>; pgsql-hackers
On Tue, Jul 14, 2020 at 12:03 PM Pavel Stehule <[email protected]> wrote:
>> I meant can this:
>> printf(_(" --filter=FILENAME read object name filter
>> expressions from file\n"));
>> be changed to:
>> printf(_(" --filter=FILENAME dump objects and data based
>> on the filter expressions from the filter file\n"));
>
> done in today patch
>
Thanks for fixing the comments.
Few comments:
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
We could use STDIN itself instead of -, it will be a more easier
option to understand.
+ /* when first char is hash, ignore whole line */
+ if (*line == '#')
+ continue;
If line starts with # we ignore that line, I feel this should be
included in the documentation.
Regards,
Vignesh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-26 19:10 Justin Pryzby <[email protected]>
parent: vignesh C <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Justin Pryzby @ 2020-07-26 19:10 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Pavel Stehule <[email protected]>; pgsql-hackers
On Sat, Jul 25, 2020 at 06:56:31PM +0530, vignesh C wrote:
> On Tue, Jul 14, 2020 at 12:03 PM Pavel Stehule <[email protected]> wrote:
> >> I meant can this:
> >> printf(_(" --filter=FILENAME read object name filter
> >> expressions from file\n"));
> >> be changed to:
> >> printf(_(" --filter=FILENAME dump objects and data based
> >> on the filter expressions from the filter file\n"));
> >
> > done in today patch
>
> Thanks for fixing the comments.
> Few comments:
> + /* use "-" as symbol for stdin */
> + if (strcmp(filename, "-") != 0)
> + {
> + fp = fopen(filename, "r");
> + if (!fp)
> + fatal("could not open the input file \"%s\": %m",
> + filename);
> + }
> + else
> + fp = stdin;
>
> We could use STDIN itself instead of -, it will be a more easier
> option to understand.
I think "-" is used widely for commandline tools, and STDIN is not (even though
it's commonly used by programmers). For example, since last year, pg_restore
-f - means stdout.
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-27 05:25 Pavel Stehule <[email protected]>
parent: vignesh C <[email protected]>
1 sibling, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-07-27 05:25 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Justin Pryzby <[email protected]>; pgsql-hackers
so 25. 7. 2020 v 15:26 odesílatel vignesh C <[email protected]> napsal:
> On Tue, Jul 14, 2020 at 12:03 PM Pavel Stehule <[email protected]>
> wrote:
> >> I meant can this:
> >> printf(_(" --filter=FILENAME read object name filter
> >> expressions from file\n"));
> >> be changed to:
> >> printf(_(" --filter=FILENAME dump objects and data based
> >> on the filter expressions from the filter file\n"));
> >
> > done in today patch
> >
>
> Thanks for fixing the comments.
> Few comments:
> + /* use "-" as symbol for stdin */
> + if (strcmp(filename, "-") != 0)
> + {
> + fp = fopen(filename, "r");
> + if (!fp)
> + fatal("could not open the input file \"%s\": %m",
> + filename);
> + }
> + else
> + fp = stdin;
>
> We could use STDIN itself instead of -, it will be a more easier
> option to understand.
>
> + /* when first char is hash, ignore whole line */
> + if (*line == '#')
> + continue;
>
> If line starts with # we ignore that line, I feel this should be
> included in the documentation.
>
Good note - I wrote sentence to doc
+ <para>
+ The lines starting with symbol <literal>#</literal> are ignored.
+ Previous white chars (spaces, tabs) are not allowed. These
+ lines can be used for comments, notes.
+ </para>
+
> Regards,
> Vignesh
> EnterpriseDB: http://www.enterprisedb.com
>
Attachments:
[text/x-patch] pg_dump-filter-20200727.patch (8.5K, ../../CAFj8pRDG_8p1u_p5uJk5iW0xiAGNHaqe5Y4Uaf_d4cvPkcv4_Q@mail.gmail.com/3-pg_dump-filter-20200727.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7a37fd8045..0e0fbc90db 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -755,6 +755,56 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file.
+ If you use "-" as a filename, the filters are read from stdin.
+ The lines of this file must have the following format:
+<synopsis>
+(+|-)[tnfd] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first character specifies whether the object is to be included
+ (<literal>+</literal>) or excluded (<literal>-</literal>), and the
+ second character specifies the type of object to be filtered:
+ <literal>t</literal> (table),
+ <literal>n</literal> (schema),
+ <literal>f</literal> (foreign table),
+ <literal>d</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign table
+ <literal>some_foreign_table</literal>, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
++t mytable1
++f some_foreign_table
+-d mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The lines starting with symbol <literal>#</literal> are ignored.
+ Previous white chars (spaces, tabs) are not allowed. These
+ lines can be used for comments, notes.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 94459b3539..8df1f91cb6 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -290,6 +290,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
@@ -364,6 +365,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -603,6 +605,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1022,6 +1028,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18434,3 +18442,211 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+#define FILTER_INITIAL_LINE_SIZE 1024
+#define PG_GETLINE_EXTEND_LINE_SIZE 1024
+
+/*
+ * getline is originally GNU function, and should not be everywhere still.
+ * Use own reduced implementation.
+ */
+static size_t
+pg_getline(char **lineptr, size_t *n, FILE *fp)
+{
+ size_t total_chars = 0;
+
+ while (!feof(fp) && !ferror(fp))
+ {
+ char *str;
+ size_t chars;
+
+ str = fgets(*lineptr + total_chars,
+ *n - total_chars,
+ fp);
+
+ if (ferror(fp))
+ return -1;
+
+ if (str)
+ {
+ chars = strlen(str);
+ total_chars += chars;
+
+ if (chars > 0 && str[chars - 1] == '\n')
+ return total_chars;
+
+ /* check, if there is good enough space for next content */
+ if (*n - total_chars < 2)
+ {
+ *n += PG_GETLINE_EXTEND_LINE_SIZE;
+ *lineptr = pg_realloc(*lineptr, *n);
+ }
+ }
+ else
+ break;
+ }
+
+ if (ferror(fp))
+ return -1;
+
+ return total_chars > 0 ? total_chars : -1;
+}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ char *line;
+ ssize_t chars;
+ size_t line_size = FILTER_INITIAL_LINE_SIZE;
+ int lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ line = pg_malloc(line_size);
+
+ while ((chars = pg_getline(&line, &line_size, fp)) != -1)
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+
+ lineno += 1;
+
+ if (line[chars - 1] == '\n')
+ line[chars - 1] = '\0';
+
+ /* ignore empty rows */
+ if (*line == '\0')
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*line == '#')
+ continue;
+
+ if (chars < 2)
+ exit_invalid_filter_format(fp,
+ filename,
+ "line too short",
+ line,
+ lineno);
+
+ if (line[0] == '+')
+ is_include = true;
+ else if (line[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid option type (use [+-]",
+ line,
+ lineno);
+
+ objecttype = line[1];
+ objectname = &line[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ line,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ line,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ line,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid object type (use [tndf])",
+ line,
+ lineno);
+ }
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ pg_free(line);
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-07-27 05:28 Pavel Stehule <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-07-27 05:28 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: vignesh C <[email protected]>; pgsql-hackers
ne 26. 7. 2020 v 21:10 odesílatel Justin Pryzby <[email protected]>
napsal:
> On Sat, Jul 25, 2020 at 06:56:31PM +0530, vignesh C wrote:
> > On Tue, Jul 14, 2020 at 12:03 PM Pavel Stehule <[email protected]>
> wrote:
> > >> I meant can this:
> > >> printf(_(" --filter=FILENAME read object name filter
> > >> expressions from file\n"));
> > >> be changed to:
> > >> printf(_(" --filter=FILENAME dump objects and data based
> > >> on the filter expressions from the filter file\n"));
> > >
> > > done in today patch
> >
> > Thanks for fixing the comments.
> > Few comments:
> > + /* use "-" as symbol for stdin */
> > + if (strcmp(filename, "-") != 0)
> > + {
> > + fp = fopen(filename, "r");
> > + if (!fp)
> > + fatal("could not open the input file \"%s\": %m",
> > + filename);
> > + }
> > + else
> > + fp = stdin;
> >
> > We could use STDIN itself instead of -, it will be a more easier
> > option to understand.
>
> I think "-" is used widely for commandline tools, and STDIN is not (even
> though
> it's commonly used by programmers). For example, since last year,
> pg_restore
> -f - means stdout.
>
yes, STDIN is used by programming languages, but it is not usual in command
line tools. And because it was used by pg_restore, then we should not use
new inconsistency.
Regards
Pavel
> --
> Justin
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-08-13 21:41 Justin Pryzby <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Justin Pryzby @ 2020-08-13 21:41 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: vignesh C <[email protected]>; pgsql-hackers
On Mon, Jul 27, 2020 at 07:25:54AM +0200, Pavel Stehule wrote:
> so 25. 7. 2020 v 15:26 odesílatel vignesh C <[email protected]> napsal:
>
> > On Tue, Jul 14, 2020 at 12:03 PM Pavel Stehule <[email protected]>
> > wrote:
> > >> I meant can this:
> > >> printf(_(" --filter=FILENAME read object name filter
> > >> expressions from file\n"));
> > >> be changed to:
> > >> printf(_(" --filter=FILENAME dump objects and data based
> > >> on the filter expressions from the filter file\n"));
> > >
> > > done in today patch
This looks good to my eyes - marked RFC.
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-09-03 19:48 Justin Pryzby <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Justin Pryzby @ 2020-09-03 19:48 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: pgsql-hackers
On Sun, Jul 05, 2020 at 10:08:09PM +0200, Pavel Stehule wrote:
> st 1. 7. 2020 v 23:24 odesílatel Justin Pryzby <[email protected]> napsal:
>
> > On Thu, Jun 11, 2020 at 09:36:18AM +0200, Pavel Stehule wrote:
> > > st 10. 6. 2020 v 0:30 odesílatel Justin Pryzby <[email protected]>> napsal:
> > Also, your getline is dynamically re-allocating lines of arbitrary length.
> > Possibly that's not needed. We'll typically read "+t schema.relname",
> > which is
> > 132 chars. Maybe it's sufficient to do
> > char buf[1024];
> > fgets(buf);
> > if strchr(buf, '\n') == NULL: error();
> > ret = pstrdup(buf);
>
> 63 bytes is max effective identifier size, but it is not max size of
> identifiers. It is very probably so buff with 1024 bytes will be enough for
> all, but I do not want to increase any new magic limit. More when dynamic
> implementation is not too hard.
Maybe you'd want to use a StrInfo like recent patches (8f8154a50).
> Table name can be very long - sometimes the data names (table names) can be
> stored in external storages with full length and should not be practical to
> require truncating in filter file.
>
> For this case it is very effective, because a resized (increased) buffer is
> used for following rows, so realloc should not be often. So when I have to
> choose between two implementations with similar complexity, I prefer more
> dynamic code without hardcoded limits. This dynamic hasn't any overhead.
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-09-03 20:08 Alvaro Herrera <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Alvaro Herrera @ 2020-09-03 20:08 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
On 2020-Jul-27, Pavel Stehule wrote:
> +/*
> + * getline is originally GNU function, and should not be everywhere still.
> + * Use own reduced implementation.
> + */
> +static size_t
> +pg_getline(char **lineptr, size_t *n, FILE *fp)
> +{
So, Tom added a coding pattern for doing this in commit 8f8154a503c7,
which is ostensibly also to be used in pg_regress [1] -- maybe it'd be
useful to have this in src/common?
[1] https://postgr.es/m/m_1NfbowTqSJnrC6rq1a9cQK7E-CHQE7B6Kz9w6fNH-OiV-4mcsdMw7UP2oA2_6dZmXvAMjbSPZjW9U7...
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-09-04 00:15 Tom Lane <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Tom Lane @ 2020-09-04 00:15 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Pavel Stehule <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Alvaro Herrera <[email protected]> writes:
> So, Tom added a coding pattern for doing this in commit 8f8154a503c7,
> which is ostensibly also to be used in pg_regress [1] -- maybe it'd be
> useful to have this in src/common?
Done, see pg_get_line() added by 67a472d71.
regards, tom lane
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-09-04 03:21 Pavel Stehule <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-09-04 03:21 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
pá 4. 9. 2020 v 2:15 odesílatel Tom Lane <[email protected]> napsal:
> Alvaro Herrera <[email protected]> writes:
> > So, Tom added a coding pattern for doing this in commit 8f8154a503c7,
> > which is ostensibly also to be used in pg_regress [1] -- maybe it'd be
> > useful to have this in src/common?
>
> Done, see pg_get_line() added by 67a472d71.
>
Here is updated patch for pg_dump
Regards
Pavel
>
> regards, tom lane
>
Attachments:
[text/x-patch] pg_dump_filter-20200904.patch (7.6K, ../../CAFj8pRBDbbmBf=eQTvpkf0=JJsG2t6qxMGPqpJRrq-M2WJQESA@mail.gmail.com/3-pg_dump_filter-20200904.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0b2e2de87b..48dd75bfb1 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -755,6 +755,56 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file.
+ If you use "-" as a filename, the filters are read from stdin.
+ The lines of this file must have the following format:
+<synopsis>
+(+|-)[tnfd] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first character specifies whether the object is to be included
+ (<literal>+</literal>) or excluded (<literal>-</literal>), and the
+ second character specifies the type of object to be filtered:
+ <literal>t</literal> (table),
+ <literal>n</literal> (schema),
+ <literal>f</literal> (foreign table),
+ <literal>d</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign table
+ <literal>some_foreign_table</literal>, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
++t mytable1
++f some_foreign_table
+-d mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The lines starting with symbol <literal>#</literal> are ignored.
+ Previous white chars (spaces, tabs) are not allowed. These
+ lines can be used for comments, notes.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index d3ca54e4dc..1d3c97ad97 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -53,6 +53,7 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
@@ -291,6 +292,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
@@ -365,6 +367,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -604,6 +607,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1023,6 +1030,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18473,3 +18482,157 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ char *line;
+ int lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ while ((line = pg_get_line(fp)))
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+
+ lineno += 1;
+
+ (void) pg_strip_crlf(line);
+
+ /* ignore empty rows */
+ if (*line == '\0')
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*line == '#')
+ continue;
+
+ if (line[1] == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "line too short",
+ line,
+ lineno);
+
+ if (line[0] == '+')
+ is_include = true;
+ else if (line[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid option type (use [+-]",
+ line,
+ lineno);
+
+ objecttype = line[1];
+ objectname = &line[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ line,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ line,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ line,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid object type (use [tndf])",
+ line,
+ lineno);
+
+ pfree(line);
+ }
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-09-07 05:08 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-09-07 05:08 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
pá 4. 9. 2020 v 5:21 odesílatel Pavel Stehule <[email protected]>
napsal:
>
>
> pá 4. 9. 2020 v 2:15 odesílatel Tom Lane <[email protected]> napsal:
>
>> Alvaro Herrera <[email protected]> writes:
>> > So, Tom added a coding pattern for doing this in commit 8f8154a503c7,
>> > which is ostensibly also to be used in pg_regress [1] -- maybe it'd be
>> > useful to have this in src/common?
>>
>> Done, see pg_get_line() added by 67a472d71.
>>
>
> Here is updated patch for pg_dump
>
another update based on pg_get_line_append function
Regards
Pavel
> Regards
>
> Pavel
>
>
>
>>
>> regards, tom lane
>>
>
Attachments:
[text/x-patch] pg_dump-filter-20200907.patch (7.8K, ../../CAFj8pRDAeVP0tf9QVRyj1395pxvemYx8UZO4RXHYNQR5xB4syg@mail.gmail.com/3-pg_dump-filter-20200907.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0b2e2de87b..48dd75bfb1 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -755,6 +755,56 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file.
+ If you use "-" as a filename, the filters are read from stdin.
+ The lines of this file must have the following format:
+<synopsis>
+(+|-)[tnfd] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first character specifies whether the object is to be included
+ (<literal>+</literal>) or excluded (<literal>-</literal>), and the
+ second character specifies the type of object to be filtered:
+ <literal>t</literal> (table),
+ <literal>n</literal> (schema),
+ <literal>f</literal> (foreign table),
+ <literal>d</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign table
+ <literal>some_foreign_table</literal>, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
++t mytable1
++f some_foreign_table
+-d mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The lines starting with symbol <literal>#</literal> are ignored.
+ Previous white chars (spaces, tabs) are not allowed. These
+ lines can be used for comments, notes.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 784bceaec3..6349be55e9 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -53,9 +53,11 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -291,6 +293,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
@@ -365,6 +368,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -604,6 +608,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1023,6 +1031,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18477,3 +18487,162 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+
+ while (pg_get_line_append(fp, &line))
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+ char *str = line.data;
+
+ lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* ignore empty rows */
+ if (*str == '\0')
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*str == '#')
+ continue;
+
+ if (str[1] == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "line too short",
+ str,
+ lineno);
+
+ if (str[0] == '+')
+ is_include = true;
+ else if (str[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid option type (use [+-]",
+ str,
+ lineno);
+
+ objecttype = str[1];
+ objectname = &str[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ str,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ str,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ str,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid object type (use [tndf])",
+ str,
+ lineno);
+
+ resetStringInfo(&line);
+ }
+
+ pfree(line.data);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-09-07 12:14 Surafel Temesgen <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Surafel Temesgen @ 2020-09-07 12:14 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Hi Pavel
On Fri, Sep 4, 2020 at 6:22 AM Pavel Stehule <[email protected]>
wrote:
>
> Here is updated patch for pg_dump
>
>
pg_dumpall also has –exclude-database=pattern and –no-comments option
doesn't that qualify it to benefits from this feature? And please add a
test case for this option
regards
Surafel
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-09-11 08:50 Pavel Stehule <[email protected]>
parent: Surafel Temesgen <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-09-11 08:50 UTC (permalink / raw)
To: Surafel Temesgen <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Hi
po 7. 9. 2020 v 14:14 odesílatel Surafel Temesgen <[email protected]>
napsal:
> Hi Pavel
>
> On Fri, Sep 4, 2020 at 6:22 AM Pavel Stehule <[email protected]>
> wrote:
>
>>
>> Here is updated patch for pg_dump
>>
>>
> pg_dumpall also has –exclude-database=pattern and –no-comments option
> doesn't that qualify it to benefits from this feature? And please add a
> test case for this option
>
This patch is related to pg_dump (in this moment), so pg_dumpall options
are out of scope.
I am not sure if pg_dumpall needs this functionality - maybe, but I can use
bash or some similar for implementation of this feature. There is no
requirement to do it all necessary work under one transaction, one snapshot.
For pg_dump can be used different format, because it uses different
granularity. Some like "{+/-} dbname"
"--no-comments" is a global parameter without arguments. I don't understand
how this parameter can be related to this feature?
I am working on regress tests.
Regards
Pavel
> regards
>
> Surafel
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-09-12 08:12 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-09-12 08:12 UTC (permalink / raw)
To: Surafel Temesgen <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Hi
pá 11. 9. 2020 v 10:50 odesílatel Pavel Stehule <[email protected]>
napsal:
> Hi
>
> po 7. 9. 2020 v 14:14 odesílatel Surafel Temesgen <[email protected]>
> napsal:
>
>> Hi Pavel
>>
>> On Fri, Sep 4, 2020 at 6:22 AM Pavel Stehule <[email protected]>
>> wrote:
>>
>>>
>>> Here is updated patch for pg_dump
>>>
>>>
>> pg_dumpall also has –exclude-database=pattern and –no-comments option
>> doesn't that qualify it to benefits from this feature? And please add a
>> test case for this option
>>
>
> This patch is related to pg_dump (in this moment), so pg_dumpall options
> are out of scope.
>
> I am not sure if pg_dumpall needs this functionality - maybe, but I can
> use bash or some similar for implementation of this feature. There is no
> requirement to do it all necessary work under one transaction, one snapshot.
>
> For pg_dump can be used different format, because it uses different
> granularity. Some like "{+/-} dbname"
>
> "--no-comments" is a global parameter without arguments. I don't
> understand how this parameter can be related to this feature?
>
> I am working on regress tests.
>
There is a updated version with regress tests
Regards
Pavel
> Regards
>
> Pavel
>
>
>> regards
>>
>> Surafel
>>
>>
>
Attachments:
[text/x-patch] pg_dump-filter-20200912.patch (12.3K, ../../CAFj8pRCJ-F6RGmLsoO37oz75kZcwKr39c5Y4YWbd3e=nkwDOPA@mail.gmail.com/3-pg_dump-filter-20200912.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0b2e2de87b..c4f818f305 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -755,6 +755,56 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file.
+ If you use "-" as a filename, the filters are read from stdin.
+ The lines of this file must have the following format:
+<synopsis>
+(+|-)[tnfd] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first character specifies whether the object is to be included
+ (<literal>+</literal>) or excluded (<literal>-</literal>), and the
+ second character specifies the type of object to be filtered:
+ <literal>t</literal> (table),
+ <literal>n</literal> (schema),
+ <literal>f</literal> (foreign server),
+ <literal>d</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
++t mytable1
++f some_foreign_server
+-d mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The lines starting with symbol <literal>#</literal> are ignored.
+ Previous white chars (spaces, tabs) are not allowed. These
+ lines can be used for comments, notes. Empty lines are ignored too.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 784bceaec3..94a8b2e187 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -53,9 +53,11 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -291,6 +293,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
@@ -365,6 +368,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -604,6 +608,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1023,6 +1031,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18477,3 +18487,164 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+
+ while (pg_get_line_append(fp, &line))
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+ char *str = line.data;
+
+ lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* ignore empty rows */
+ if (*str == '\0')
+ goto resetline;
+
+ /* when first char is hash, ignore whole line */
+ if (*str == '#')
+ goto resetline;
+
+ if (str[1] == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "line too short",
+ str,
+ lineno);
+
+ if (str[0] == '+')
+ is_include = true;
+ else if (str[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid option type (use [+-]",
+ str,
+ lineno);
+
+ objecttype = str[1];
+ objectname = &str[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ str,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ str,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ str,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid object type (use [tndf])",
+ str,
+ lineno);
+
+resetline:
+
+ resetStringInfo(&line);
+ }
+
+ pfree(line.data);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filter.pl b/src/bin/pg_dump/t/004_pg_dump_filter.pl
new file mode 100644
index 0000000000..957442c5e4
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filter.pl
@@ -0,0 +1,129 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 22;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = get_new_node('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "+t table_one\n";
+print $inputfile "+t table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "-d table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-t table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-n public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "+f doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid format of filter file/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "+";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid format of filter file/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "+d sometable";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/include filter is not supported for this type of object/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-f someforeignserver";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/exclude filter is not supported for this type of object/,
+ "broken format check");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-09-24 17:47 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-09-24 17:47 UTC (permalink / raw)
To: Surafel Temesgen <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Hi
rebase + minor change - using pg_get_line_buf instead pg_get_line_append
Regards
Pavel
Attachments:
[text/x-patch] pg_dump-filter-20200924.patch (12.2K, ../../CAFj8pRDCOgPLo1WNC8iniP0g6nGQdpVj-k5T3BsYrmOBtDSShw@mail.gmail.com/3-pg_dump-filter-20200924.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index e09ed0a4c3..9d7ebaf922 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -757,6 +757,56 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file.
+ If you use "-" as a filename, the filters are read from stdin.
+ The lines of this file must have the following format:
+<synopsis>
+(+|-)[tnfd] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first character specifies whether the object is to be included
+ (<literal>+</literal>) or excluded (<literal>-</literal>), and the
+ second character specifies the type of object to be filtered:
+ <literal>t</literal> (table),
+ <literal>n</literal> (schema),
+ <literal>f</literal> (foreign server),
+ <literal>d</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
++t mytable1
++f some_foreign_server
+-d mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The lines starting with symbol <literal>#</literal> are ignored.
+ Previous white chars (spaces, tabs) are not allowed. These
+ lines can be used for comments, notes. Empty lines are ignored too.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index f021bb72f4..a7824c5bf6 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -53,9 +53,11 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -290,6 +292,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
@@ -364,6 +367,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -603,6 +607,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1022,6 +1030,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18470,3 +18480,160 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+
+ while (pg_get_line_buf(fp, &line))
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+ char *str = line.data;
+
+ lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* ignore empty rows */
+ if (*str == '\0')
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*str == '#')
+ continue;
+
+ if (str[1] == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "line too short",
+ str,
+ lineno);
+
+ if (str[0] == '+')
+ is_include = true;
+ else if (str[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid option type (use [+-]",
+ str,
+ lineno);
+
+ objecttype = str[1];
+ objectname = &str[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ str,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ str,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ str,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid object type (use [tndf])",
+ str,
+ lineno);
+ }
+
+ pfree(line.data);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filter.pl b/src/bin/pg_dump/t/004_pg_dump_filter.pl
new file mode 100644
index 0000000000..957442c5e4
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filter.pl
@@ -0,0 +1,129 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 22;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = get_new_node('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "+t table_one\n";
+print $inputfile "+t table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "-d table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-t table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-n public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "+f doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid format of filter file/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "+";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid format of filter file/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "+d sometable";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/include filter is not supported for this type of object/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-f someforeignserver";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/exclude filter is not supported for this type of object/,
+ "broken format check");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-10 20:09 Stephen Frost <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Stephen Frost @ 2020-11-10 20:09 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Greetings,
* Pavel Stehule ([email protected]) wrote:
> rebase + minor change - using pg_get_line_buf instead pg_get_line_append
I started looking at this and went back through the thread and while I
tend to agree that JSON may not be a good choice for this, it's not the
only possible alternative. There is no doubt that pg_dump is already a
sophisticated data export tool, and likely to continue to gain new
features, such that having a configuration file for it would be very
handy, but this clearly isn't really going in a direction that would
allow for that.
Perhaps this feature could co-exist with a full blown configuration for
pg_dump, but even then there's certainly issues with what's proposed-
how would you handle explicitly asking for a table which is named
" mytable" to be included or excluded? Or a table which has a newline
in it? Using a standardized format which supports the full range of
what we do in a table name, explicitly and clearly, would address these
issues and also give us the flexibility to extend the options which
could be used through the configuration file beyond just the filters in
the future.
Unlike for the pg_basebackup manifest, which we generate and read
entirely programatically, a config file for pg_dump would almost
certainly be updated manually (or, at least, parts of it would be and
perhaps other parts generated), which means it'd really be ideal to have
a proper way to support comments in it (something that the proposed
format also doesn't really get right- # must be the *first* character,
and you can only have whole-line comments..?), avoid extra unneeded
punctuation (or, at times, allow it- such as trailing commas in lists),
cleanly handle multi-line strings (consider the oft discussed idea
around having pg_dump support a WHERE clause for exporting data from
tables...), etc.
Overall, -1 from me on this approach. Maybe it could be fixed up to
handle all the different names of objects that we support today
(something which, imv, is really a clear requirement for this feature to
be committed), but I suspect you'd end up half-way to yet another
configuration format when we could be working to support something like
TOML or maybe YAML... but if you want my 2c, TOML seems closer to what
we do for postgresql.conf and getting that over to something that's
standardized, while a crazy long shot, is a general nice idea, imv.
Thanks,
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-11 05:32 Pavel Stehule <[email protected]>
parent: Stephen Frost <[email protected]>
0 siblings, 3 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-11-11 05:32 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Hi
út 10. 11. 2020 v 21:09 odesílatel Stephen Frost <[email protected]>
napsal:
> Greetings,
>
> * Pavel Stehule ([email protected]) wrote:
> > rebase + minor change - using pg_get_line_buf instead pg_get_line_append
>
> I started looking at this and went back through the thread and while I
> tend to agree that JSON may not be a good choice for this, it's not the
> only possible alternative. There is no doubt that pg_dump is already a
> sophisticated data export tool, and likely to continue to gain new
> features, such that having a configuration file for it would be very
> handy, but this clearly isn't really going in a direction that would
> allow for that.
>
> Perhaps this feature could co-exist with a full blown configuration for
> pg_dump, but even then there's certainly issues with what's proposed-
> how would you handle explicitly asking for a table which is named
> " mytable" to be included or excluded? Or a table which has a newline
> in it? Using a standardized format which supports the full range of
> what we do in a table name, explicitly and clearly, would address these
> issues and also give us the flexibility to extend the options which
> could be used through the configuration file beyond just the filters in
> the future.
>
This is the correct argument - I will check a possibility to use strange
names, but there is the same possibility and functionality like we allow
from the command line. So you can use double quoted names. I'll check it.
> Unlike for the pg_basebackup manifest, which we generate and read
> entirely programatically, a config file for pg_dump would almost
> certainly be updated manually (or, at least, parts of it would be and
> perhaps other parts generated), which means it'd really be ideal to have
> a proper way to support comments in it (something that the proposed
> format also doesn't really get right- # must be the *first* character,
> and you can only have whole-line comments..?), avoid extra unneeded
> punctuation (or, at times, allow it- such as trailing commas in lists),
> cleanly handle multi-line strings (consider the oft discussed idea
> around having pg_dump support a WHERE clause for exporting data from
> tables...), etc.
>
I think the proposed feature is very far to be the config file for pg_dump
(it implements a option "--filter"). This is not the target. It is not
designed for this. This is just an alternative for options like -t, -T, ...
and I am sure so nobody will generate this file manually. Main target of
this patch is eliminating problems with the max length of the command line.
So it is really not designed to be the config file for pg_dump.
>
> Overall, -1 from me on this approach. Maybe it could be fixed up to
> handle all the different names of objects that we support today
> (something which, imv, is really a clear requirement for this feature to
> be committed), but I suspect you'd end up half-way to yet another
> configuration format when we could be working to support something like
> TOML or maybe YAML... but if you want my 2c, TOML seems closer to what
> we do for postgresql.conf and getting that over to something that's
> standardized, while a crazy long shot, is a general nice idea, imv.
>
I have nothing against TOML, but I don't see a sense of usage in this
patch. This patch doesn't implement a config file for pg_dump, and I don't
see any sense or benefits of it. The TOML is designed for different
purposes. TOML is good for manual creating, but it is not this case.
Typical usage of this patch is some like, and TOML syntax (or JSON) is not
good for this.
psql -c "select '+t' || quote_ident(relname) from pg_class where relname
..." | pg_dump --filter=/dev/stdin
I can imagine some benefits of saved configure files for postgres
applications - but it should be designed generally and implemented
generally. Probably you would use one for pg_dump, psql, pg_restore, ....
But it is a different feature with different usage. This patch doesn't
implement option "--config", it implements option "--filter".
Regards
Pavel
> Thanks,
>
> Stephen
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-11 05:49 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
2 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-11-11 05:49 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Hi
Perhaps this feature could co-exist with a full blown configuration for
>> pg_dump, but even then there's certainly issues with what's proposed-
>> how would you handle explicitly asking for a table which is named
>> " mytable" to be included or excluded? Or a table which has a newline
>> in it? Using a standardized format which supports the full range of
>> what we do in a table name, explicitly and clearly, would address these
>> issues and also give us the flexibility to extend the options which
>> could be used through the configuration file beyond just the filters in
>> the future.
>>
>
> This is the correct argument - I will check a possibility to use strange
> names, but there is the same possibility and functionality like we allow
> from the command line. So you can use double quoted names. I'll check it.
>
I checked
echo "+t \"bad Name\"" | /usr/local/pgsql/master/bin/pg_dump
--filter=/dev/stdin
It is working without any problem
Regards
Pavel
^ permalink raw reply [nested|flat] 181+ messages in thread
* [PATCH 1/2] proposal: possibility to read dumped table's name from file
@ 2020-11-11 05:54 Pavel Stehule <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-11-11 05:54 UTC (permalink / raw)
--00000000000087718205b3ce6fcd
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Hi
=C4=8Dt 24. 9. 2020 v 19:47 odes=C3=ADlatel Pavel Stehule <pavel.stehule@gm=
ail.com>
napsal:
> Hi
>
> rebase + minor change - using pg_get_line_buf instead pg_get_line_append
>
>
fresh rebase
Regards
>
> Pavel
>
<div dir="ltr"><div>Hi<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">čt 24. 9. 2020 v 19:47 odesílatel Pavel Stehule <<a href="mailto:[email protected]">[email protected]</a>> napsal:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi</div><div><br></div><div>rebase + minor change - using pg_get_line_buf instead pg_get_line_append</div><div><br></div></div></blockquote><div><br></div><div>fresh rebase</div><div><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div>Regards <br></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>Pavel<br></div></div>
</blockquote></div></div>
---
doc/src/sgml/ref/pg_dump.sgml | 50 +++++++
src/bin/pg_dump/pg_dump.c | 168 ++++++++++++++++++++++++
src/bin/pg_dump/t/004_pg_dump_filter.pl | 129 ++++++++++++++++++
3 files changed, 347 insertions(+)
create mode 100644 src/bin/pg_dump/t/004_pg_dump_filter.pl
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0aa35cf0c3..068821583f 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -751,6 +751,56 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file.
+ If you use "-" as a filename, the filters are read from stdin.
+ The lines of this file must have the following format:
+<synopsis>
+(+|-)[tnfd] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first character specifies whether the object is to be included
+ (<literal>+</literal>) or excluded (<literal>-</literal>), and the
+ second character specifies the type of object to be filtered:
+ <literal>t</literal> (table),
+ <literal>n</literal> (schema),
+ <literal>f</literal> (foreign server),
+ <literal>d</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
++t mytable1
++f some_foreign_server
+-d mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The lines starting with symbol <literal>#</literal> are ignored.
+ Previous white chars (spaces, tabs) are not allowed. These
+ lines can be used for comments, notes. Empty lines are ignored too.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index c68db75b97..f1f9e8321e 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -54,9 +54,11 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -294,6 +296,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
@@ -367,6 +370,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -390,6 +394,7 @@ main(int argc, char **argv)
{"rows-per-insert", required_argument, NULL, 10},
{"include-foreign-data", required_argument, NULL, 11},
{"index-collation-versions-unknown", no_argument, &dopt.coll_unknown, 1},
+ {"include-foreign-data-file", required_argument, NULL, 17},
{NULL, 0, NULL, 0}
};
@@ -607,6 +612,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1035,6 +1044,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18632,3 +18643,160 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+
+ while (pg_get_line_buf(fp, &line))
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+ char *str = line.data;
+
+ lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* ignore empty rows */
+ if (*str == '\0')
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*str == '#')
+ continue;
+
+ if (str[1] == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "line too short",
+ str,
+ lineno);
+
+ if (str[0] == '+')
+ is_include = true;
+ else if (str[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid option type (use [+-]",
+ str,
+ lineno);
+
+ objecttype = str[1];
+ objectname = &str[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ str,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ str,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ str,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid object type (use [tndf])",
+ str,
+ lineno);
+ }
+
+ pfree(line.data);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filter.pl b/src/bin/pg_dump/t/004_pg_dump_filter.pl
new file mode 100644
index 0000000000..957442c5e4
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filter.pl
@@ -0,0 +1,129 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 22;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = get_new_node('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "+t table_one\n";
+print $inputfile "+t table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "-d table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-t table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-n public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "+f doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid format of filter file/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "+";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid format of filter file/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "+d sometable";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/include filter is not supported for this type of object/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-f someforeignserver";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/exclude filter is not supported for this type of object/,
+ "broken format check");
--
2.17.0
--SFyWQ0h3ruR435lw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0002-f-Read-additional-cmdline-args-from-config.patch.not"
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-11 05:54 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-11-11 05:54 UTC (permalink / raw)
To: Surafel Temesgen <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers
Hi
čt 24. 9. 2020 v 19:47 odesílatel Pavel Stehule <[email protected]>
napsal:
> Hi
>
> rebase + minor change - using pg_get_line_buf instead pg_get_line_append
>
>
fresh rebase
Regards
>
> Pavel
>
Attachments:
[text/x-patch] pg_dump-filter-20201011.patch (12.5K, ../../CAFj8pRCJ4C9rnKha=XDA3k8RAboSgJ68tNh-NfaRurwNwVbFHQ@mail.gmail.com/3-pg_dump-filter-20201011.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0aa35cf0c3..068821583f 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -751,6 +751,56 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file.
+ If you use "-" as a filename, the filters are read from stdin.
+ The lines of this file must have the following format:
+<synopsis>
+(+|-)[tnfd] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first character specifies whether the object is to be included
+ (<literal>+</literal>) or excluded (<literal>-</literal>), and the
+ second character specifies the type of object to be filtered:
+ <literal>t</literal> (table),
+ <literal>n</literal> (schema),
+ <literal>f</literal> (foreign server),
+ <literal>d</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
++t mytable1
++f some_foreign_server
+-d mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The lines starting with symbol <literal>#</literal> are ignored.
+ Previous white chars (spaces, tabs) are not allowed. These
+ lines can be used for comments, notes. Empty lines are ignored too.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index c68db75b97..f1f9e8321e 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -54,9 +54,11 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -294,6 +296,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
@@ -367,6 +370,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -390,6 +394,7 @@ main(int argc, char **argv)
{"rows-per-insert", required_argument, NULL, 10},
{"include-foreign-data", required_argument, NULL, 11},
{"index-collation-versions-unknown", no_argument, &dopt.coll_unknown, 1},
+ {"include-foreign-data-file", required_argument, NULL, 17},
{NULL, 0, NULL, 0}
};
@@ -607,6 +612,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1035,6 +1044,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18632,3 +18643,160 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+
+ while (pg_get_line_buf(fp, &line))
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+ char *str = line.data;
+
+ lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* ignore empty rows */
+ if (*str == '\0')
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*str == '#')
+ continue;
+
+ if (str[1] == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "line too short",
+ str,
+ lineno);
+
+ if (str[0] == '+')
+ is_include = true;
+ else if (str[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid option type (use [+-]",
+ str,
+ lineno);
+
+ objecttype = str[1];
+ objectname = &str[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ str,
+ lineno);
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ str,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ str,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid object type (use [tndf])",
+ str,
+ lineno);
+ }
+
+ pfree(line.data);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filter.pl b/src/bin/pg_dump/t/004_pg_dump_filter.pl
new file mode 100644
index 0000000000..957442c5e4
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filter.pl
@@ -0,0 +1,129 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 22;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = get_new_node('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "+t table_one\n";
+print $inputfile "+t table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "-d table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-t table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-n public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "+f doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid format of filter file/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "+";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid format of filter file/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "+d sometable";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/include filter is not supported for this type of object/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-f someforeignserver";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/exclude filter is not supported for this type of object/,
+ "broken format check");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-11 07:09 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
2 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-11-11 07:09 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
st 11. 11. 2020 v 6:32 odesílatel Pavel Stehule <[email protected]>
napsal:
> Hi
>
> út 10. 11. 2020 v 21:09 odesílatel Stephen Frost <[email protected]>
> napsal:
>
>> Greetings,
>>
>> * Pavel Stehule ([email protected]) wrote:
>> > rebase + minor change - using pg_get_line_buf instead pg_get_line_append
>>
>> I started looking at this and went back through the thread and while I
>> tend to agree that JSON may not be a good choice for this, it's not the
>> only possible alternative. There is no doubt that pg_dump is already a
>> sophisticated data export tool, and likely to continue to gain new
>> features, such that having a configuration file for it would be very
>> handy, but this clearly isn't really going in a direction that would
>> allow for that.
>>
>> Perhaps this feature could co-exist with a full blown configuration for
>> pg_dump, but even then there's certainly issues with what's proposed-
>> how would you handle explicitly asking for a table which is named
>> " mytable" to be included or excluded? Or a table which has a newline
>> in it? Using a standardized format which supports the full range of
>> what we do in a table name, explicitly and clearly, would address these
>> issues and also give us the flexibility to extend the options which
>> could be used through the configuration file beyond just the filters in
>> the future.
>>
>
> This is the correct argument - I will check a possibility to use strange
> names, but there is the same possibility and functionality like we allow
> from the command line. So you can use double quoted names. I'll check it.
>
>
>> Unlike for the pg_basebackup manifest, which we generate and read
>> entirely programatically, a config file for pg_dump would almost
>> certainly be updated manually (or, at least, parts of it would be and
>> perhaps other parts generated), which means it'd really be ideal to have
>> a proper way to support comments in it (something that the proposed
>> format also doesn't really get right- # must be the *first* character,
>> and you can only have whole-line comments..?), avoid extra unneeded
>> punctuation (or, at times, allow it- such as trailing commas in lists),
>> cleanly handle multi-line strings (consider the oft discussed idea
>> around having pg_dump support a WHERE clause for exporting data from
>> tables...), etc.
>>
>
> I think the proposed feature is very far to be the config file for pg_dump
> (it implements a option "--filter"). This is not the target. It is not
> designed for this. This is just an alternative for options like -t, -T, ...
> and I am sure so nobody will generate this file manually. Main target of
> this patch is eliminating problems with the max length of the command line.
> So it is really not designed to be the config file for pg_dump.
>
>
>>
>> Overall, -1 from me on this approach. Maybe it could be fixed up to
>> handle all the different names of objects that we support today
>> (something which, imv, is really a clear requirement for this feature to
>> be committed), but I suspect you'd end up half-way to yet another
>> configuration format when we could be working to support something like
>> TOML or maybe YAML... but if you want my 2c, TOML seems closer to what
>> we do for postgresql.conf and getting that over to something that's
>> standardized, while a crazy long shot, is a general nice idea, imv.
>>
>
> I have nothing against TOML, but I don't see a sense of usage in this
> patch. This patch doesn't implement a config file for pg_dump, and I don't
> see any sense or benefits of it. The TOML is designed for different
> purposes. TOML is good for manual creating, but it is not this case.
> Typical usage of this patch is some like, and TOML syntax (or JSON) is not
> good for this.
>
> psql -c "select '+t' || quote_ident(relname) from pg_class where relname
> ..." | pg_dump --filter=/dev/stdin
>
> I can imagine some benefits of saved configure files for postgres
> applications - but it should be designed generally and implemented
> generally. Probably you would use one for pg_dump, psql, pg_restore, ....
> But it is a different feature with different usage. This patch doesn't
> implement option "--config", it implements option "--filter".
>
Some generic configuration for postgres binary applications is an
interesting idea. And TOML language can be well for this purpose. We can
parametrize applications by command line and by system variables. But
filtering objects is a really different case - although there is some small
intersection, and it will be used very differently, and I don't think so
one language can be practical for both cases. The object filtering is an
independent feature, and both features can coexist together.
Regards
Pavel
> Regards
>
> Pavel
>
>
>
>> Thanks,
>>
>> Stephen
>>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-11 15:17 Alvaro Herrera <[email protected]>
parent: Pavel Stehule <[email protected]>
2 siblings, 1 reply; 181+ messages in thread
From: Alvaro Herrera @ 2020-11-11 15:17 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
On 2020-Nov-11, Pavel Stehule wrote:
> I think the proposed feature is very far to be the config file for pg_dump
> (it implements a option "--filter"). This is not the target. It is not
> designed for this. This is just an alternative for options like -t, -T, ...
> and I am sure so nobody will generate this file manually. Main target of
> this patch is eliminating problems with the max length of the command line.
> So it is really not designed to be the config file for pg_dump.
I agree that a replacement for existing command line arguments is a good
goal, but at the same time it's good to keep in mind the request that
more object types are supported as dumpable. While it's not necessary
that this infrastructure supports all object types in the first cut,
it'd be good to have it support that. I would propose that instead of a
single letter 't' etc we support keywords, maybe similar to those
returned by getObjectTypeDescription() (with additions -- for example
for "table data"). Then we can extend for other object types later
without struggling to find good letters for each.
Of course we could allow abbrevations for common cases, such that "t"
means "table".
For example: it'll be useful to support selective dumping of functions,
materialized views, foreign objects, etc.
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-12 07:45 Pavel Stehule <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-11-12 07:45 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; vignesh C <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
st 11. 11. 2020 v 16:17 odesílatel Alvaro Herrera <[email protected]>
napsal:
> On 2020-Nov-11, Pavel Stehule wrote:
>
> > I think the proposed feature is very far to be the config file for
> pg_dump
> > (it implements a option "--filter"). This is not the target. It is not
> > designed for this. This is just an alternative for options like -t, -T,
> ...
> > and I am sure so nobody will generate this file manually. Main target of
> > this patch is eliminating problems with the max length of the command
> line.
> > So it is really not designed to be the config file for pg_dump.
>
> I agree that a replacement for existing command line arguments is a good
> goal, but at the same time it's good to keep in mind the request that
> more object types are supported as dumpable. While it's not necessary
> that this infrastructure supports all object types in the first cut,
> it'd be good to have it support that. I would propose that instead of a
> single letter 't' etc we support keywords, maybe similar to those
> returned by getObjectTypeDescription() (with additions -- for example
> for "table data"). Then we can extend for other object types later
> without struggling to find good letters for each.
>
> Of course we could allow abbrevations for common cases, such that "t"
> means "table".
>
> For example: it'll be useful to support selective dumping of functions,
> materialized views, foreign objects, etc.
>
Implementation of this is trivial.
The hard work is mapping pg_dump options on database objects. t -> table is
simple, but n -> schema looks a little bit inconsistent - although it is
consistent with pg_dump. d or D - there is no system object like data. I am
afraid so there are two independent systems - pg_dump options, and database
objects, and it can be hard or not very practical to join these systems.
Unfortunately there is not good consistency in the short options of pg_dump
today. More - a lot of object names are multi words with inner space. This
is not too practical.
What about supporting two syntaxes?
1. first short current +-tndf filter - but the implementation should not be
limited to one char - there can be any string until space
2. long syntax - all these pg_dump options has long options, and then we
can extend this feature without any problem in future
table|exclude-table|exclude-table-data|schema|exclude-schema|include-foreign-data=PATTERN
so the content of filter file can looks like:
+t mytable
+t tabprefix*
-t bigtable
table=mytable2
exclude-table=mytable2
This format allows quick work for most common database objects, and it is
extensible and consistent with pg_dump's long options.
What do you think about it?
Personally, I am thinking that it is over-engineering a little bit, maybe
we can implement this feature just test first string after +- symbols
(instead first char like now) - and any enhanced syntax can be implemented
in future when there will be this requirement. Second syntax can be
implemented very simply, because it can be identified by first char
processing. We can implement second syntax only too. It will work too, but
I think so short syntax is more practical for daily work (for common
options). I expect so 99% percent of this objects will be "+t tablename".
Regards
Pavel
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-17 21:53 Justin Pryzby <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Justin Pryzby @ 2020-11-17 21:53 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
On Wed, Nov 11, 2020 at 06:49:43AM +0100, Pavel Stehule wrote:
> Perhaps this feature could co-exist with a full blown configuration for
> >> pg_dump, but even then there's certainly issues with what's proposed-
> >> how would you handle explicitly asking for a table which is named
> >> " mytable" to be included or excluded? Or a table which has a newline
> >> in it? Using a standardized format which supports the full range of
> >> what we do in a table name, explicitly and clearly, would address these
> >> issues and also give us the flexibility to extend the options which
> >> could be used through the configuration file beyond just the filters in
> >> the future.
I think it's a reasonable question - why would a new configuration file option
include support for only a handful of existing arguments but not the rest.
> > This is the correct argument - I will check a possibility to use strange
> > names, but there is the same possibility and functionality like we allow
> > from the command line. So you can use double quoted names. I'll check it.
>
> I checked
> echo "+t \"bad Name\"" | /usr/local/pgsql/master/bin/pg_dump --filter=/dev/stdin
> It is working without any problem
I think it couldn't possibly work with newlines, since you call pg_get_line().
I realize that entering a newline into the shell would also be a PITA, but that
could be one *more* reason to support a config file - to allow terrible table
names to be in a file and avoid writing dash tee quote something enter else
quote in a pg_dump command, or shell script.
I fooled with argument parsing to handle reading from a file in the quickest
way. As written, this fails to handle multiple config files, and special table
names, which need to support arbitrary, logical lines, with quotes surrounding
newlines or other special chars. As written, the --config file is parsed
*after* all other arguments, so it could override previous args (like
--no-blobs --no-blogs, --file, --format, --compress, --lock-wait), which I
guess is bad, so the config file should be processed *during* argument parsing.
Unfortunately, I think that suggests duplicating parsing of all/most the
argument parsing for config file support - I'd be happy if someone suggested a
better way.
BTW, in your most recent patch:
s/empty rows/empty lines/
unbalanced parens: "invalid option type (use [+-]"
@cfbot: I renamed the patch so please ignore it.
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-18 14:46 Stephen Frost <[email protected]>
parent: Justin Pryzby <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Stephen Frost @ 2020-11-18 14:46 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Pavel Stehule <[email protected]>; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Greetings,
* Justin Pryzby ([email protected]) wrote:
> On Wed, Nov 11, 2020 at 06:49:43AM +0100, Pavel Stehule wrote:
> > >> Perhaps this feature could co-exist with a full blown configuration for
> > >> pg_dump, but even then there's certainly issues with what's proposed-
> > >> how would you handle explicitly asking for a table which is named
> > >> " mytable" to be included or excluded? Or a table which has a newline
> > >> in it? Using a standardized format which supports the full range of
> > >> what we do in a table name, explicitly and clearly, would address these
> > >> issues and also give us the flexibility to extend the options which
> > >> could be used through the configuration file beyond just the filters in
> > >> the future.
>
> I think it's a reasonable question - why would a new configuration file option
> include support for only a handful of existing arguments but not the rest.
Even if the first version of having a config file for pg_dump only
supported some options, that would be reasonable imv, but I dislike the
idea of building it in such a way that it'll be awkward to add more
options to it in the future, something that I definitely think people
would like to see (I know I would...).
> > > This is the correct argument - I will check a possibility to use strange
> > > names, but there is the same possibility and functionality like we allow
> > > from the command line. So you can use double quoted names. I'll check it.
> >
> > I checked
> > echo "+t \"bad Name\"" | /usr/local/pgsql/master/bin/pg_dump --filter=/dev/stdin
> > It is working without any problem
>
> I think it couldn't possibly work with newlines, since you call pg_get_line().
Yeah, I didn't really believe that it actually worked but hadn't had a
chance to demonstrate that it didn't yet.
> I realize that entering a newline into the shell would also be a PITA, but that
> could be one *more* reason to support a config file - to allow terrible table
> names to be in a file and avoid writing dash tee quote something enter else
> quote in a pg_dump command, or shell script.
Agreed.
> I fooled with argument parsing to handle reading from a file in the quickest
> way. As written, this fails to handle multiple config files, and special table
> names, which need to support arbitrary, logical lines, with quotes surrounding
> newlines or other special chars. As written, the --config file is parsed
> *after* all other arguments, so it could override previous args (like
> --no-blobs --no-blogs, --file, --format, --compress, --lock-wait), which I
> guess is bad, so the config file should be processed *during* argument parsing.
> Unfortunately, I think that suggests duplicating parsing of all/most the
> argument parsing for config file support - I'd be happy if someone suggested a
> better way.
This still feels like we're trying to quickly hack-and-slash at adding a
config file option rather than thinking through what a sensible design
for a pg_dump config file would look like. Having a way to avoid having
multiple places in the code that has to handle all the possible options
is a nice idea but, as I tried to allude to up-thread, I fully expect
that once we've got this config file capability that we're going to want
to add things to it that would be difficult to utilize through the
command-line and so I expect these code paths to diverge anyway.
I would imagine something like:
[connection]
db-host=whatever
db-port=5433
...
[output]
owners = true
privileges = false
format = "custom"
file = "myoutput.dump"
# This is a comment
[include]
tables = [ "sometable", "table with spaces",
"table with quoted \"",
"""this is my table
with a carriage return""", "anothertable" ]
table-patterns = [ "table*" ]
schemas = [ "myschema" ]
[exclude]
tables = [ "similar to include" ]
functions = [ "somefunction(int)" ]
etc, etc ...
Thanks,
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-19 19:51 Pavel Stehule <[email protected]>
parent: Justin Pryzby <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-11-19 19:51 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
út 17. 11. 2020 v 22:53 odesílatel Justin Pryzby <[email protected]>
napsal:
> On Wed, Nov 11, 2020 at 06:49:43AM +0100, Pavel Stehule wrote:
> > Perhaps this feature could co-exist with a full blown configuration for
> > >> pg_dump, but even then there's certainly issues with what's proposed-
> > >> how would you handle explicitly asking for a table which is named
> > >> " mytable" to be included or excluded? Or a table which has a
> newline
> > >> in it? Using a standardized format which supports the full range of
> > >> what we do in a table name, explicitly and clearly, would address
> these
> > >> issues and also give us the flexibility to extend the options which
> > >> could be used through the configuration file beyond just the filters
> in
> > >> the future.
>
> I think it's a reasonable question - why would a new configuration file
> option
> include support for only a handful of existing arguments but not the rest.
>
I don't see a strong technical problem - enhancing parsing is not hard
work, but I miss a use case for this. The option "--filter" tries to solve
a problem with limited command line size. This is a clean use case and
there and supported options are options that can be used repeatedly on the
command line. Nothing less, nothing more. The format that is used is
designed just for this purpose.
When we would implement an alternative configuration to command line and
system environments, then the use case should be defined first. When the
use case is defined, we can talk about implementation and about good
format. There are a lot of interesting formats, but I miss a reason why the
usage of this alternative configuration can be helpful for pg_dump. Using
external libraries for richer formats means a new dependency, necessity to
solve portability issues, and maybe other issues, and for this there should
be a good use case. Passing a list of tables for dumping doesn't need a
rich format.
I cannot imagine using a config file with generated object names and some
other options together. Maybe if these configurations will not be too long
(then handy written) configuration can be usable. But when I think about
using pg_dump from some bash scripts, then much more practical is using
usual command line options and passing a list of objects by pipe. I really
miss the use case for special pg_dump's config file, and if there is, then
it is very different from a use case for "--filter" option.
> > > This is the correct argument - I will check a possibility to use
> strange
> > > names, but there is the same possibility and functionality like we
> allow
> > > from the command line. So you can use double quoted names. I'll check
> it.
> >
> > I checked
> > echo "+t \"bad Name\"" | /usr/local/pgsql/master/bin/pg_dump
> --filter=/dev/stdin
> > It is working without any problem
>
> I think it couldn't possibly work with newlines, since you call
> pg_get_line().
> I realize that entering a newline into the shell would also be a PITA, but
> that
> could be one *more* reason to support a config file - to allow terrible
> table
> names to be in a file and avoid writing dash tee quote something enter else
> quote in a pg_dump command, or shell script.
>
New patch is working with names that contains multilines
[pavel@localhost postgresql.master]$ psql -At -X -c "select '+t ' ||
quote_ident(table_name) from information_schema.tables where table_name
like 'foo%'"| /usr/local/pgsql/master/bin/pg_dump --filter=/dev/stdin
--
-- PostgreSQL database dump
--
-- Dumped from database version 14devel
-- Dumped by pg_dump version 14devel
-
-- Name: foo boo; Type: TABLE; Schema: public; Owner: pavel
--
CREATE TABLE public."foo
boo" (
a integer
);
ALTER TABLE public."foo
boo" OWNER TO pavel;
--
-- Data for Name: foo boo; Type: TABLE DATA; Schema: public; Owner: pavel
--
COPY public."foo
boo" (a) FROM stdin;
\.
--
-- PostgreSQL database dump complete
--
> I fooled with argument parsing to handle reading from a file in the
> quickest
> way. As written, this fails to handle multiple config files, and special
> table
> names, which need to support arbitrary, logical lines, with quotes
> surrounding
> newlines or other special chars. As written, the --config file is parsed
> *after* all other arguments, so it could override previous args (like
> --no-blobs --no-blogs, --file, --format, --compress, --lock-wait), which I
> guess is bad, so the config file should be processed *during* argument
> parsing.
> Unfortunately, I think that suggests duplicating parsing of all/most the
> argument parsing for config file support - I'd be happy if someone
> suggested a
> better way.
>
> BTW, in your most recent patch:
> s/empty rows/empty lines/
> unbalanced parens: "invalid option type (use [+-]"
>
should be fixed now, thank you for check
Regards
Pavel
> @cfbot: I renamed the patch so please ignore it.
>
> --
> Justin
>
Attachments:
[text/x-patch] pg_dump-filter-option-20201119.patch (13.7K, ../../CAFj8pRBuEOCTGR8VwhQi3_dNg6v=g5zBB4kmACRsnAFiWJKWVA@mail.gmail.com/3-pg_dump-filter-option-20201119.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0aa35cf0c3..068821583f 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -751,6 +751,56 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file.
+ If you use "-" as a filename, the filters are read from stdin.
+ The lines of this file must have the following format:
+<synopsis>
+(+|-)[tnfd] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first character specifies whether the object is to be included
+ (<literal>+</literal>) or excluded (<literal>-</literal>), and the
+ second character specifies the type of object to be filtered:
+ <literal>t</literal> (table),
+ <literal>n</literal> (schema),
+ <literal>f</literal> (foreign server),
+ <literal>d</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
++t mytable1
++f some_foreign_server
+-d mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The lines starting with symbol <literal>#</literal> are ignored.
+ Previous white chars (spaces, tabs) are not allowed. These
+ lines can be used for comments, notes. Empty lines are ignored too.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index dc1d41dd8d..074d57ad3c 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -54,9 +54,11 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -294,6 +296,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
@@ -367,6 +370,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -390,6 +394,7 @@ main(int argc, char **argv)
{"rows-per-insert", required_argument, NULL, 10},
{"include-foreign-data", required_argument, NULL, 11},
{"index-collation-versions-unknown", no_argument, &dopt.coll_unknown, 1},
+ {"include-foreign-data-file", required_argument, NULL, 17},
{NULL, 0, NULL, 0}
};
@@ -607,6 +612,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1035,6 +1044,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18635,3 +18646,219 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+ PQExpBuffer quoted_name = NULL;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+
+ while (pg_get_line_buf(fp, &line))
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+ char *str = line.data;
+
+ lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* ignore empty lines */
+ if (*str == '\0')
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*str == '#')
+ continue;
+
+ if (str[1] == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "line too short",
+ str,
+ lineno);
+
+ if (str[0] == '+')
+ is_include = true;
+ else if (str[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid option type (use [+-])",
+ str,
+ lineno);
+
+ objecttype = str[1];
+ objectname = &str[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ str,
+ lineno);
+
+ if (*objectname == '"')
+ {
+ PQExpBuffer quoted_name;
+ char *ptr = objectname + 1;
+
+ quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+
+ while (1)
+ {
+ if (*ptr == '\0')
+ {
+ if (!pg_get_line_buf(fp, &line))
+ exit_invalid_filter_format(fp,
+ filename,
+ "unexpected end of file",
+ "",
+ lineno);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ ptr = line.data;
+ lineno += 1;
+ }
+
+ appendPQExpBufferChar(quoted_name, *ptr);
+ if (*ptr++ == '"')
+ {
+ if (*ptr == '"')
+ ptr++;
+ else
+ break;
+ }
+ }
+
+ /* check garbage after identifier */
+ while (*ptr != '\0')
+ {
+ if (!isspace(*ptr))
+ exit_invalid_filter_format(fp,
+ filename,
+ "unexpected chars after object name",
+ ptr,
+ lineno);
+ ptr++;
+ }
+
+ objectname = quoted_name->data;
+ }
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ str,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ str,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid object type (use [tndf])",
+ str,
+ lineno);
+ if (quoted_name)
+ {
+ destroyPQExpBuffer(quoted_name);
+ quoted_name = NULL;
+ }
+ }
+
+ pfree(line.data);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filter.pl b/src/bin/pg_dump/t/004_pg_dump_filter.pl
new file mode 100644
index 0000000000..957442c5e4
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filter.pl
@@ -0,0 +1,129 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 22;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = get_new_node('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "+t table_one\n";
+print $inputfile "+t table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "-d table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-t table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-n public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "+f doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid format of filter file/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "+";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid format of filter file/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "+d sometable";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/include filter is not supported for this type of object/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-f someforeignserver";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/exclude filter is not supported for this type of object/,
+ "broken format check");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-19 19:57 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-11-19 19:57 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
>> BTW, in your most recent patch:
>> s/empty rows/empty lines/
>> unbalanced parens: "invalid option type (use [+-]"
>>
>
> should be fixed now, thank you for check
>
minor update - fixed handling of processing names with double quotes inside
>
> Regards
>
> Pavel
>
>
>
>> @cfbot: I renamed the patch so please ignore it.
>>
>> --
>> Justin
>>
>
Attachments:
[text/x-patch] pg_dump-filter-option-20201119-2.patch (13.7K, ../../CAFj8pRBoZ1yY7Q_UBCiRFicKvtpxdzkuyus701A4wKe9z9d-9Q@mail.gmail.com/3-pg_dump-filter-option-20201119-2.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0aa35cf0c3..068821583f 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -751,6 +751,56 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file.
+ If you use "-" as a filename, the filters are read from stdin.
+ The lines of this file must have the following format:
+<synopsis>
+(+|-)[tnfd] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first character specifies whether the object is to be included
+ (<literal>+</literal>) or excluded (<literal>-</literal>), and the
+ second character specifies the type of object to be filtered:
+ <literal>t</literal> (table),
+ <literal>n</literal> (schema),
+ <literal>f</literal> (foreign server),
+ <literal>d</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
++t mytable1
++f some_foreign_server
+-d mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The lines starting with symbol <literal>#</literal> are ignored.
+ Previous white chars (spaces, tabs) are not allowed. These
+ lines can be used for comments, notes. Empty lines are ignored too.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index dc1d41dd8d..e2567ca606 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -54,9 +54,11 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -294,6 +296,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
@@ -367,6 +370,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -390,6 +394,7 @@ main(int argc, char **argv)
{"rows-per-insert", required_argument, NULL, 10},
{"include-foreign-data", required_argument, NULL, 11},
{"index-collation-versions-unknown", no_argument, &dopt.coll_unknown, 1},
+ {"include-foreign-data-file", required_argument, NULL, 17},
{NULL, 0, NULL, 0}
};
@@ -607,6 +612,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1035,6 +1044,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18635,3 +18646,219 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+ PQExpBuffer quoted_name = NULL;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+
+ while (pg_get_line_buf(fp, &line))
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+ char *str = line.data;
+
+ lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* ignore empty lines */
+ if (*str == '\0')
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*str == '#')
+ continue;
+
+ if (str[1] == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "line too short",
+ str,
+ lineno);
+
+ if (str[0] == '+')
+ is_include = true;
+ else if (str[0] == '-')
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid option type (use [+-])",
+ str,
+ lineno);
+
+ objecttype = str[1];
+ objectname = &str[2];
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ str,
+ lineno);
+
+ if (*objectname == '"')
+ {
+ PQExpBuffer quoted_name;
+ char *ptr = objectname + 1;
+
+ quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+
+ while (1)
+ {
+ if (*ptr == '\0')
+ {
+ if (!pg_get_line_buf(fp, &line))
+ exit_invalid_filter_format(fp,
+ filename,
+ "unexpected end of file",
+ "",
+ lineno);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ ptr = line.data;
+ lineno += 1;
+ }
+
+ appendPQExpBufferChar(quoted_name, *ptr);
+ if (*ptr++ == '"')
+ {
+ if (*ptr == '"')
+ appendPQExpBufferChar(quoted_name, *ptr++);
+ else
+ break;
+ }
+ }
+
+ /* check garbage after identifier */
+ while (*ptr != '\0')
+ {
+ if (!isspace(*ptr))
+ exit_invalid_filter_format(fp,
+ filename,
+ "unexpected chars after object name",
+ ptr,
+ lineno);
+ ptr++;
+ }
+
+ objectname = quoted_name->data;
+ }
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'n')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ str,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ str,
+ lineno);
+ }
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "invalid object type (use [tndf])",
+ str,
+ lineno);
+ if (quoted_name)
+ {
+ destroyPQExpBuffer(quoted_name);
+ quoted_name = NULL;
+ }
+ }
+
+ pfree(line.data);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filter.pl b/src/bin/pg_dump/t/004_pg_dump_filter.pl
new file mode 100644
index 0000000000..957442c5e4
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filter.pl
@@ -0,0 +1,129 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 22;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = get_new_node('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "+t table_one\n";
+print $inputfile "+t table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "-d table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-t table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-n public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "+f doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid format of filter file/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "+";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid format of filter file/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "+d sometable";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/include filter is not supported for this type of object/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-f someforeignserver";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/exclude filter is not supported for this type of object/,
+ "broken format check");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-25 18:25 Dean Rasheed <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Dean Rasheed @ 2020-11-25 18:25 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
On Thu, 19 Nov 2020 at 19:57, Pavel Stehule <[email protected]> wrote:
>
> minor update - fixed handling of processing names with double quotes inside
>
I see this is marked RFC, but reading the thread it doesn't feel like
we have reached consensus on the design for this feature.
I agree that being able to configure pg_dump via a config file would
be very useful, but the syntax proposed here feels much more like a
hacked-up syntax designed to meet this one use case, rather than a
good general-purpose design that can be easily extended.
IMO, a pg_dump config file should be able to specify all options
currently supported through the command line, and vice versa (subject
to command line length limits), with a single common code path for
handling options. That way, any new options we add will work on the
command line and in config files. Likewise, the user should only need
to learn one set of options, and have the choice of specifying them on
the command line or in a config file (or a mix of both).
I can imagine eventually supporting multiple different file formats,
each just being a different representation of the same data, so
perhaps this could work with 2 new options:
--option-file-format=plain|yaml|json|...
--option-file=filename
with "plain" being the default initial implementation, which might be
something like our current postgresql.conf file format.
Also, I think we should allow multiple "--option-file" arguments
(e.g., to list different object types in different files), and for a
config file to contain its own "--option-file" arguments, to allow
config files to include other config files.
The current design feels far too limited to me, and requires new code
and new syntax to be added each time we extend it, so I'm -1 on this
patch as it stands.
Regards,
Dean
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-25 19:29 Pavel Stehule <[email protected]>
parent: Dean Rasheed <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-11-25 19:29 UTC (permalink / raw)
To: Dean Rasheed <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Hi
st 25. 11. 2020 v 19:25 odesílatel Dean Rasheed <[email protected]>
napsal:
> On Thu, 19 Nov 2020 at 19:57, Pavel Stehule <[email protected]>
> wrote:
> >
> > minor update - fixed handling of processing names with double quotes
> inside
> >
>
> I see this is marked RFC, but reading the thread it doesn't feel like
> we have reached consensus on the design for this feature.
>
> I agree that being able to configure pg_dump via a config file would
> be very useful, but the syntax proposed here feels much more like a
> hacked-up syntax designed to meet this one use case, rather than a
> good general-purpose design that can be easily extended.
>
Nobody sent a real use case for introducing the config file. There was a
discussion about formats, and you introduce other dimensions and
variability.
But I don't understand why? What is a use case? What is a benefit against
command line, or libpq variables? And why should config files be better as
a solution for limited length of command line, when I need to dump
thousands of tables exactly specified?
Regards
Pavel
> IMO, a pg_dump config file should be able to specify all options
> currently supported through the command line, and vice versa (subject
> to command line length limits), with a single common code path for
> handling options. That way, any new options we add will work on the
> command line and in config files. Likewise, the user should only need
> to learn one set of options, and have the choice of specifying them on
> the command line or in a config file (or a mix of both).
>
> I can imagine eventually supporting multiple different file formats,
> each just being a different representation of the same data, so
> perhaps this could work with 2 new options:
>
> --option-file-format=plain|yaml|json|...
> --option-file=filename
>
> with "plain" being the default initial implementation, which might be
> something like our current postgresql.conf file format.
>
> Also, I think we should allow multiple "--option-file" arguments
> (e.g., to list different object types in different files), and for a
> config file to contain its own "--option-file" arguments, to allow
> config files to include other config files.
>
> The current design feels far too limited to me, and requires new code
> and new syntax to be added each time we extend it, so I'm -1 on this
> patch as it stands.
This new syntax tries to be consistent and simple. It really doesn't try to
implement an alternative configuration file for pg_dump. The code is simple
and can be easily extended.
What are the benefits of supporting multiple formats?
> Regards,
> Dean
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-25 20:00 Tom Lane <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 2 replies; 181+ messages in thread
From: Tom Lane @ 2020-11-25 20:00 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Pavel Stehule <[email protected]> writes:
> st 25. 11. 2020 v 19:25 odesílatel Dean Rasheed <[email protected]>
> napsal:
>> I agree that being able to configure pg_dump via a config file would
>> be very useful, but the syntax proposed here feels much more like a
>> hacked-up syntax designed to meet this one use case, rather than a
>> good general-purpose design that can be easily extended.
> But I don't understand why? What is a use case? What is a benefit against
> command line, or libpq variables? And why should config files be better as
> a solution for limited length of command line, when I need to dump
> thousands of tables exactly specified?
Because next week somebody will want to dump thousands of functions
selected by name, or schemas selected by name, etc etc. I agree with
the position that we don't want a single-purpose solution. The idea
that the syntax should match the command line switch syntax seems
reasonable, though I'm not wedded to it. (One thing to consider is
how painful will it be for people to quote table names containing
funny characters, for instance. On the command line, we largely
depend on the shell's quoting behavior to solve that, but we'd not
have that infrastructure when reading from a file.)
> What are the benefits of supporting multiple formats?
Yeah, that part of Dean's sketch seemed like overkill to me too.
I wasn't very excited about multiple switch files either, though
depending on how the implementation is done, that could be simple
enough to be in the might-as-well category.
One other point that I'm wondering about is that there's really no
value in doing anything here until you get to some thousands of
table names; as long as the list fits in the shell's command line
length limit, you might as well just make a shell script file.
Does pg_dump really have sane performance for that situation, or
are we soon going to be fielding requests to make it not be O(N^2)
in the number of listed tables?
regards, tom lane
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-26 06:42 Pavel Stehule <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-11-26 06:42 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
st 25. 11. 2020 v 21:00 odesílatel Tom Lane <[email protected]> napsal:
> Pavel Stehule <[email protected]> writes:
> > st 25. 11. 2020 v 19:25 odesílatel Dean Rasheed <
> [email protected]>
> > napsal:
> >> I agree that being able to configure pg_dump via a config file would
> >> be very useful, but the syntax proposed here feels much more like a
> >> hacked-up syntax designed to meet this one use case, rather than a
> >> good general-purpose design that can be easily extended.
>
> > But I don't understand why? What is a use case? What is a benefit against
> > command line, or libpq variables? And why should config files be better
> as
> > a solution for limited length of command line, when I need to dump
> > thousands of tables exactly specified?
>
> Because next week somebody will want to dump thousands of functions
> selected by name, or schemas selected by name, etc etc. I agree with
> the position that we don't want a single-purpose solution. The idea
> that the syntax should match the command line switch syntax seems
> reasonable, though I'm not wedded to it. (One thing to consider is
> how painful will it be for people to quote table names containing
> funny characters, for instance. On the command line, we largely
> depend on the shell's quoting behavior to solve that, but we'd not
> have that infrastructure when reading from a file.)
>
This is not a problem with the current patch - and the last version of this
patch supports well obscure names.
There was a requirement for supporting all and future pg_dump options - ok
it can make sense. I have not a problem to use instead a line format
"option argument" or "long-option=argument"
This format - can it be a solution? I'll try to rewrite the parser for this
format.
It is implementable, but this is in collision with Stephen's requirement
for human well readable format designed for handy writing. There are
requests that have no intersection. Well readable format needs a more
complex parser. And machine generating in this format needs more fork -
generating flat file is more simple and more robust than generating JSON or
YAML.
>
> > What are the benefits of supporting multiple formats?
>
> Yeah, that part of Dean's sketch seemed like overkill to me too.
>
> I wasn't very excited about multiple switch files either, though
> depending on how the implementation is done, that could be simple
> enough to be in the might-as-well category.
>
> One other point that I'm wondering about is that there's really no
> value in doing anything here until you get to some thousands of
> table names; as long as the list fits in the shell's command line
> length limit, you might as well just make a shell script file.
> Does pg_dump really have sane performance for that situation, or
> are we soon going to be fielding requests to make it not be O(N^2)
> in the number of listed tables?
>
Performance is another factor, but the command line limit can be easily
touched when table names have maximum width.
> regards, tom lane
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-26 11:50 Dean Rasheed <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Dean Rasheed @ 2020-11-26 11:50 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Justin Pryzby <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
On Thu, 26 Nov 2020 at 06:43, Pavel Stehule <[email protected]> wrote:
>
> st 25. 11. 2020 v 21:00 odesílatel Tom Lane <[email protected]> napsal:
>>
>> (One thing to consider is
>> how painful will it be for people to quote table names containing
>> funny characters, for instance. On the command line, we largely
>> depend on the shell's quoting behavior to solve that, but we'd not
>> have that infrastructure when reading from a file.)
>
> This is not a problem with the current patch - and the last version of this patch supports well obscure names.
>
Actually, that raises a different possible benefit of passing options
in an options file -- if the user wants to pass in a table name
pattern, it can be a nuisance if the shell's argument processing does
additional unwanted things like globbing and environment variable
substitutions. Using an options file could provide a handy way to
ensure that any option values are interpreted exactly as written,
without any additional mangling.
> There was a requirement for supporting all and future pg_dump options - ok it can make sense. I have not a problem to use instead a line format
>
> "option argument" or "long-option=argument"
>
> This format - can it be a solution? I'll try to rewrite the parser for this format.
>
Yes, that's the sort of thing I was thinking of, to make the feature
more general-purpose.
> It is implementable, but this is in collision with Stephen's requirement for human well readable format designed for handy writing. There are requests that have no intersection. Well readable format needs a more complex parser. And machine generating in this format needs more fork - generating flat file is more simple and more robust than generating JSON or YAML.
>
To be clear, I wasn't suggesting that this patch implement multiple
formats. Just the "plain" format above. Other formats like YAML might
well be more human-readable, and be able to take advantage of values
that are lists to avoid repeating option names, and they would have
the advantage of being readable and writable by other standard tools,
which might be useful. But I think such things would be for the
future. Maybe no one will ever add support for other formats, or maybe
someone will just write a separate external tool to convert YAML or
JSON to our plain format. I don't know. But supporting all pg_dump
options makes more things possible.
>> I wasn't very excited about multiple switch files either, though
>> depending on how the implementation is done, that could be simple
>> enough to be in the might-as-well category.
>>
That's what I was hoping.
Regards,
Dean
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-26 16:02 Tom Lane <[email protected]>
parent: Dean Rasheed <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Tom Lane @ 2020-11-26 16:02 UTC (permalink / raw)
To: Dean Rasheed <[email protected]>; +Cc: Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Dean Rasheed <[email protected]> writes:
> Actually, that raises a different possible benefit of passing options
> in an options file -- if the user wants to pass in a table name
> pattern, it can be a nuisance if the shell's argument processing does
> additional unwanted things like globbing and environment variable
> substitutions. Using an options file could provide a handy way to
> ensure that any option values are interpreted exactly as written,
> without any additional mangling.
Huh? Any format we might devise, or borrow, will have to have some
kind of escaping/quoting convention. The idea that "we don't need
that" tends to lead to very ugly workarounds later.
I do agree that the shell's quoting conventions are pretty messy
and so those aren't the ones we should borrow. We could do a lot
worse than to use some established data format like JSON or YAML.
Given that we already have src/common/jsonapi.c, it seems like
JSON would be the better choice of those two.
regards, tom lane
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-27 16:56 Stephen Frost <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Stephen Frost @ 2020-11-27 16:56 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Dean Rasheed <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Greetings,
* Tom Lane ([email protected]) wrote:
> Dean Rasheed <[email protected]> writes:
> > Actually, that raises a different possible benefit of passing options
> > in an options file -- if the user wants to pass in a table name
> > pattern, it can be a nuisance if the shell's argument processing does
> > additional unwanted things like globbing and environment variable
> > substitutions. Using an options file could provide a handy way to
> > ensure that any option values are interpreted exactly as written,
> > without any additional mangling.
>
> Huh? Any format we might devise, or borrow, will have to have some
> kind of escaping/quoting convention. The idea that "we don't need
> that" tends to lead to very ugly workarounds later.
Agreed.
> I do agree that the shell's quoting conventions are pretty messy
> and so those aren't the ones we should borrow. We could do a lot
> worse than to use some established data format like JSON or YAML.
> Given that we already have src/common/jsonapi.c, it seems like
> JSON would be the better choice of those two.
JSON doesn't support comments, something that's really useful to have in
configuration files, so I don't agree that it's a sensible thing to use
in this case. JSON also isn't very forgiving, which is also
unfortunate and makes for a poor choice.
This is why I was suggesting TOML up-thread, which is MIT licensed, has
been around for a number of years, supports comments, has sensible
quoting that's easier to deal with than the shell, and has a C (C99)
implementation. It's also used in quite a few other projects.
In a quick look, I suspect it might also be something that could be used
to replace our existing hand-hacked postgresql.conf parser and give us
the ability to handle things a bit cleaner there too...
Thanks,
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-27 18:45 Stephen Frost <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Stephen Frost @ 2020-11-27 18:45 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Greetings,
* Pavel Stehule ([email protected]) wrote:
> > I agree that being able to configure pg_dump via a config file would
> > be very useful, but the syntax proposed here feels much more like a
> > hacked-up syntax designed to meet this one use case, rather than a
> > good general-purpose design that can be easily extended.
>
> Nobody sent a real use case for introducing the config file. There was a
> discussion about formats, and you introduce other dimensions and
> variability.
I'm a bit baffled by this because it seems abundently clear to me that
being able to have a config file for pg_dump would be extremely helpful.
There's no shortage of times that I've had to hack up a shell script and
figure out quoting and set up the right set of options for pg_dump,
resulting in things like:
pg_dump \
--host=myserver.com \
--username=postgres \
--schema=public \
--schema=myschema \
--no-comments \
--no-tablespaces \
--file=somedir \
--format=d \
--jobs=5
which really is pretty grotty. Being able to have a config file that
has proper comments would be much better and we could start to extend to
things like "please export schema A to directory A, schema B to
directory B" and other ways of selecting source and destination, and
imagine if we could validate it too, eg:
pg_dump --config=whatever --dry-run
or --check-config maybe.
This isn't a new concept either- export and import tools for other
databases have similar support, eg: Oracle's imp/exp tool, mysqldump
(see: https://dev.mysql.com/doc/refman/8.0/en/option-files.html which
has a TOML-looking format too), pgloader of course has a config file,
etc. We certainly aren't in novel territory here.
Thanks,
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-28 20:14 Pavel Stehule <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-11-28 20:14 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Hi
st 25. 11. 2020 v 21:00 odesílatel Tom Lane <[email protected]> napsal:
> Pavel Stehule <[email protected]> writes:
> > st 25. 11. 2020 v 19:25 odesílatel Dean Rasheed <
> [email protected]>
> > napsal:
> >> I agree that being able to configure pg_dump via a config file would
> >> be very useful, but the syntax proposed here feels much more like a
> >> hacked-up syntax designed to meet this one use case, rather than a
> >> good general-purpose design that can be easily extended.
>
> > But I don't understand why? What is a use case? What is a benefit against
> > command line, or libpq variables? And why should config files be better
> as
> > a solution for limited length of command line, when I need to dump
> > thousands of tables exactly specified?
>
> Because next week somebody will want to dump thousands of functions
> selected by name, or schemas selected by name, etc etc. I agree with
> the position that we don't want a single-purpose solution. The idea
> that the syntax should match the command line switch syntax seems
> reasonable, though I'm not wedded to it. (One thing to consider is
> how painful will it be for people to quote table names containing
> funny characters, for instance. On the command line, we largely
> depend on the shell's quoting behavior to solve that, but we'd not
> have that infrastructure when reading from a file.)
>
> > What are the benefits of supporting multiple formats?
>
> Yeah, that part of Dean's sketch seemed like overkill to me too.
>
> I wasn't very excited about multiple switch files either, though
> depending on how the implementation is done, that could be simple
> enough to be in the might-as-well category.
>
> One other point that I'm wondering about is that there's really no
> value in doing anything here until you get to some thousands of
> table names; as long as the list fits in the shell's command line
> length limit, you might as well just make a shell script file.
> Does pg_dump really have sane performance for that situation, or
> are we soon going to be fielding requests to make it not be O(N^2)
> in the number of listed tables?
>
Here is a fresh implementation. I used the name of a new option -
"options-file". Looks more accurate than "config", but the name can be
changed easily anytime.
Any short or long option can be read from this file in simple format - one
option per line. Arguments inside double quotes can be multi lined. Row
comments started by # and can be used everywhere.
The implementation is very generic - support of new options doesn't require
change of this new part code. The parser can ignore white spaces almost
everywhere, where it has sense.
The option should start with "-" or "--" in the options file too, because
this is necessary for good detection if the option is short or long.
Regards
Pavel
> regards, tom lane
>
Attachments:
[text/x-patch] pg_dump-options-file.patch (28.0K, ../../CAFj8pRCtfaoD0QOAxrcQCqvJBbTr4ryhBkk2MJQhUJK-dLjdPw@mail.gmail.com/3-pg_dump-options-file.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0aa35cf0c3..efdb53f069 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -956,6 +956,34 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--options-file=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read options from file (one option per line). Short or long options
+ are supported. If you use "-" as a filename, the filters are read
+ from stdin.
+ </para>
+
+ <para>
+ With the following options file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
+-t mytable1
+--include-foreign-data=some_foreign_server
+--exclude-table-data=mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The text after symbol <literal>#</literal> is ignored. This can
+ be used for comments, notes. Empty lines are ignored too.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--quote-all-identifiers</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index dc1d41dd8d..96acb8a1d5 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -54,9 +54,11 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -129,6 +131,17 @@ static const CatalogId nilCatalogId = {0, 0};
static bool have_extra_float_digits = false;
static int extra_float_digits;
+static const char *filename = NULL;
+static const char *format = "p";
+static bool g_verbose = false;
+static const char *dumpencoding = NULL;
+static const char *dumpsnapshot = NULL;
+static char *use_role = NULL;
+static long rowsPerInsert;
+static int numWorkers = 1;
+static int compressLevel = -1;
+static char *optsfilename = NULL;
+
/*
* The default number of rows per INSERT when
* --inserts is specified without --rows-per-insert
@@ -294,14 +307,226 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_options_from_file(char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname);
+
+#define OPTIONS_FILE_OPT_NUMBER 12
+
+/*
+ * It assign the values of options to related DumpOption fields or to
+ * some global values. It is called from twice. First, for processing
+ * the command line argumens. Second, for processing an options from
+ * options file.
+ */
+static bool
+process_option(int opt,
+ char *optargstr,
+ DumpOptions *dopt,
+ const char *progname)
+{
+ char *endptr;
+
+ switch (opt)
+ {
+ case 'a': /* Dump data only */
+ dopt->dataOnly = true;
+ break;
+
+ case 'b': /* Dump blobs */
+ dopt->outputBlobs = true;
+ break;
+
+ case 'B': /* Don't dump blobs */
+ dopt->dontOutputBlobs = true;
+ break;
+
+ case 'c': /* clean (i.e., drop) schema prior to create */
+ dopt->outputClean = 1;
+ break;
+
+ case 'C': /* Create DB */
+ dopt->outputCreateDB = 1;
+ break;
+
+ case 'd': /* database name */
+ dopt->cparams.dbname = pg_strdup(optargstr);
+ break;
+
+ case 'E': /* Dump encoding */
+ dumpencoding = pg_strdup(optargstr);
+ break;
+
+ case 'f':
+ filename = pg_strdup(optargstr);
+ break;
+
+ case 'F':
+ format = pg_strdup(optargstr);
+ break;
+
+ case 'h': /* server host */
+ dopt->cparams.pghost = pg_strdup(optargstr);
+ break;
+
+ case 'j': /* number of dump jobs */
+ numWorkers = atoi(optargstr);
+ break;
+
+ case 'n': /* include schema(s) */
+ simple_string_list_append(&schema_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'N': /* exclude schema(s) */
+ simple_string_list_append(&schema_exclude_patterns, optargstr);
+ break;
+
+ case 'O': /* Don't reconnect to match owner */
+ dopt->outputNoOwner = 1;
+ break;
+
+ case 'p': /* server port */
+ dopt->cparams.pgport = pg_strdup(optargstr);
+ break;
+
+ case 'R':
+ /* no-op, still accepted for backwards compatibility */
+ break;
+
+ case 's': /* dump schema only */
+ dopt->schemaOnly = true;
+ break;
+
+ case 'S': /* Username for superuser in plain text output */
+ dopt->outputSuperuser = pg_strdup(optargstr);
+ break;
+
+ case 't': /* include table(s) */
+ simple_string_list_append(&table_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'T': /* exclude table(s) */
+ simple_string_list_append(&table_exclude_patterns, optargstr);
+ break;
+
+ case 'U':
+ dopt->cparams.username = pg_strdup(optargstr);
+ break;
+
+ case 'v': /* verbose */
+ g_verbose = true;
+ pg_logging_increase_verbosity();
+ break;
+
+ case 'w':
+ dopt->cparams.promptPassword = TRI_NO;
+ break;
+
+ case 'W':
+ dopt->cparams.promptPassword = TRI_YES;
+ break;
+
+ case 'x': /* skip ACL dump */
+ dopt->aclsSkip = true;
+ break;
+
+ case 'Z': /* Compression Level */
+ compressLevel = atoi(optargstr);
+ if (compressLevel < 0 || compressLevel > 9)
+ {
+ pg_log_error("compression level must be in range 0..9");
+ return false;
+ }
+ break;
+
+ case 0:
+ /* This covers the long options. */
+ break;
+
+ case 2: /* lock-wait-timeout */
+ dopt->lockWaitTimeout = pg_strdup(optargstr);
+ break;
+
+ case 3: /* SET ROLE */
+ use_role = pg_strdup(optargstr);
+ break;
+
+ case 4: /* exclude table(s) data */
+ simple_string_list_append(&tabledata_exclude_patterns, optargstr);
+ break;
+
+ case 5: /* section */
+ set_dump_section(optargstr, &dopt->dumpSections);
+ break;
+
+ case 6: /* snapshot */
+ dumpsnapshot = pg_strdup(optargstr);
+ break;
+
+ case 7: /* no-sync */
+ dosync = false;
+ break;
+
+ case 8:
+ have_extra_float_digits = true;
+ extra_float_digits = atoi(optargstr);
+ if (extra_float_digits < -15 || extra_float_digits > 3)
+ {
+ pg_log_error("extra_float_digits must be in range -15..3");
+ return false;
+ }
+ break;
+
+ case 9: /* inserts */
+
+ /*
+ * dump_inserts also stores --rows-per-insert, careful not to
+ * overwrite that.
+ */
+ if (dopt->dump_inserts == 0)
+ dopt->dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
+ break;
+ case 10: /* rows per insert */
+ errno = 0;
+ rowsPerInsert = strtol(optargstr, &endptr, 10);
+
+ if (endptr == optargstr || *endptr != '\0' ||
+ rowsPerInsert <= 0 || rowsPerInsert > INT_MAX ||
+ errno == ERANGE)
+ {
+ pg_log_error("rows-per-insert must be in range %d..%d",
+ 1, INT_MAX);
+ return false;
+ }
+ dopt->dump_inserts = (int) rowsPerInsert;
+ break;
+
+ case 11: /* include foreign data */
+ simple_string_list_append(&foreign_servers_include_patterns,
+ optargstr);
+ break;
+
+ case OPTIONS_FILE_OPT_NUMBER: /* filter implementation */
+ optsfilename = pg_strdup(optargstr);
+ break;
+
+ default:
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ return false;
+ }
+
+ return true;
+}
int
main(int argc, char **argv)
{
int c;
- const char *filename = NULL;
- const char *format = "p";
TableInfo *tblinfo;
int numTables;
DumpableObject **dobjs;
@@ -309,20 +534,11 @@ main(int argc, char **argv)
DumpableObject *boundaryObjs;
int i;
int optindex;
- char *endptr;
RestoreOptions *ropt;
Archive *fout; /* the script file */
- bool g_verbose = false;
- const char *dumpencoding = NULL;
- const char *dumpsnapshot = NULL;
- char *use_role = NULL;
- long rowsPerInsert;
- int numWorkers = 1;
- int compressLevel = -1;
int plainText = 0;
ArchiveFormat archiveFormat = archUnknown;
ArchiveMode archiveMode;
-
static DumpOptions dopt;
static struct option long_options[] = {
@@ -387,13 +603,17 @@ main(int argc, char **argv)
{"no-subscriptions", no_argument, &dopt.no_subscriptions, 1},
{"no-sync", no_argument, NULL, 7},
{"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1},
+ {"options-file", required_argument, NULL, OPTIONS_FILE_OPT_NUMBER},
{"rows-per-insert", required_argument, NULL, 10},
{"include-foreign-data", required_argument, NULL, 11},
{"index-collation-versions-unknown", no_argument, &dopt.coll_unknown, 1},
+ {"include-foreign-data-file", required_argument, NULL, 17},
{NULL, 0, NULL, 0}
};
+ const char *short_options = "abBcCd:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:";
+
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_dump"));
@@ -422,197 +642,20 @@ main(int argc, char **argv)
InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "abBcCd:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:",
+ while ((c = getopt_long(argc, argv, short_options,
long_options, &optindex)) != -1)
{
- switch (c)
- {
- case 'a': /* Dump data only */
- dopt.dataOnly = true;
- break;
-
- case 'b': /* Dump blobs */
- dopt.outputBlobs = true;
- break;
-
- case 'B': /* Don't dump blobs */
- dopt.dontOutputBlobs = true;
- break;
-
- case 'c': /* clean (i.e., drop) schema prior to create */
- dopt.outputClean = 1;
- break;
-
- case 'C': /* Create DB */
- dopt.outputCreateDB = 1;
- break;
-
- case 'd': /* database name */
- dopt.cparams.dbname = pg_strdup(optarg);
- break;
-
- case 'E': /* Dump encoding */
- dumpencoding = pg_strdup(optarg);
- break;
-
- case 'f':
- filename = pg_strdup(optarg);
- break;
-
- case 'F':
- format = pg_strdup(optarg);
- break;
-
- case 'h': /* server host */
- dopt.cparams.pghost = pg_strdup(optarg);
- break;
-
- case 'j': /* number of dump jobs */
- numWorkers = atoi(optarg);
- break;
-
- case 'n': /* include schema(s) */
- simple_string_list_append(&schema_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'N': /* exclude schema(s) */
- simple_string_list_append(&schema_exclude_patterns, optarg);
- break;
-
- case 'O': /* Don't reconnect to match owner */
- dopt.outputNoOwner = 1;
- break;
-
- case 'p': /* server port */
- dopt.cparams.pgport = pg_strdup(optarg);
- break;
-
- case 'R':
- /* no-op, still accepted for backwards compatibility */
- break;
-
- case 's': /* dump schema only */
- dopt.schemaOnly = true;
- break;
-
- case 'S': /* Username for superuser in plain text output */
- dopt.outputSuperuser = pg_strdup(optarg);
- break;
-
- case 't': /* include table(s) */
- simple_string_list_append(&table_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'T': /* exclude table(s) */
- simple_string_list_append(&table_exclude_patterns, optarg);
- break;
-
- case 'U':
- dopt.cparams.username = pg_strdup(optarg);
- break;
-
- case 'v': /* verbose */
- g_verbose = true;
- pg_logging_increase_verbosity();
- break;
-
- case 'w':
- dopt.cparams.promptPassword = TRI_NO;
- break;
-
- case 'W':
- dopt.cparams.promptPassword = TRI_YES;
- break;
-
- case 'x': /* skip ACL dump */
- dopt.aclsSkip = true;
- break;
-
- case 'Z': /* Compression Level */
- compressLevel = atoi(optarg);
- if (compressLevel < 0 || compressLevel > 9)
- {
- pg_log_error("compression level must be in range 0..9");
- exit_nicely(1);
- }
- break;
-
- case 0:
- /* This covers the long options. */
- break;
-
- case 2: /* lock-wait-timeout */
- dopt.lockWaitTimeout = pg_strdup(optarg);
- break;
-
- case 3: /* SET ROLE */
- use_role = pg_strdup(optarg);
- break;
-
- case 4: /* exclude table(s) data */
- simple_string_list_append(&tabledata_exclude_patterns, optarg);
- break;
-
- case 5: /* section */
- set_dump_section(optarg, &dopt.dumpSections);
- break;
-
- case 6: /* snapshot */
- dumpsnapshot = pg_strdup(optarg);
- break;
-
- case 7: /* no-sync */
- dosync = false;
- break;
-
- case 8:
- have_extra_float_digits = true;
- extra_float_digits = atoi(optarg);
- if (extra_float_digits < -15 || extra_float_digits > 3)
- {
- pg_log_error("extra_float_digits must be in range -15..3");
- exit_nicely(1);
- }
- break;
-
- case 9: /* inserts */
-
- /*
- * dump_inserts also stores --rows-per-insert, careful not to
- * overwrite that.
- */
- if (dopt.dump_inserts == 0)
- dopt.dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
- break;
-
- case 10: /* rows per insert */
- errno = 0;
- rowsPerInsert = strtol(optarg, &endptr, 10);
-
- if (endptr == optarg || *endptr != '\0' ||
- rowsPerInsert <= 0 || rowsPerInsert > INT_MAX ||
- errno == ERANGE)
- {
- pg_log_error("rows-per-insert must be in range %d..%d",
- 1, INT_MAX);
- exit_nicely(1);
- }
- dopt.dump_inserts = (int) rowsPerInsert;
- break;
-
- case 11: /* include foreign data */
- simple_string_list_append(&foreign_servers_include_patterns,
- optarg);
- break;
-
- default:
- fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
- exit_nicely(1);
- }
+ if (!process_option(c, optarg, &dopt, progname))
+ exit_nicely(1);
}
+ if (optsfilename)
+ read_options_from_file(optsfilename,
+ &dopt,
+ short_options,
+ long_options,
+ progname);
+
/*
* Non-option argument specifies database name as long as it wasn't
* already specified with -d / --dbname
@@ -1049,6 +1092,7 @@ help(const char *progname)
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
printf(_(" --no-unlogged-table-data do not dump unlogged table data\n"));
printf(_(" --on-conflict-do-nothing add ON CONFLICT DO NOTHING to INSERT commands\n"));
+ printf(_(" --options-file=FILENAME read options from options file\n"));
printf(_(" --quote-all-identifiers quote all identifiers, even if not key words\n"));
printf(_(" --rows-per-insert=NROWS number of rows per INSERT; implies --inserts\n"));
printf(_(" --section=SECTION dump named section (pre-data, data, or post-data)\n"));
@@ -18635,3 +18679,380 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_optfile_format(FILE *fp,
+ char *message,
+ char *optname,
+ int optnamelen,
+ char *line,
+ int lineno)
+{
+ Assert(message);
+
+ if (optnamelen > 0)
+ {
+ Assert(optname);
+ pg_log_error(message, optnamelen, optname, lineno);
+ }
+ else
+ pg_log_error(message, lineno);
+
+ if (line)
+ fprintf(stderr, "LINE %d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Reads an option argument from file. Supports double qoutes
+ * bounded strings. In this case multi lines strings are supported.
+ */
+static void
+read_optarg(FILE *fp,
+ char *filename,
+ char *str,
+ StringInfo line,
+ StringInfo optargument,
+ char *optname,
+ int optnamelen,
+ bool islongopt,
+ int *lineno)
+{
+ if (*str == '\0' || *str == '#')
+ {
+ if (islongopt)
+ exit_invalid_optfile_format(fp,
+ "option '--%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ else
+ exit_invalid_optfile_format(fp,
+ "option '-%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ }
+
+ resetStringInfo(optargument);
+
+ /* simple case */
+ if (*str != '"')
+ {
+ char *start = str;
+
+ /* read first white char */
+ while (*str != '\0' && *str != '#')
+ {
+ if (*str == ' ')
+ break;
+ str++;
+ }
+
+ appendBinaryStringInfo(optargument, start, str - start);
+ }
+ else
+ {
+ appendStringInfoChar(optargument, *str++);
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ /* multiline string, read next line */
+ if (!pg_get_line_buf(fp, line))
+ exit_invalid_optfile_format(fp,
+ "unexpected end of line at line %d",
+ NULL, 0,
+ NULL,
+ *lineno);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ appendStringInfoChar(optargument, '\n');
+
+ str = line->data;
+ (void) pg_strip_crlf(str);
+ *lineno += 1;
+ }
+
+ appendStringInfoChar(optargument, *str);
+
+ if (*str++ == '"')
+ {
+ if (*str == '"')
+ str++;
+ else
+ break;
+ }
+ }
+ }
+
+ /* check garbage after optarg, but ignore white spaces */
+ while (isspace(*str))
+ str++;
+
+ /* at the end there should be EOL or comment symbol */
+ if (*str != '\0' && *str != '#')
+ exit_invalid_optfile_format(fp,
+ "unexpected characters after an option's argument at line %d",
+ NULL, 0,
+ line->data,
+ *lineno);
+}
+
+/*
+ * Returns true, when long option is defined. Assign code,
+ * and check if option has required argument.
+ */
+static bool
+is_recognized_longopt(const struct option *longopts,
+ char *optname,
+ int optnamelen,
+ int *opt,
+ bool *has_arg)
+{
+ int i;
+
+ for (i = 0; longopts[i].name != NULL; i++)
+ {
+ if (strlen(longopts[i].name) == optnamelen &&
+ strncmp(optname, longopts[i].name, optnamelen) == 0)
+ {
+ *has_arg = longopts[i].has_arg == required_argument;
+
+ if (longopts[i].flag == NULL)
+ {
+ *opt = longopts[i].val;
+
+ return true;
+ }
+ else
+ {
+ *longopts[i].flag = longopts[i].val;
+ *opt = 0;
+
+ return true;
+ }
+ }
+ }
+
+ return false;
+}
+
+/*
+ * Returns true, when short option char is defined.
+ */
+static bool
+is_valid_shortopt(const char *optstring,
+ char optname,
+ bool *has_arg)
+{
+ while (*optstring != '\0')
+ {
+ if (*optstring != ':' && *optstring == optname)
+ {
+ *has_arg = optstring[1] == ':';
+ return true;
+ }
+
+ optstring++;
+ }
+
+ return false;
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_options_from_file(char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+ StringInfoData optargument;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+ initStringInfo(&optargument);
+
+ while (pg_get_line_buf(fp, &line))
+ {
+ char *optname;
+ char *str = line.data;
+ int opt;
+ int optnamelen;
+ bool has_arg;
+
+ (void) pg_strip_crlf(str);
+
+ lineno += 1;
+
+ /* skip initial spaces */
+ while (isspace(*str))
+ str++;
+
+ /* Ignore empty lines or lines with hash symbol (comment) */
+ if (*str == '\0' || *str == '#')
+ continue;
+
+ if (*str++ != '-')
+ exit_invalid_optfile_format(fp,
+ "non option arguments are not allowed in options file at line %d",
+ NULL, 0,
+ line.data,
+ lineno);
+
+ if (*str == '-')
+ {
+ /* process long option */
+ str++;
+ optname = str++;
+
+ while (!isspace(*str) && *str != '=' && *str != '\0')
+ str++;
+
+ optnamelen = str - optname;
+
+ if (is_recognized_longopt(longopts,
+ optname,
+ optnamelen,
+ &opt,
+ &has_arg))
+ {
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Don't allow --options-file inside options file.
+ * It is simple protection against cycle.
+ */
+ if (opt == OPTIONS_FILE_OPT_NUMBER)
+ exit_invalid_optfile_format(fp,
+ "option '%.*s' cannot be used in options file at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+
+ if (has_arg)
+ {
+ /* skip optional = */
+ if (*str == '=')
+ str++;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ read_optarg(fp,
+ filename,
+ str,
+ &line,
+ &optargument,
+ optname,
+ optnamelen,
+ true,
+ &lineno);
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ exit_invalid_optfile_format(fp,
+ "option '--%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ }
+ }
+ else
+ exit_invalid_optfile_format(fp,
+ "unrecognized option '--%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ }
+ else
+ {
+ /* process short option */
+ optname = str++;
+ optnamelen = 1;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (is_valid_shortopt(optstring, *optname, &has_arg))
+ {
+ if (has_arg)
+ {
+ read_optarg(fp,
+ filename,
+ str,
+ &line,
+ &optargument,
+ optname,
+ optnamelen,
+ false,
+ &lineno);
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ exit_invalid_optfile_format(fp,
+ "option '-%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ }
+ }
+ else
+ exit_invalid_optfile_format(fp,
+ "invalid option '-%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+
+ opt = *optname;
+ }
+
+ if (opt != 0 &&
+ !process_option(opt, optargument.data, dopt, progname))
+ {
+ fclose(fp);
+ exit_nicely(-1);
+ }
+ }
+
+ pfree(line.data);
+ pfree(optargument.data);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_optsfile.pl b/src/bin/pg_dump/t/004_pg_dump_optsfile.pl
new file mode 100644
index 0000000000..0955e28a85
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_optsfile.pl
@@ -0,0 +1,165 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 30;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = get_new_node('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-t table_one #comment\n";
+print $inputfile "-t table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "--exclude-table-data=table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-T table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-N public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "--include-foreign-data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: non option arguments are not allowed in options file at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid option '-' at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-t";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '-t' requires an argument at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-a someforeignserver";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '-a' doesn't allow an argument at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-r";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid option '-r' at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "--doesnt-exists";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: unrecognized option '--doesnt-exists' at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "--data-only badparameter";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '--data-only' doesn't allow an argument at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "--table";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '--table' requires an argument at line 1/,
+ "broken format check");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-28 21:14 Pavel Stehule <[email protected]>
parent: Stephen Frost <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2020-11-28 21:14 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
pá 27. 11. 2020 v 19:45 odesílatel Stephen Frost <[email protected]>
napsal:
> Greetings,
>
> * Pavel Stehule ([email protected]) wrote:
> > > I agree that being able to configure pg_dump via a config file would
> > > be very useful, but the syntax proposed here feels much more like a
> > > hacked-up syntax designed to meet this one use case, rather than a
> > > good general-purpose design that can be easily extended.
> >
> > Nobody sent a real use case for introducing the config file. There was a
> > discussion about formats, and you introduce other dimensions and
> > variability.
>
> I'm a bit baffled by this because it seems abundently clear to me that
> being able to have a config file for pg_dump would be extremely helpful.
> There's no shortage of times that I've had to hack up a shell script and
> figure out quoting and set up the right set of options for pg_dump,
> resulting in things like:
>
> pg_dump \
> --host=myserver.com \
> --username=postgres \
> --schema=public \
> --schema=myschema \
> --no-comments \
> --no-tablespaces \
> --file=somedir \
> --format=d \
> --jobs=5
>
> which really is pretty grotty. Being able to have a config file that
> has proper comments would be much better and we could start to extend to
> things like "please export schema A to directory A, schema B to
> directory B" and other ways of selecting source and destination, and
> imagine if we could validate it too, eg:
>
> pg_dump --config=whatever --dry-run
>
> or --check-config maybe.
>
> This isn't a new concept either- export and import tools for other
> databases have similar support, eg: Oracle's imp/exp tool, mysqldump
> (see: https://dev.mysql.com/doc/refman/8.0/en/option-files.html which
> has a TOML-looking format too), pgloader of course has a config file,
> etc. We certainly aren't in novel territory here
>
Still, I am not a fan of this. pg_dump is a simple tool for simple
purposes. It is not a pgloader or any ETL tool. It can be changed in
future, maybe, but still, why? And any time, there will be a question if
pg_dump is a good foundation for massive enhancement in ETL direction. The
development in C is expensive and pg_dump is too Postgres specific, so I
cannot imagine so pg_dump will be used for some complex tasks directly, and
there will be requirements for special configuration. When we have a
pgloader, then we don't need to move pg_dump in the pgloader direction.
Anyway - new patch allows to store any options (one per line) with possible
comments (everywhere in line) and argument's can be across more lines. It
hasn't any more requirements on memory or CPU.
Regards
Pavel
>
> Thanks,
>
> Stephen
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* [PATCH 1/4] Re: proposal: possibility to read dumped table's name from file
@ 2020-11-28 21:53 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Justin Pryzby @ 2020-11-28 21:53 UTC (permalink / raw)
Pavel Stehule
Nov 28
---
doc/src/sgml/ref/pg_dump.sgml | 28 +
src/bin/pg_dump/pg_dump.c | 817 ++++++++++++++++------
src/bin/pg_dump/t/004_pg_dump_optsfile.pl | 165 +++++
3 files changed, 812 insertions(+), 198 deletions(-)
create mode 100644 src/bin/pg_dump/t/004_pg_dump_optsfile.pl
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0aa35cf0c3..efdb53f069 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -956,6 +956,34 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--options-file=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read options from file (one option per line). Short or long options
+ are supported. If you use "-" as a filename, the filters are read
+ from stdin.
+ </para>
+
+ <para>
+ With the following options file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
+-t mytable1
+--include-foreign-data=some_foreign_server
+--exclude-table-data=mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The text after symbol <literal>#</literal> is ignored. This can
+ be used for comments, notes. Empty lines are ignored too.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--quote-all-identifiers</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index dc1d41dd8d..96acb8a1d5 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -54,9 +54,11 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -129,6 +131,17 @@ static const CatalogId nilCatalogId = {0, 0};
static bool have_extra_float_digits = false;
static int extra_float_digits;
+static const char *filename = NULL;
+static const char *format = "p";
+static bool g_verbose = false;
+static const char *dumpencoding = NULL;
+static const char *dumpsnapshot = NULL;
+static char *use_role = NULL;
+static long rowsPerInsert;
+static int numWorkers = 1;
+static int compressLevel = -1;
+static char *optsfilename = NULL;
+
/*
* The default number of rows per INSERT when
* --inserts is specified without --rows-per-insert
@@ -294,14 +307,226 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static void read_options_from_file(char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname);
+
+#define OPTIONS_FILE_OPT_NUMBER 12
+
+/*
+ * It assign the values of options to related DumpOption fields or to
+ * some global values. It is called from twice. First, for processing
+ * the command line argumens. Second, for processing an options from
+ * options file.
+ */
+static bool
+process_option(int opt,
+ char *optargstr,
+ DumpOptions *dopt,
+ const char *progname)
+{
+ char *endptr;
+
+ switch (opt)
+ {
+ case 'a': /* Dump data only */
+ dopt->dataOnly = true;
+ break;
+
+ case 'b': /* Dump blobs */
+ dopt->outputBlobs = true;
+ break;
+
+ case 'B': /* Don't dump blobs */
+ dopt->dontOutputBlobs = true;
+ break;
+
+ case 'c': /* clean (i.e., drop) schema prior to create */
+ dopt->outputClean = 1;
+ break;
+
+ case 'C': /* Create DB */
+ dopt->outputCreateDB = 1;
+ break;
+
+ case 'd': /* database name */
+ dopt->cparams.dbname = pg_strdup(optargstr);
+ break;
+
+ case 'E': /* Dump encoding */
+ dumpencoding = pg_strdup(optargstr);
+ break;
+
+ case 'f':
+ filename = pg_strdup(optargstr);
+ break;
+
+ case 'F':
+ format = pg_strdup(optargstr);
+ break;
+
+ case 'h': /* server host */
+ dopt->cparams.pghost = pg_strdup(optargstr);
+ break;
+
+ case 'j': /* number of dump jobs */
+ numWorkers = atoi(optargstr);
+ break;
+
+ case 'n': /* include schema(s) */
+ simple_string_list_append(&schema_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'N': /* exclude schema(s) */
+ simple_string_list_append(&schema_exclude_patterns, optargstr);
+ break;
+
+ case 'O': /* Don't reconnect to match owner */
+ dopt->outputNoOwner = 1;
+ break;
+
+ case 'p': /* server port */
+ dopt->cparams.pgport = pg_strdup(optargstr);
+ break;
+
+ case 'R':
+ /* no-op, still accepted for backwards compatibility */
+ break;
+
+ case 's': /* dump schema only */
+ dopt->schemaOnly = true;
+ break;
+
+ case 'S': /* Username for superuser in plain text output */
+ dopt->outputSuperuser = pg_strdup(optargstr);
+ break;
+
+ case 't': /* include table(s) */
+ simple_string_list_append(&table_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'T': /* exclude table(s) */
+ simple_string_list_append(&table_exclude_patterns, optargstr);
+ break;
+
+ case 'U':
+ dopt->cparams.username = pg_strdup(optargstr);
+ break;
+
+ case 'v': /* verbose */
+ g_verbose = true;
+ pg_logging_increase_verbosity();
+ break;
+
+ case 'w':
+ dopt->cparams.promptPassword = TRI_NO;
+ break;
+
+ case 'W':
+ dopt->cparams.promptPassword = TRI_YES;
+ break;
+
+ case 'x': /* skip ACL dump */
+ dopt->aclsSkip = true;
+ break;
+
+ case 'Z': /* Compression Level */
+ compressLevel = atoi(optargstr);
+ if (compressLevel < 0 || compressLevel > 9)
+ {
+ pg_log_error("compression level must be in range 0..9");
+ return false;
+ }
+ break;
+
+ case 0:
+ /* This covers the long options. */
+ break;
+
+ case 2: /* lock-wait-timeout */
+ dopt->lockWaitTimeout = pg_strdup(optargstr);
+ break;
+
+ case 3: /* SET ROLE */
+ use_role = pg_strdup(optargstr);
+ break;
+
+ case 4: /* exclude table(s) data */
+ simple_string_list_append(&tabledata_exclude_patterns, optargstr);
+ break;
+
+ case 5: /* section */
+ set_dump_section(optargstr, &dopt->dumpSections);
+ break;
+
+ case 6: /* snapshot */
+ dumpsnapshot = pg_strdup(optargstr);
+ break;
+
+ case 7: /* no-sync */
+ dosync = false;
+ break;
+
+ case 8:
+ have_extra_float_digits = true;
+ extra_float_digits = atoi(optargstr);
+ if (extra_float_digits < -15 || extra_float_digits > 3)
+ {
+ pg_log_error("extra_float_digits must be in range -15..3");
+ return false;
+ }
+ break;
+
+ case 9: /* inserts */
+
+ /*
+ * dump_inserts also stores --rows-per-insert, careful not to
+ * overwrite that.
+ */
+ if (dopt->dump_inserts == 0)
+ dopt->dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
+ break;
+ case 10: /* rows per insert */
+ errno = 0;
+ rowsPerInsert = strtol(optargstr, &endptr, 10);
+
+ if (endptr == optargstr || *endptr != '\0' ||
+ rowsPerInsert <= 0 || rowsPerInsert > INT_MAX ||
+ errno == ERANGE)
+ {
+ pg_log_error("rows-per-insert must be in range %d..%d",
+ 1, INT_MAX);
+ return false;
+ }
+ dopt->dump_inserts = (int) rowsPerInsert;
+ break;
+
+ case 11: /* include foreign data */
+ simple_string_list_append(&foreign_servers_include_patterns,
+ optargstr);
+ break;
+
+ case OPTIONS_FILE_OPT_NUMBER: /* filter implementation */
+ optsfilename = pg_strdup(optargstr);
+ break;
+
+ default:
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ return false;
+ }
+
+ return true;
+}
int
main(int argc, char **argv)
{
int c;
- const char *filename = NULL;
- const char *format = "p";
TableInfo *tblinfo;
int numTables;
DumpableObject **dobjs;
@@ -309,20 +534,11 @@ main(int argc, char **argv)
DumpableObject *boundaryObjs;
int i;
int optindex;
- char *endptr;
RestoreOptions *ropt;
Archive *fout; /* the script file */
- bool g_verbose = false;
- const char *dumpencoding = NULL;
- const char *dumpsnapshot = NULL;
- char *use_role = NULL;
- long rowsPerInsert;
- int numWorkers = 1;
- int compressLevel = -1;
int plainText = 0;
ArchiveFormat archiveFormat = archUnknown;
ArchiveMode archiveMode;
-
static DumpOptions dopt;
static struct option long_options[] = {
@@ -387,13 +603,17 @@ main(int argc, char **argv)
{"no-subscriptions", no_argument, &dopt.no_subscriptions, 1},
{"no-sync", no_argument, NULL, 7},
{"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1},
+ {"options-file", required_argument, NULL, OPTIONS_FILE_OPT_NUMBER},
{"rows-per-insert", required_argument, NULL, 10},
{"include-foreign-data", required_argument, NULL, 11},
{"index-collation-versions-unknown", no_argument, &dopt.coll_unknown, 1},
+ {"include-foreign-data-file", required_argument, NULL, 17},
{NULL, 0, NULL, 0}
};
+ const char *short_options = "abBcCd:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:";
+
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_dump"));
@@ -422,197 +642,20 @@ main(int argc, char **argv)
InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "abBcCd:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:",
+ while ((c = getopt_long(argc, argv, short_options,
long_options, &optindex)) != -1)
{
- switch (c)
- {
- case 'a': /* Dump data only */
- dopt.dataOnly = true;
- break;
-
- case 'b': /* Dump blobs */
- dopt.outputBlobs = true;
- break;
-
- case 'B': /* Don't dump blobs */
- dopt.dontOutputBlobs = true;
- break;
-
- case 'c': /* clean (i.e., drop) schema prior to create */
- dopt.outputClean = 1;
- break;
-
- case 'C': /* Create DB */
- dopt.outputCreateDB = 1;
- break;
-
- case 'd': /* database name */
- dopt.cparams.dbname = pg_strdup(optarg);
- break;
-
- case 'E': /* Dump encoding */
- dumpencoding = pg_strdup(optarg);
- break;
-
- case 'f':
- filename = pg_strdup(optarg);
- break;
-
- case 'F':
- format = pg_strdup(optarg);
- break;
-
- case 'h': /* server host */
- dopt.cparams.pghost = pg_strdup(optarg);
- break;
-
- case 'j': /* number of dump jobs */
- numWorkers = atoi(optarg);
- break;
-
- case 'n': /* include schema(s) */
- simple_string_list_append(&schema_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'N': /* exclude schema(s) */
- simple_string_list_append(&schema_exclude_patterns, optarg);
- break;
-
- case 'O': /* Don't reconnect to match owner */
- dopt.outputNoOwner = 1;
- break;
-
- case 'p': /* server port */
- dopt.cparams.pgport = pg_strdup(optarg);
- break;
-
- case 'R':
- /* no-op, still accepted for backwards compatibility */
- break;
-
- case 's': /* dump schema only */
- dopt.schemaOnly = true;
- break;
-
- case 'S': /* Username for superuser in plain text output */
- dopt.outputSuperuser = pg_strdup(optarg);
- break;
-
- case 't': /* include table(s) */
- simple_string_list_append(&table_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'T': /* exclude table(s) */
- simple_string_list_append(&table_exclude_patterns, optarg);
- break;
-
- case 'U':
- dopt.cparams.username = pg_strdup(optarg);
- break;
-
- case 'v': /* verbose */
- g_verbose = true;
- pg_logging_increase_verbosity();
- break;
-
- case 'w':
- dopt.cparams.promptPassword = TRI_NO;
- break;
-
- case 'W':
- dopt.cparams.promptPassword = TRI_YES;
- break;
-
- case 'x': /* skip ACL dump */
- dopt.aclsSkip = true;
- break;
-
- case 'Z': /* Compression Level */
- compressLevel = atoi(optarg);
- if (compressLevel < 0 || compressLevel > 9)
- {
- pg_log_error("compression level must be in range 0..9");
- exit_nicely(1);
- }
- break;
-
- case 0:
- /* This covers the long options. */
- break;
-
- case 2: /* lock-wait-timeout */
- dopt.lockWaitTimeout = pg_strdup(optarg);
- break;
-
- case 3: /* SET ROLE */
- use_role = pg_strdup(optarg);
- break;
-
- case 4: /* exclude table(s) data */
- simple_string_list_append(&tabledata_exclude_patterns, optarg);
- break;
-
- case 5: /* section */
- set_dump_section(optarg, &dopt.dumpSections);
- break;
-
- case 6: /* snapshot */
- dumpsnapshot = pg_strdup(optarg);
- break;
-
- case 7: /* no-sync */
- dosync = false;
- break;
-
- case 8:
- have_extra_float_digits = true;
- extra_float_digits = atoi(optarg);
- if (extra_float_digits < -15 || extra_float_digits > 3)
- {
- pg_log_error("extra_float_digits must be in range -15..3");
- exit_nicely(1);
- }
- break;
-
- case 9: /* inserts */
-
- /*
- * dump_inserts also stores --rows-per-insert, careful not to
- * overwrite that.
- */
- if (dopt.dump_inserts == 0)
- dopt.dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
- break;
-
- case 10: /* rows per insert */
- errno = 0;
- rowsPerInsert = strtol(optarg, &endptr, 10);
-
- if (endptr == optarg || *endptr != '\0' ||
- rowsPerInsert <= 0 || rowsPerInsert > INT_MAX ||
- errno == ERANGE)
- {
- pg_log_error("rows-per-insert must be in range %d..%d",
- 1, INT_MAX);
- exit_nicely(1);
- }
- dopt.dump_inserts = (int) rowsPerInsert;
- break;
-
- case 11: /* include foreign data */
- simple_string_list_append(&foreign_servers_include_patterns,
- optarg);
- break;
-
- default:
- fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
- exit_nicely(1);
- }
+ if (!process_option(c, optarg, &dopt, progname))
+ exit_nicely(1);
}
+ if (optsfilename)
+ read_options_from_file(optsfilename,
+ &dopt,
+ short_options,
+ long_options,
+ progname);
+
/*
* Non-option argument specifies database name as long as it wasn't
* already specified with -d / --dbname
@@ -1049,6 +1092,7 @@ help(const char *progname)
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
printf(_(" --no-unlogged-table-data do not dump unlogged table data\n"));
printf(_(" --on-conflict-do-nothing add ON CONFLICT DO NOTHING to INSERT commands\n"));
+ printf(_(" --options-file=FILENAME read options from options file\n"));
printf(_(" --quote-all-identifiers quote all identifiers, even if not key words\n"));
printf(_(" --rows-per-insert=NROWS number of rows per INSERT; implies --inserts\n"));
printf(_(" --section=SECTION dump named section (pre-data, data, or post-data)\n"));
@@ -18635,3 +18679,380 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_optfile_format(FILE *fp,
+ char *message,
+ char *optname,
+ int optnamelen,
+ char *line,
+ int lineno)
+{
+ Assert(message);
+
+ if (optnamelen > 0)
+ {
+ Assert(optname);
+ pg_log_error(message, optnamelen, optname, lineno);
+ }
+ else
+ pg_log_error(message, lineno);
+
+ if (line)
+ fprintf(stderr, "LINE %d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Reads an option argument from file. Supports double qoutes
+ * bounded strings. In this case multi lines strings are supported.
+ */
+static void
+read_optarg(FILE *fp,
+ char *filename,
+ char *str,
+ StringInfo line,
+ StringInfo optargument,
+ char *optname,
+ int optnamelen,
+ bool islongopt,
+ int *lineno)
+{
+ if (*str == '\0' || *str == '#')
+ {
+ if (islongopt)
+ exit_invalid_optfile_format(fp,
+ "option '--%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ else
+ exit_invalid_optfile_format(fp,
+ "option '-%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ }
+
+ resetStringInfo(optargument);
+
+ /* simple case */
+ if (*str != '"')
+ {
+ char *start = str;
+
+ /* read first white char */
+ while (*str != '\0' && *str != '#')
+ {
+ if (*str == ' ')
+ break;
+ str++;
+ }
+
+ appendBinaryStringInfo(optargument, start, str - start);
+ }
+ else
+ {
+ appendStringInfoChar(optargument, *str++);
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ /* multiline string, read next line */
+ if (!pg_get_line_buf(fp, line))
+ exit_invalid_optfile_format(fp,
+ "unexpected end of line at line %d",
+ NULL, 0,
+ NULL,
+ *lineno);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ appendStringInfoChar(optargument, '\n');
+
+ str = line->data;
+ (void) pg_strip_crlf(str);
+ *lineno += 1;
+ }
+
+ appendStringInfoChar(optargument, *str);
+
+ if (*str++ == '"')
+ {
+ if (*str == '"')
+ str++;
+ else
+ break;
+ }
+ }
+ }
+
+ /* check garbage after optarg, but ignore white spaces */
+ while (isspace(*str))
+ str++;
+
+ /* at the end there should be EOL or comment symbol */
+ if (*str != '\0' && *str != '#')
+ exit_invalid_optfile_format(fp,
+ "unexpected characters after an option's argument at line %d",
+ NULL, 0,
+ line->data,
+ *lineno);
+}
+
+/*
+ * Returns true, when long option is defined. Assign code,
+ * and check if option has required argument.
+ */
+static bool
+is_recognized_longopt(const struct option *longopts,
+ char *optname,
+ int optnamelen,
+ int *opt,
+ bool *has_arg)
+{
+ int i;
+
+ for (i = 0; longopts[i].name != NULL; i++)
+ {
+ if (strlen(longopts[i].name) == optnamelen &&
+ strncmp(optname, longopts[i].name, optnamelen) == 0)
+ {
+ *has_arg = longopts[i].has_arg == required_argument;
+
+ if (longopts[i].flag == NULL)
+ {
+ *opt = longopts[i].val;
+
+ return true;
+ }
+ else
+ {
+ *longopts[i].flag = longopts[i].val;
+ *opt = 0;
+
+ return true;
+ }
+ }
+ }
+
+ return false;
+}
+
+/*
+ * Returns true, when short option char is defined.
+ */
+static bool
+is_valid_shortopt(const char *optstring,
+ char optname,
+ bool *has_arg)
+{
+ while (*optstring != '\0')
+ {
+ if (*optstring != ':' && *optstring == optname)
+ {
+ *has_arg = optstring[1] == ':';
+ return true;
+ }
+
+ optstring++;
+ }
+
+ return false;
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_options_from_file(char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+ StringInfoData optargument;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+ initStringInfo(&optargument);
+
+ while (pg_get_line_buf(fp, &line))
+ {
+ char *optname;
+ char *str = line.data;
+ int opt;
+ int optnamelen;
+ bool has_arg;
+
+ (void) pg_strip_crlf(str);
+
+ lineno += 1;
+
+ /* skip initial spaces */
+ while (isspace(*str))
+ str++;
+
+ /* Ignore empty lines or lines with hash symbol (comment) */
+ if (*str == '\0' || *str == '#')
+ continue;
+
+ if (*str++ != '-')
+ exit_invalid_optfile_format(fp,
+ "non option arguments are not allowed in options file at line %d",
+ NULL, 0,
+ line.data,
+ lineno);
+
+ if (*str == '-')
+ {
+ /* process long option */
+ str++;
+ optname = str++;
+
+ while (!isspace(*str) && *str != '=' && *str != '\0')
+ str++;
+
+ optnamelen = str - optname;
+
+ if (is_recognized_longopt(longopts,
+ optname,
+ optnamelen,
+ &opt,
+ &has_arg))
+ {
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Don't allow --options-file inside options file.
+ * It is simple protection against cycle.
+ */
+ if (opt == OPTIONS_FILE_OPT_NUMBER)
+ exit_invalid_optfile_format(fp,
+ "option '%.*s' cannot be used in options file at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+
+ if (has_arg)
+ {
+ /* skip optional = */
+ if (*str == '=')
+ str++;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ read_optarg(fp,
+ filename,
+ str,
+ &line,
+ &optargument,
+ optname,
+ optnamelen,
+ true,
+ &lineno);
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ exit_invalid_optfile_format(fp,
+ "option '--%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ }
+ }
+ else
+ exit_invalid_optfile_format(fp,
+ "unrecognized option '--%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ }
+ else
+ {
+ /* process short option */
+ optname = str++;
+ optnamelen = 1;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (is_valid_shortopt(optstring, *optname, &has_arg))
+ {
+ if (has_arg)
+ {
+ read_optarg(fp,
+ filename,
+ str,
+ &line,
+ &optargument,
+ optname,
+ optnamelen,
+ false,
+ &lineno);
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ exit_invalid_optfile_format(fp,
+ "option '-%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ }
+ }
+ else
+ exit_invalid_optfile_format(fp,
+ "invalid option '-%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+
+ opt = *optname;
+ }
+
+ if (opt != 0 &&
+ !process_option(opt, optargument.data, dopt, progname))
+ {
+ fclose(fp);
+ exit_nicely(-1);
+ }
+ }
+
+ pfree(line.data);
+ pfree(optargument.data);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_optsfile.pl b/src/bin/pg_dump/t/004_pg_dump_optsfile.pl
new file mode 100644
index 0000000000..0955e28a85
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_optsfile.pl
@@ -0,0 +1,165 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 30;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = get_new_node('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-t table_one #comment\n";
+print $inputfile "-t table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "--exclude-table-data=table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-T table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-N public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "--include-foreign-data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: non option arguments are not allowed in options file at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid option '-' at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-t";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '-t' requires an argument at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-a someforeignserver";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '-a' doesn't allow an argument at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-r";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid option '-r' at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "--doesnt-exists";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: unrecognized option '--doesnt-exists' at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "--data-only badparameter";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '--data-only' doesn't allow an argument at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "--table";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '--table' requires an argument at line 1/,
+ "broken format check");
--
2.17.0
--7CZp05NP8/gJM8Cl
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0002-Fix-short-options-with-double-quotes.patch"
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-28 23:49 Justin Pryzby <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Justin Pryzby @ 2020-11-28 23:49 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
On Sat, Nov 28, 2020 at 09:14:35PM +0100, Pavel Stehule wrote:
> Any short or long option can be read from this file in simple format - one
> option per line. Arguments inside double quotes can be multi lined. Row
> comments started by # and can be used everywhere.
Does this support even funkier table names ?
This tests a large number and fraction of characters in dbname/username, so all
of pg_dump has to continue supporting that:
./src/bin/pg_dump/t/010_dump_connstr.pl
I tested and it seems to work with -t "fooå"
But it didn't work with -t "foo\nbar" (literal newline). Fix attached.
If you send another patch, please consider including a test case for quoted
names in long and short options.
> +static char *optsfilename = NULL;
> + * It assign the values of options to related DumpOption fields or to
> + * some global values. It is called from twice. First, for processing
> + * the command line argumens. Second, for processing an options from
> + * options file.
This didn't support multiple config files, nor config files which include
config files, as Dean and I mentioned. I think the argument parsers should
themselves call the config file parser, as need be, so the last option
specification should override previous ones.
For example pg_dump --config-file=./pg_dump.conf --blobs should have blobs even
if the config file says --no-blobs. (Command-line arguments normally take
precedence over config files, certainly if the argument is specified "later").
I think it'd be ok if it's recursive. I made a quick hack to do that.
I doubt this will satisfy Stephen. Personally, I would use this if it were a
plain and simple text config file (which for our purposes I would pass on
stdin), and I would almost certainly not use it if it were json. But it'd be
swell if there were a standard config file format, that handled postgresql.conf
and maybe pg_hba.conf.
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2020-11-29 14:09 Pavel Stehule <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2020-11-29 14:09 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Tom Lane <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Hi
ne 29. 11. 2020 v 0:49 odesílatel Justin Pryzby <[email protected]>
napsal:
> On Sat, Nov 28, 2020 at 09:14:35PM +0100, Pavel Stehule wrote:
> > Any short or long option can be read from this file in simple format -
> one
> > option per line. Arguments inside double quotes can be multi lined. Row
> > comments started by # and can be used everywhere.
>
here is updated patch
> Does this support even funkier table names ?
>
> This tests a large number and fraction of characters in dbname/username,
> so all
> of pg_dump has to continue supporting that:
> ./src/bin/pg_dump/t/010_dump_connstr.pl
>
> I tested and it seems to work with -t "fooå"
> But it didn't work with -t "foo\nbar" (literal newline). Fix attached.
> If you send another patch, please consider including a test case for quoted
> names in long and short options.
>
I implemented some basic backslash escaping. I will write more tests, when
there will be good agreement on the main concept.
>
> > +static char *optsfilename = NULL;
>
> > + * It assign the values of options to related DumpOption fields or to
> > + * some global values. It is called from twice. First, for processing
> > + * the command line argumens. Second, for processing an options from
> > + * options file.
>
> This didn't support multiple config files, nor config files which include
> config files, as Dean and I mentioned. I think the argument parsers should
> themselves call the config file parser, as need be, so the last option
> specification should override previous ones.
>
> For example pg_dump --config-file=./pg_dump.conf --blobs should have blobs
> even
> if the config file says --no-blobs. (Command-line arguments normally take
> precedence over config files, certainly if the argument is specified
> "later").
> I think it'd be ok if it's recursive. I made a quick hack to do that.
>
I did it. I used a different design than you. Making "dopt" be a global
variable looks too invasive. Almost all functions there expect "dopt" as an
argument. But I think it is not necessary.
I implemented two iterations of argument's processing. 1. for options file
(more options-file options are allowed, and nesting is allowed too), 2. all
other arguments from the command line. Any options file is processed only
once - second processing is ignored. So there is no problem with cycles.
The name of the new option - "config-file" or "options-file" ? I prefer
"options-file". "config-file" is valid too, but "options-file" is more
specific, more descriptive (it is self descriptive).
I merged your patch with a fix of typos.
Regards
Pavel
> I doubt this will satisfy Stephen. Personally, I would use this if it
> were a
> plain and simple text config file (which for our purposes I would pass on
> stdin), and I would almost certainly not use it if it were json. But it'd
> be
> swell if there were a standard config file format, that handled
> postgresql.conf
> and maybe pg_hba.conf.
>
> --
> Justin
>
Attachments:
[text/x-patch] pg_dump-options-file-2.patch (30.2K, ../../CAFj8pRCrbDPAWyHjwZauEX78jyGBowOoiaWySA0ibRYoWzZTMw@mail.gmail.com/3-pg_dump-options-file-2.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 0aa35cf0c3..1446f377a9 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -956,6 +956,42 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--options-file=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read options from file (one option per line). Short or long options
+ are supported. If you use "-" as a filename, the filters are read
+ from stdin.
+ </para>
+
+ <para>
+ With the following options file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
+-t mytable1
+--include-foreign-data=some_foreign_server
+--exclude-table-data=mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The text after symbol <literal>#</literal> is ignored. This can
+ be used for comments, notes. Empty lines are ignored too.
+ </para>
+
+ <para>
+ The option <option>--options-file</option> can be used more times,
+ and the nesting is allowed. The options from options files are
+ processed first, other options from command line later. Any option
+ file is processed only one time. In next time the processing is
+ ignored.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--quote-all-identifiers</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index dc1d41dd8d..26deebd260 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -54,9 +54,11 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -123,18 +125,35 @@ static SimpleOidList tabledata_exclude_oids = {NULL, NULL};
static SimpleStringList foreign_servers_include_patterns = {NULL, NULL};
static SimpleOidList foreign_servers_include_oids = {NULL, NULL};
+static SimpleStringList optsfilenames_processed = {NULL, NULL};
+
static const CatalogId nilCatalogId = {0, 0};
/* override for standard extra_float_digits setting */
static bool have_extra_float_digits = false;
static int extra_float_digits;
+static const char *filename = NULL;
+static const char *format = "p";
+static bool g_verbose = false;
+static const char *dumpencoding = NULL;
+static const char *dumpsnapshot = NULL;
+static char *use_role = NULL;
+static long rowsPerInsert;
+static int numWorkers = 1;
+static int compressLevel = -1;
+
/*
* The default number of rows per INSERT when
* --inserts is specified without --rows-per-insert
*/
#define DUMP_DEFAULT_ROWS_PER_INSERT 1
+/*
+ * Option's code of "options-file" option
+ */
+#define OPTIONS_FILE_OPTNUM 12
+
/*
* Macro for producing quoted, schema-qualified name of a dumpable object.
*/
@@ -294,14 +313,221 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(TableInfo *tbinfo);
+static bool read_options_from_file(char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname);
+
+/*
+ * It assigns the values of options to related DumpOption fields or to
+ * some global values. Options file loading is not processed here.
+ */
+static bool
+process_option(int opt,
+ char *optargstr,
+ DumpOptions *dopt,
+ const char *progname)
+{
+ char *endptr;
+
+ switch (opt)
+ {
+ case 'a': /* Dump data only */
+ dopt->dataOnly = true;
+ break;
+
+ case 'b': /* Dump blobs */
+ dopt->outputBlobs = true;
+ break;
+
+ case 'B': /* Don't dump blobs */
+ dopt->dontOutputBlobs = true;
+ break;
+
+ case 'c': /* clean (i.e., drop) schema prior to create */
+ dopt->outputClean = 1;
+ break;
+
+ case 'C': /* Create DB */
+ dopt->outputCreateDB = 1;
+ break;
+
+ case 'd': /* database name */
+ dopt->cparams.dbname = pg_strdup(optargstr);
+ break;
+
+ case 'E': /* Dump encoding */
+ dumpencoding = pg_strdup(optargstr);
+ break;
+
+ case 'f':
+ filename = pg_strdup(optargstr);
+ break;
+
+ case 'F':
+ format = pg_strdup(optargstr);
+ break;
+
+ case 'h': /* server host */
+ dopt->cparams.pghost = pg_strdup(optargstr);
+ break;
+
+ case 'j': /* number of dump jobs */
+ numWorkers = atoi(optargstr);
+ break;
+
+ case 'n': /* include schema(s) */
+ simple_string_list_append(&schema_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'N': /* exclude schema(s) */
+ simple_string_list_append(&schema_exclude_patterns, optargstr);
+ break;
+
+ case 'O': /* Don't reconnect to match owner */
+ dopt->outputNoOwner = 1;
+ break;
+
+ case 'p': /* server port */
+ dopt->cparams.pgport = pg_strdup(optargstr);
+ break;
+
+ case 'R':
+ /* no-op, still accepted for backwards compatibility */
+ break;
+
+ case 's': /* dump schema only */
+ dopt->schemaOnly = true;
+ break;
+
+ case 'S': /* Username for superuser in plain text output */
+ dopt->outputSuperuser = pg_strdup(optargstr);
+ break;
+
+ case 't': /* include table(s) */
+ simple_string_list_append(&table_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'T': /* exclude table(s) */
+ simple_string_list_append(&table_exclude_patterns, optargstr);
+ break;
+
+ case 'U':
+ dopt->cparams.username = pg_strdup(optargstr);
+ break;
+
+ case 'v': /* verbose */
+ g_verbose = true;
+ pg_logging_increase_verbosity();
+ break;
+
+ case 'w':
+ dopt->cparams.promptPassword = TRI_NO;
+ break;
+
+ case 'W':
+ dopt->cparams.promptPassword = TRI_YES;
+ break;
+
+ case 'x': /* skip ACL dump */
+ dopt->aclsSkip = true;
+ break;
+
+ case 'Z': /* Compression Level */
+ compressLevel = atoi(optargstr);
+ if (compressLevel < 0 || compressLevel > 9)
+ {
+ pg_log_error("compression level must be in range 0..9");
+ return false;
+ }
+ break;
+
+ case 0:
+ /* This covers the long options. */
+ break;
+
+ case 2: /* lock-wait-timeout */
+ dopt->lockWaitTimeout = pg_strdup(optargstr);
+ break;
+
+ case 3: /* SET ROLE */
+ use_role = pg_strdup(optargstr);
+ break;
+
+ case 4: /* exclude table(s) data */
+ simple_string_list_append(&tabledata_exclude_patterns, optargstr);
+ break;
+
+ case 5: /* section */
+ set_dump_section(optargstr, &dopt->dumpSections);
+ break;
+
+ case 6: /* snapshot */
+ dumpsnapshot = pg_strdup(optargstr);
+ break;
+
+ case 7: /* no-sync */
+ dosync = false;
+ break;
+
+ case 8:
+ have_extra_float_digits = true;
+ extra_float_digits = atoi(optargstr);
+ if (extra_float_digits < -15 || extra_float_digits > 3)
+ {
+ pg_log_error("extra_float_digits must be in range -15..3");
+ return false;
+ }
+ break;
+
+ case 9: /* inserts */
+
+ /*
+ * dump_inserts also stores --rows-per-insert, careful not to
+ * overwrite that.
+ */
+ if (dopt->dump_inserts == 0)
+ dopt->dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
+ break;
+
+ case 10: /* rows per insert */
+ errno = 0;
+ rowsPerInsert = strtol(optargstr, &endptr, 10);
+
+ if (endptr == optargstr || *endptr != '\0' ||
+ rowsPerInsert <= 0 || rowsPerInsert > INT_MAX ||
+ errno == ERANGE)
+ {
+ pg_log_error("rows-per-insert must be in range %d..%d",
+ 1, INT_MAX);
+ return false;
+ }
+ dopt->dump_inserts = (int) rowsPerInsert;
+ break;
+
+ case 11: /* include foreign data */
+ simple_string_list_append(&foreign_servers_include_patterns,
+ optargstr);
+ break;
+
+ case OPTIONS_FILE_OPTNUM: /* reading options file */
+ break; /* should not be processed here ever */
+
+ default:
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ return false;
+ }
+ return true;
+}
int
main(int argc, char **argv)
{
int c;
- const char *filename = NULL;
- const char *format = "p";
TableInfo *tblinfo;
int numTables;
DumpableObject **dobjs;
@@ -309,20 +535,11 @@ main(int argc, char **argv)
DumpableObject *boundaryObjs;
int i;
int optindex;
- char *endptr;
RestoreOptions *ropt;
Archive *fout; /* the script file */
- bool g_verbose = false;
- const char *dumpencoding = NULL;
- const char *dumpsnapshot = NULL;
- char *use_role = NULL;
- long rowsPerInsert;
- int numWorkers = 1;
- int compressLevel = -1;
int plainText = 0;
ArchiveFormat archiveFormat = archUnknown;
ArchiveMode archiveMode;
-
static DumpOptions dopt;
static struct option long_options[] = {
@@ -387,13 +604,17 @@ main(int argc, char **argv)
{"no-subscriptions", no_argument, &dopt.no_subscriptions, 1},
{"no-sync", no_argument, NULL, 7},
{"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1},
+ {"options-file", required_argument, NULL, OPTIONS_FILE_OPTNUM},
{"rows-per-insert", required_argument, NULL, 10},
{"include-foreign-data", required_argument, NULL, 11},
{"index-collation-versions-unknown", no_argument, &dopt.coll_unknown, 1},
+ {"include-foreign-data-file", required_argument, NULL, 17},
{NULL, 0, NULL, 0}
};
+ const char *short_options = "abBcCd:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:";
+
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_dump"));
@@ -422,197 +643,33 @@ main(int argc, char **argv)
InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "abBcCd:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:",
+ /*
+ * We would to process options-file opts before other options.
+ * This implements higher priority of options from command line.
+ * Later processed options wins because overwrite result of
+ * early processed options.
+ */
+ while ((c = getopt_long(argc, argv, short_options,
long_options, &optindex)) != -1)
{
- switch (c)
+ if (c == OPTIONS_FILE_OPTNUM)
{
- case 'a': /* Dump data only */
- dopt.dataOnly = true;
- break;
-
- case 'b': /* Dump blobs */
- dopt.outputBlobs = true;
- break;
-
- case 'B': /* Don't dump blobs */
- dopt.dontOutputBlobs = true;
- break;
-
- case 'c': /* clean (i.e., drop) schema prior to create */
- dopt.outputClean = 1;
- break;
-
- case 'C': /* Create DB */
- dopt.outputCreateDB = 1;
- break;
-
- case 'd': /* database name */
- dopt.cparams.dbname = pg_strdup(optarg);
- break;
-
- case 'E': /* Dump encoding */
- dumpencoding = pg_strdup(optarg);
- break;
-
- case 'f':
- filename = pg_strdup(optarg);
- break;
-
- case 'F':
- format = pg_strdup(optarg);
- break;
-
- case 'h': /* server host */
- dopt.cparams.pghost = pg_strdup(optarg);
- break;
-
- case 'j': /* number of dump jobs */
- numWorkers = atoi(optarg);
- break;
-
- case 'n': /* include schema(s) */
- simple_string_list_append(&schema_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'N': /* exclude schema(s) */
- simple_string_list_append(&schema_exclude_patterns, optarg);
- break;
-
- case 'O': /* Don't reconnect to match owner */
- dopt.outputNoOwner = 1;
- break;
-
- case 'p': /* server port */
- dopt.cparams.pgport = pg_strdup(optarg);
- break;
-
- case 'R':
- /* no-op, still accepted for backwards compatibility */
- break;
-
- case 's': /* dump schema only */
- dopt.schemaOnly = true;
- break;
-
- case 'S': /* Username for superuser in plain text output */
- dopt.outputSuperuser = pg_strdup(optarg);
- break;
-
- case 't': /* include table(s) */
- simple_string_list_append(&table_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'T': /* exclude table(s) */
- simple_string_list_append(&table_exclude_patterns, optarg);
- break;
-
- case 'U':
- dopt.cparams.username = pg_strdup(optarg);
- break;
-
- case 'v': /* verbose */
- g_verbose = true;
- pg_logging_increase_verbosity();
- break;
-
- case 'w':
- dopt.cparams.promptPassword = TRI_NO;
- break;
-
- case 'W':
- dopt.cparams.promptPassword = TRI_YES;
- break;
-
- case 'x': /* skip ACL dump */
- dopt.aclsSkip = true;
- break;
-
- case 'Z': /* Compression Level */
- compressLevel = atoi(optarg);
- if (compressLevel < 0 || compressLevel > 9)
- {
- pg_log_error("compression level must be in range 0..9");
- exit_nicely(1);
- }
- break;
-
- case 0:
- /* This covers the long options. */
- break;
-
- case 2: /* lock-wait-timeout */
- dopt.lockWaitTimeout = pg_strdup(optarg);
- break;
-
- case 3: /* SET ROLE */
- use_role = pg_strdup(optarg);
- break;
-
- case 4: /* exclude table(s) data */
- simple_string_list_append(&tabledata_exclude_patterns, optarg);
- break;
-
- case 5: /* section */
- set_dump_section(optarg, &dopt.dumpSections);
- break;
-
- case 6: /* snapshot */
- dumpsnapshot = pg_strdup(optarg);
- break;
-
- case 7: /* no-sync */
- dosync = false;
- break;
-
- case 8:
- have_extra_float_digits = true;
- extra_float_digits = atoi(optarg);
- if (extra_float_digits < -15 || extra_float_digits > 3)
- {
- pg_log_error("extra_float_digits must be in range -15..3");
- exit_nicely(1);
- }
- break;
-
- case 9: /* inserts */
-
- /*
- * dump_inserts also stores --rows-per-insert, careful not to
- * overwrite that.
- */
- if (dopt.dump_inserts == 0)
- dopt.dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
- break;
-
- case 10: /* rows per insert */
- errno = 0;
- rowsPerInsert = strtol(optarg, &endptr, 10);
-
- if (endptr == optarg || *endptr != '\0' ||
- rowsPerInsert <= 0 || rowsPerInsert > INT_MAX ||
- errno == ERANGE)
- {
- pg_log_error("rows-per-insert must be in range %d..%d",
- 1, INT_MAX);
- exit_nicely(1);
- }
- dopt.dump_inserts = (int) rowsPerInsert;
- break;
-
- case 11: /* include foreign data */
- simple_string_list_append(&foreign_servers_include_patterns,
- optarg);
- break;
-
- default:
- fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ if (!read_options_from_file(optarg, &dopt, short_options,
+ long_options, progname))
exit_nicely(1);
}
}
+ /* reset getopt_long index */
+ optind = 1;
+
+ while ((c = getopt_long(argc, argv, short_options,
+ long_options, &optindex)) != -1)
+ {
+ if (!process_option(c, optarg, &dopt, progname))
+ exit_nicely(1);
+ }
+
/*
* Non-option argument specifies database name as long as it wasn't
* already specified with -d / --dbname
@@ -1049,6 +1106,7 @@ help(const char *progname)
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
printf(_(" --no-unlogged-table-data do not dump unlogged table data\n"));
printf(_(" --on-conflict-do-nothing add ON CONFLICT DO NOTHING to INSERT commands\n"));
+ printf(_(" --options-file=FILENAME read options from options file\n"));
printf(_(" --quote-all-identifiers quote all identifiers, even if not key words\n"));
printf(_(" --rows-per-insert=NROWS number of rows per INSERT; implies --inserts\n"));
printf(_(" --section=SECTION dump named section (pre-data, data, or post-data)\n"));
@@ -18635,3 +18693,450 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and close input file
+ */
+static void
+invalid_optfile_format(FILE *fp,
+ char *message,
+ char *optname,
+ int optnamelen,
+ char *line,
+ int lineno)
+{
+ Assert(message);
+
+ if (optnamelen > 0)
+ {
+ Assert(optname);
+ pg_log_error(message, optnamelen, optname, lineno);
+ }
+ else
+ pg_log_error(message, lineno);
+
+ if (line)
+ fprintf(stderr, "LINE %d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+}
+
+/*
+ * Reads an option argument from file. Supports double-quoted
+ * strings. In this case multi-line strings are supported.
+ */
+static bool
+read_optarg(FILE *fp,
+ char *filename,
+ char *str,
+ StringInfo line,
+ StringInfo optargument,
+ char *optname,
+ int optnamelen,
+ bool islongopt,
+ int *lineno)
+{
+ if (*str == '\0' || *str == '#')
+ {
+ if (islongopt)
+ invalid_optfile_format(fp,
+ "option '--%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ else
+ invalid_optfile_format(fp,
+ "option '-%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ return false;
+ }
+
+ resetStringInfo(optargument);
+
+ /* simple case */
+ if (*str != '"')
+ {
+ char *start = str;
+
+ /* read first white char */
+ while (*str != '\0' && *str != '#')
+ {
+ if (*str == ' ')
+ break;
+ str++;
+ }
+
+ appendBinaryStringInfo(optargument, start, str - start);
+ }
+ else
+ {
+ appendStringInfoChar(optargument, *str++);
+
+ while (1)
+ {
+ int c = *str++;
+
+ if (c == '\0')
+ {
+ /* multiline string, read next line */
+ if (!pg_get_line_buf(fp, line))
+ {
+ invalid_optfile_format(fp,
+ "unexpected end of line at line %d",
+ NULL, 0,
+ NULL,
+ *lineno);
+ return false;
+ }
+
+ if (ferror(fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m",
+ filename);
+ return false;
+ }
+
+ appendStringInfoChar(optargument, '\n');
+
+ str = line->data;
+ (void) pg_strip_crlf(str);
+ *lineno += 1;
+
+ c = *str++;
+ }
+
+ if (c == '\\')
+ {
+ c = *str++;
+
+ if (c == '\0')
+ {
+ invalid_optfile_format(fp,
+ "unexpected end of line at line %d",
+ NULL, 0,
+ NULL,
+ *lineno);
+ return false;
+ }
+
+ switch (c)
+ {
+ case 'n':
+ appendStringInfoChar(optargument, '\n');
+ break;
+
+ case 't':
+ appendStringInfoChar(optargument, '\t');
+ break;
+
+ case '\\':
+ appendStringInfoChar(optargument, '\\');
+ break;
+
+ default:
+ appendStringInfoChar(optargument, '\\');
+ appendStringInfoChar(optargument, c);
+ }
+ }
+ else
+ {
+ appendStringInfoChar(optargument, c);
+
+ if (c == '"')
+ {
+ if (*str == '"')
+ str++;
+ else
+ break;
+ }
+ }
+ }
+ }
+
+ /* check garbage after optarg, but ignore white-space */
+ while (isspace(*str))
+ str++;
+
+ /* at the end there should be EOL or comment symbol */
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "unexpected characters after an option's argument at line %d",
+ NULL, 0,
+ line->data,
+ *lineno);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Returns true, when long option is defined. Assign code,
+ * and check if option has required argument.
+ */
+static bool
+is_recognized_longopt(const struct option *longopts,
+ char *optname,
+ int optnamelen,
+ int *opt,
+ bool *has_arg)
+{
+ int i;
+
+ for (i = 0; longopts[i].name != NULL; i++)
+ {
+ if (strlen(longopts[i].name) == optnamelen &&
+ strncmp(optname, longopts[i].name, optnamelen) == 0)
+ {
+ *has_arg = longopts[i].has_arg == required_argument;
+
+ if (longopts[i].flag == NULL)
+ {
+ *opt = longopts[i].val;
+
+ return true;
+ }
+ else
+ {
+ *longopts[i].flag = longopts[i].val;
+ *opt = 0;
+
+ return true;
+ }
+ }
+ }
+
+ return false;
+}
+
+/*
+ * Returns true, when short option char is defined.
+ */
+static bool
+is_valid_shortopt(const char *optstring,
+ char optname,
+ bool *has_arg)
+{
+ while (*optstring != '\0')
+ {
+ if (*optstring != ':' && *optstring == optname)
+ {
+ *has_arg = optstring[1] == ':';
+ return true;
+ }
+
+ optstring++;
+ }
+
+ return false;
+}
+
+/*
+ * Read dumped object specification from file. Returns
+ * true, when processing was ok.
+ */
+static bool
+read_options_from_file(char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+ StringInfoData optargument;
+
+ /* Ignore already processed files */
+ if (simple_string_list_member(&optsfilenames_processed,
+ filename))
+ {
+ pg_log_warning("the options file \"%s\" was processed already, skip this",
+ filename);
+ return true;
+ }
+
+ simple_string_list_append(&optsfilenames_processed, filename);
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ {
+ pg_log_error("could not open the input file \"%s\": %m",
+ filename);
+ return false;
+ }
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+ initStringInfo(&optargument);
+
+ while (pg_get_line_buf(fp, &line))
+ {
+ char *optname;
+ char *str = line.data;
+ int opt;
+ int optnamelen;
+ bool has_arg;
+
+ (void) pg_strip_crlf(str);
+
+ lineno += 1;
+
+ /* skip initial spaces */
+ while (isspace(*str))
+ str++;
+
+ /* Ignore empty lines or lines with hash symbol (comment) */
+ if (*str == '\0' || *str == '#')
+ continue;
+
+ if (*str++ != '-')
+ {
+ invalid_optfile_format(fp,
+ "non option arguments are not allowed in options file at line %d",
+ NULL, 0,
+ line.data,
+ lineno);
+ return false;
+ }
+
+ if (*str == '-')
+ {
+ /* process long option */
+ str++;
+ optname = str++;
+
+ while (!isspace(*str) && *str != '=' && *str != '\0')
+ str++;
+
+ optnamelen = str - optname;
+
+ if (is_recognized_longopt(longopts, optname, optnamelen,
+ &opt, &has_arg))
+ {
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (has_arg)
+ {
+ /* skip optional = */
+ if (*str == '=')
+ str++;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (!read_optarg(fp, filename, str, &line, &optargument,
+ optname, optnamelen, true, &lineno))
+ return false;
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "option '--%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ }
+ else
+ {
+ invalid_optfile_format(fp,
+ "unrecognized option '--%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ else
+ {
+ /* process short option */
+ optname = str++;
+ opt = *optname;
+
+ optnamelen = 1;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (is_valid_shortopt(optstring, opt, &has_arg))
+ {
+ if (has_arg)
+ {
+ if (!read_optarg(fp, filename, str, &line, &optargument,
+ optname, optnamelen, false, &lineno))
+ return false;
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "option '-%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ }
+ else
+ {
+ invalid_optfile_format(fp,
+ "invalid option '-%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+
+ /* nested options file reading */
+ if (opt == OPTIONS_FILE_OPTNUM)
+ {
+ if (!read_options_from_file(optargument.data, dopt, optstring,
+ longopts, progname))
+ {
+ fclose(fp);
+ return false;
+ }
+ }
+ else if (opt != 0 &&
+ !process_option(opt, optargument.data, dopt, progname))
+ {
+ fclose(fp);
+ return false;
+ }
+ }
+
+ pfree(line.data);
+ pfree(optargument.data);
+
+ if (ferror(fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m", filename);
+ return false;
+ }
+
+ if (fp != stdin)
+ fclose(fp);
+
+ return true;
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_optsfile.pl b/src/bin/pg_dump/t/004_pg_dump_optsfile.pl
new file mode 100644
index 0000000000..0955e28a85
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_optsfile.pl
@@ -0,0 +1,165 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 30;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = get_new_node('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-t table_one #comment\n";
+print $inputfile "-t table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "--exclude-table-data=table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-T table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-N public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "--include-foreign-data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: non option arguments are not allowed in options file at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid option '-' at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-t";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '-t' requires an argument at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-a someforeignserver";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '-a' doesn't allow an argument at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-r";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid option '-r' at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "--doesnt-exists";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: unrecognized option '--doesnt-exists' at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "--data-only badparameter";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '--data-only' doesn't allow an argument at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "--table";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '--table' requires an argument at line 1/,
+ "broken format check");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-02-16 19:32 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2021-02-16 19:32 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Tom Lane <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Hi
rebase
Regards
Pavel
Attachments:
[text/x-patch] pg_dump-options-file-3.patch (30.2K, ../../CAFj8pRDcEjL-pEWmZj-3p5W6Xg29xbd6D2hdyPxZ5Znw0OXqJA@mail.gmail.com/3-pg_dump-options-file-3.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index bcbb7a25fb..f24b3b5262 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -956,6 +956,42 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--options-file=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read options from file (one option per line). Short or long options
+ are supported. If you use "-" as a filename, the filters are read
+ from stdin.
+ </para>
+
+ <para>
+ With the following options file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
+-t mytable1
+--include-foreign-data=some_foreign_server
+--exclude-table-data=mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The text after symbol <literal>#</literal> is ignored. This can
+ be used for comments, notes. Empty lines are ignored too.
+ </para>
+
+ <para>
+ The option <option>--options-file</option> can be used more times,
+ and the nesting is allowed. The options from options files are
+ processed first, other options from command line later. Any option
+ file is processed only one time. In next time the processing is
+ ignored.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--quote-all-identifiers</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index eb988d7eb4..17fef1fedf 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -54,9 +54,11 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -123,18 +125,35 @@ static SimpleOidList tabledata_exclude_oids = {NULL, NULL};
static SimpleStringList foreign_servers_include_patterns = {NULL, NULL};
static SimpleOidList foreign_servers_include_oids = {NULL, NULL};
+static SimpleStringList optsfilenames_processed = {NULL, NULL};
+
static const CatalogId nilCatalogId = {0, 0};
/* override for standard extra_float_digits setting */
static bool have_extra_float_digits = false;
static int extra_float_digits;
+static const char *filename = NULL;
+static const char *format = "p";
+static bool g_verbose = false;
+static const char *dumpencoding = NULL;
+static const char *dumpsnapshot = NULL;
+static char *use_role = NULL;
+static long rowsPerInsert;
+static int numWorkers = 1;
+static int compressLevel = -1;
+
/*
* The default number of rows per INSERT when
* --inserts is specified without --rows-per-insert
*/
#define DUMP_DEFAULT_ROWS_PER_INSERT 1
+/*
+ * Option's code of "options-file" option
+ */
+#define OPTIONS_FILE_OPTNUM 12
+
/*
* Macro for producing quoted, schema-qualified name of a dumpable object.
*/
@@ -296,14 +315,221 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static bool read_options_from_file(const char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname);
+
+/*
+ * It assigns the values of options to related DumpOption fields or to
+ * some global values. Options file loading is not processed here.
+ */
+static bool
+process_option(int opt,
+ char *optargstr,
+ DumpOptions *dopt,
+ const char *progname)
+{
+ char *endptr;
+
+ switch (opt)
+ {
+ case 'a': /* Dump data only */
+ dopt->dataOnly = true;
+ break;
+
+ case 'b': /* Dump blobs */
+ dopt->outputBlobs = true;
+ break;
+
+ case 'B': /* Don't dump blobs */
+ dopt->dontOutputBlobs = true;
+ break;
+
+ case 'c': /* clean (i.e., drop) schema prior to create */
+ dopt->outputClean = 1;
+ break;
+
+ case 'C': /* Create DB */
+ dopt->outputCreateDB = 1;
+ break;
+
+ case 'd': /* database name */
+ dopt->cparams.dbname = pg_strdup(optargstr);
+ break;
+
+ case 'E': /* Dump encoding */
+ dumpencoding = pg_strdup(optargstr);
+ break;
+
+ case 'f':
+ filename = pg_strdup(optargstr);
+ break;
+
+ case 'F':
+ format = pg_strdup(optargstr);
+ break;
+
+ case 'h': /* server host */
+ dopt->cparams.pghost = pg_strdup(optargstr);
+ break;
+
+ case 'j': /* number of dump jobs */
+ numWorkers = atoi(optargstr);
+ break;
+
+ case 'n': /* include schema(s) */
+ simple_string_list_append(&schema_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'N': /* exclude schema(s) */
+ simple_string_list_append(&schema_exclude_patterns, optargstr);
+ break;
+
+ case 'O': /* Don't reconnect to match owner */
+ dopt->outputNoOwner = 1;
+ break;
+
+ case 'p': /* server port */
+ dopt->cparams.pgport = pg_strdup(optargstr);
+ break;
+
+ case 'R':
+ /* no-op, still accepted for backwards compatibility */
+ break;
+
+ case 's': /* dump schema only */
+ dopt->schemaOnly = true;
+ break;
+
+ case 'S': /* Username for superuser in plain text output */
+ dopt->outputSuperuser = pg_strdup(optargstr);
+ break;
+
+ case 't': /* include table(s) */
+ simple_string_list_append(&table_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'T': /* exclude table(s) */
+ simple_string_list_append(&table_exclude_patterns, optargstr);
+ break;
+
+ case 'U':
+ dopt->cparams.username = pg_strdup(optargstr);
+ break;
+
+ case 'v': /* verbose */
+ g_verbose = true;
+ pg_logging_increase_verbosity();
+ break;
+
+ case 'w':
+ dopt->cparams.promptPassword = TRI_NO;
+ break;
+
+ case 'W':
+ dopt->cparams.promptPassword = TRI_YES;
+ break;
+
+ case 'x': /* skip ACL dump */
+ dopt->aclsSkip = true;
+ break;
+
+ case 'Z': /* Compression Level */
+ compressLevel = atoi(optargstr);
+ if (compressLevel < 0 || compressLevel > 9)
+ {
+ pg_log_error("compression level must be in range 0..9");
+ return false;
+ }
+ break;
+
+ case 0:
+ /* This covers the long options. */
+ break;
+
+ case 2: /* lock-wait-timeout */
+ dopt->lockWaitTimeout = pg_strdup(optargstr);
+ break;
+
+ case 3: /* SET ROLE */
+ use_role = pg_strdup(optargstr);
+ break;
+
+ case 4: /* exclude table(s) data */
+ simple_string_list_append(&tabledata_exclude_patterns, optargstr);
+ break;
+
+ case 5: /* section */
+ set_dump_section(optargstr, &dopt->dumpSections);
+ break;
+
+ case 6: /* snapshot */
+ dumpsnapshot = pg_strdup(optargstr);
+ break;
+
+ case 7: /* no-sync */
+ dosync = false;
+ break;
+
+ case 8:
+ have_extra_float_digits = true;
+ extra_float_digits = atoi(optargstr);
+ if (extra_float_digits < -15 || extra_float_digits > 3)
+ {
+ pg_log_error("extra_float_digits must be in range -15..3");
+ return false;
+ }
+ break;
+
+ case 9: /* inserts */
+
+ /*
+ * dump_inserts also stores --rows-per-insert, careful not to
+ * overwrite that.
+ */
+ if (dopt->dump_inserts == 0)
+ dopt->dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
+ break;
+
+ case 10: /* rows per insert */
+ errno = 0;
+ rowsPerInsert = strtol(optargstr, &endptr, 10);
+
+ if (endptr == optargstr || *endptr != '\0' ||
+ rowsPerInsert <= 0 || rowsPerInsert > INT_MAX ||
+ errno == ERANGE)
+ {
+ pg_log_error("rows-per-insert must be in range %d..%d",
+ 1, INT_MAX);
+ return false;
+ }
+ dopt->dump_inserts = (int) rowsPerInsert;
+ break;
+
+ case 11: /* include foreign data */
+ simple_string_list_append(&foreign_servers_include_patterns,
+ optargstr);
+ break;
+
+ case OPTIONS_FILE_OPTNUM: /* reading options file */
+ break; /* should not be processed here ever */
+
+ default:
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ return false;
+ }
+ return true;
+}
int
main(int argc, char **argv)
{
int c;
- const char *filename = NULL;
- const char *format = "p";
TableInfo *tblinfo;
int numTables;
DumpableObject **dobjs;
@@ -311,20 +537,11 @@ main(int argc, char **argv)
DumpableObject *boundaryObjs;
int i;
int optindex;
- char *endptr;
RestoreOptions *ropt;
Archive *fout; /* the script file */
- bool g_verbose = false;
- const char *dumpencoding = NULL;
- const char *dumpsnapshot = NULL;
- char *use_role = NULL;
- long rowsPerInsert;
- int numWorkers = 1;
- int compressLevel = -1;
int plainText = 0;
ArchiveFormat archiveFormat = archUnknown;
ArchiveMode archiveMode;
-
static DumpOptions dopt;
static struct option long_options[] = {
@@ -389,13 +606,17 @@ main(int argc, char **argv)
{"no-subscriptions", no_argument, &dopt.no_subscriptions, 1},
{"no-sync", no_argument, NULL, 7},
{"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1},
+ {"options-file", required_argument, NULL, OPTIONS_FILE_OPTNUM},
{"rows-per-insert", required_argument, NULL, 10},
{"include-foreign-data", required_argument, NULL, 11},
{"index-collation-versions-unknown", no_argument, &dopt.coll_unknown, 1},
+ {"include-foreign-data-file", required_argument, NULL, 17},
{NULL, 0, NULL, 0}
};
+ const char *short_options = "abBcCd:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:";
+
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_dump"));
@@ -424,197 +645,33 @@ main(int argc, char **argv)
InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "abBcCd:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:",
+ /*
+ * We would to process options-file opts before other options.
+ * This implements higher priority of options from command line.
+ * Later processed options wins because overwrite result of
+ * early processed options.
+ */
+ while ((c = getopt_long(argc, argv, short_options,
long_options, &optindex)) != -1)
{
- switch (c)
+ if (c == OPTIONS_FILE_OPTNUM)
{
- case 'a': /* Dump data only */
- dopt.dataOnly = true;
- break;
-
- case 'b': /* Dump blobs */
- dopt.outputBlobs = true;
- break;
-
- case 'B': /* Don't dump blobs */
- dopt.dontOutputBlobs = true;
- break;
-
- case 'c': /* clean (i.e., drop) schema prior to create */
- dopt.outputClean = 1;
- break;
-
- case 'C': /* Create DB */
- dopt.outputCreateDB = 1;
- break;
-
- case 'd': /* database name */
- dopt.cparams.dbname = pg_strdup(optarg);
- break;
-
- case 'E': /* Dump encoding */
- dumpencoding = pg_strdup(optarg);
- break;
-
- case 'f':
- filename = pg_strdup(optarg);
- break;
-
- case 'F':
- format = pg_strdup(optarg);
- break;
-
- case 'h': /* server host */
- dopt.cparams.pghost = pg_strdup(optarg);
- break;
-
- case 'j': /* number of dump jobs */
- numWorkers = atoi(optarg);
- break;
-
- case 'n': /* include schema(s) */
- simple_string_list_append(&schema_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'N': /* exclude schema(s) */
- simple_string_list_append(&schema_exclude_patterns, optarg);
- break;
-
- case 'O': /* Don't reconnect to match owner */
- dopt.outputNoOwner = 1;
- break;
-
- case 'p': /* server port */
- dopt.cparams.pgport = pg_strdup(optarg);
- break;
-
- case 'R':
- /* no-op, still accepted for backwards compatibility */
- break;
-
- case 's': /* dump schema only */
- dopt.schemaOnly = true;
- break;
-
- case 'S': /* Username for superuser in plain text output */
- dopt.outputSuperuser = pg_strdup(optarg);
- break;
-
- case 't': /* include table(s) */
- simple_string_list_append(&table_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'T': /* exclude table(s) */
- simple_string_list_append(&table_exclude_patterns, optarg);
- break;
-
- case 'U':
- dopt.cparams.username = pg_strdup(optarg);
- break;
-
- case 'v': /* verbose */
- g_verbose = true;
- pg_logging_increase_verbosity();
- break;
-
- case 'w':
- dopt.cparams.promptPassword = TRI_NO;
- break;
-
- case 'W':
- dopt.cparams.promptPassword = TRI_YES;
- break;
-
- case 'x': /* skip ACL dump */
- dopt.aclsSkip = true;
- break;
-
- case 'Z': /* Compression Level */
- compressLevel = atoi(optarg);
- if (compressLevel < 0 || compressLevel > 9)
- {
- pg_log_error("compression level must be in range 0..9");
- exit_nicely(1);
- }
- break;
-
- case 0:
- /* This covers the long options. */
- break;
-
- case 2: /* lock-wait-timeout */
- dopt.lockWaitTimeout = pg_strdup(optarg);
- break;
-
- case 3: /* SET ROLE */
- use_role = pg_strdup(optarg);
- break;
-
- case 4: /* exclude table(s) data */
- simple_string_list_append(&tabledata_exclude_patterns, optarg);
- break;
-
- case 5: /* section */
- set_dump_section(optarg, &dopt.dumpSections);
- break;
-
- case 6: /* snapshot */
- dumpsnapshot = pg_strdup(optarg);
- break;
-
- case 7: /* no-sync */
- dosync = false;
- break;
-
- case 8:
- have_extra_float_digits = true;
- extra_float_digits = atoi(optarg);
- if (extra_float_digits < -15 || extra_float_digits > 3)
- {
- pg_log_error("extra_float_digits must be in range -15..3");
- exit_nicely(1);
- }
- break;
-
- case 9: /* inserts */
-
- /*
- * dump_inserts also stores --rows-per-insert, careful not to
- * overwrite that.
- */
- if (dopt.dump_inserts == 0)
- dopt.dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
- break;
-
- case 10: /* rows per insert */
- errno = 0;
- rowsPerInsert = strtol(optarg, &endptr, 10);
-
- if (endptr == optarg || *endptr != '\0' ||
- rowsPerInsert <= 0 || rowsPerInsert > INT_MAX ||
- errno == ERANGE)
- {
- pg_log_error("rows-per-insert must be in range %d..%d",
- 1, INT_MAX);
- exit_nicely(1);
- }
- dopt.dump_inserts = (int) rowsPerInsert;
- break;
-
- case 11: /* include foreign data */
- simple_string_list_append(&foreign_servers_include_patterns,
- optarg);
- break;
-
- default:
- fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ if (!read_options_from_file(optarg, &dopt, short_options,
+ long_options, progname))
exit_nicely(1);
}
}
+ /* reset getopt_long index */
+ optind = 1;
+
+ while ((c = getopt_long(argc, argv, short_options,
+ long_options, &optindex)) != -1)
+ {
+ if (!process_option(c, optarg, &dopt, progname))
+ exit_nicely(1);
+ }
+
/*
* Non-option argument specifies database name as long as it wasn't
* already specified with -d / --dbname
@@ -1051,6 +1108,7 @@ help(const char *progname)
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
printf(_(" --no-unlogged-table-data do not dump unlogged table data\n"));
printf(_(" --on-conflict-do-nothing add ON CONFLICT DO NOTHING to INSERT commands\n"));
+ printf(_(" --options-file=FILENAME read options from options file\n"));
printf(_(" --quote-all-identifiers quote all identifiers, even if not key words\n"));
printf(_(" --rows-per-insert=NROWS number of rows per INSERT; implies --inserts\n"));
printf(_(" --section=SECTION dump named section (pre-data, data, or post-data)\n"));
@@ -18779,3 +18837,450 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and close input file
+ */
+static void
+invalid_optfile_format(FILE *fp,
+ char *message,
+ char *optname,
+ int optnamelen,
+ char *line,
+ int lineno)
+{
+ Assert(message);
+
+ if (optnamelen > 0)
+ {
+ Assert(optname);
+ pg_log_error(message, optnamelen, optname, lineno);
+ }
+ else
+ pg_log_error(message, lineno);
+
+ if (line)
+ fprintf(stderr, "LINE %d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+}
+
+/*
+ * Reads an option argument from file. Supports double-quoted
+ * strings. In this case multi-line strings are supported.
+ */
+static bool
+read_optarg(FILE *fp,
+ const char *filename,
+ char *str,
+ StringInfo line,
+ StringInfo optargument,
+ char *optname,
+ int optnamelen,
+ bool islongopt,
+ int *lineno)
+{
+ if (*str == '\0' || *str == '#')
+ {
+ if (islongopt)
+ invalid_optfile_format(fp,
+ "option '--%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ else
+ invalid_optfile_format(fp,
+ "option '-%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ return false;
+ }
+
+ resetStringInfo(optargument);
+
+ /* simple case */
+ if (*str != '"')
+ {
+ char *start = str;
+
+ /* read first white char */
+ while (*str != '\0' && *str != '#')
+ {
+ if (*str == ' ')
+ break;
+ str++;
+ }
+
+ appendBinaryStringInfo(optargument, start, str - start);
+ }
+ else
+ {
+ appendStringInfoChar(optargument, *str++);
+
+ while (1)
+ {
+ int c = *str++;
+
+ if (c == '\0')
+ {
+ /* multiline string, read next line */
+ if (!pg_get_line_buf(fp, line))
+ {
+ invalid_optfile_format(fp,
+ "unexpected end of line at line %d",
+ NULL, 0,
+ NULL,
+ *lineno);
+ return false;
+ }
+
+ if (ferror(fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m",
+ filename);
+ return false;
+ }
+
+ appendStringInfoChar(optargument, '\n');
+
+ str = line->data;
+ (void) pg_strip_crlf(str);
+ *lineno += 1;
+
+ c = *str++;
+ }
+
+ if (c == '\\')
+ {
+ c = *str++;
+
+ if (c == '\0')
+ {
+ invalid_optfile_format(fp,
+ "unexpected end of line at line %d",
+ NULL, 0,
+ NULL,
+ *lineno);
+ return false;
+ }
+
+ switch (c)
+ {
+ case 'n':
+ appendStringInfoChar(optargument, '\n');
+ break;
+
+ case 't':
+ appendStringInfoChar(optargument, '\t');
+ break;
+
+ case '\\':
+ appendStringInfoChar(optargument, '\\');
+ break;
+
+ default:
+ appendStringInfoChar(optargument, '\\');
+ appendStringInfoChar(optargument, c);
+ }
+ }
+ else
+ {
+ appendStringInfoChar(optargument, c);
+
+ if (c == '"')
+ {
+ if (*str == '"')
+ str++;
+ else
+ break;
+ }
+ }
+ }
+ }
+
+ /* check garbage after optarg, but ignore white-space */
+ while (isspace(*str))
+ str++;
+
+ /* at the end there should be EOL or comment symbol */
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "unexpected characters after an option's argument at line %d",
+ NULL, 0,
+ line->data,
+ *lineno);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Returns true, when long option is defined. Assign code,
+ * and check if option has required argument.
+ */
+static bool
+is_recognized_longopt(const struct option *longopts,
+ char *optname,
+ int optnamelen,
+ int *opt,
+ bool *has_arg)
+{
+ int i;
+
+ for (i = 0; longopts[i].name != NULL; i++)
+ {
+ if (strlen(longopts[i].name) == optnamelen &&
+ strncmp(optname, longopts[i].name, optnamelen) == 0)
+ {
+ *has_arg = longopts[i].has_arg == required_argument;
+
+ if (longopts[i].flag == NULL)
+ {
+ *opt = longopts[i].val;
+
+ return true;
+ }
+ else
+ {
+ *longopts[i].flag = longopts[i].val;
+ *opt = 0;
+
+ return true;
+ }
+ }
+ }
+
+ return false;
+}
+
+/*
+ * Returns true, when short option char is defined.
+ */
+static bool
+is_valid_shortopt(const char *optstring,
+ char optname,
+ bool *has_arg)
+{
+ while (*optstring != '\0')
+ {
+ if (*optstring != ':' && *optstring == optname)
+ {
+ *has_arg = optstring[1] == ':';
+ return true;
+ }
+
+ optstring++;
+ }
+
+ return false;
+}
+
+/*
+ * Read dumped object specification from file. Returns
+ * true, when processing was ok.
+ */
+static bool
+read_options_from_file(const char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+ StringInfoData optargument;
+
+ /* Ignore already processed files */
+ if (simple_string_list_member(&optsfilenames_processed,
+ filename))
+ {
+ pg_log_warning("the options file \"%s\" was processed already, skip this",
+ filename);
+ return true;
+ }
+
+ simple_string_list_append(&optsfilenames_processed, filename);
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ {
+ pg_log_error("could not open the input file \"%s\": %m",
+ filename);
+ return false;
+ }
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+ initStringInfo(&optargument);
+
+ while (pg_get_line_buf(fp, &line))
+ {
+ char *optname;
+ char *str = line.data;
+ int opt;
+ int optnamelen;
+ bool has_arg;
+
+ (void) pg_strip_crlf(str);
+
+ lineno += 1;
+
+ /* skip initial spaces */
+ while (isspace(*str))
+ str++;
+
+ /* Ignore empty lines or lines with hash symbol (comment) */
+ if (*str == '\0' || *str == '#')
+ continue;
+
+ if (*str++ != '-')
+ {
+ invalid_optfile_format(fp,
+ "non option arguments are not allowed in options file at line %d",
+ NULL, 0,
+ line.data,
+ lineno);
+ return false;
+ }
+
+ if (*str == '-')
+ {
+ /* process long option */
+ str++;
+ optname = str++;
+
+ while (!isspace(*str) && *str != '=' && *str != '\0')
+ str++;
+
+ optnamelen = str - optname;
+
+ if (is_recognized_longopt(longopts, optname, optnamelen,
+ &opt, &has_arg))
+ {
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (has_arg)
+ {
+ /* skip optional = */
+ if (*str == '=')
+ str++;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (!read_optarg(fp, filename, str, &line, &optargument,
+ optname, optnamelen, true, &lineno))
+ return false;
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "option '--%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ }
+ else
+ {
+ invalid_optfile_format(fp,
+ "unrecognized option '--%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ else
+ {
+ /* process short option */
+ optname = str++;
+ opt = *optname;
+
+ optnamelen = 1;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (is_valid_shortopt(optstring, opt, &has_arg))
+ {
+ if (has_arg)
+ {
+ if (!read_optarg(fp, filename, str, &line, &optargument,
+ optname, optnamelen, false, &lineno))
+ return false;
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "option '-%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ }
+ else
+ {
+ invalid_optfile_format(fp,
+ "invalid option '-%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+
+ /* nested options file reading */
+ if (opt == OPTIONS_FILE_OPTNUM)
+ {
+ if (!read_options_from_file(optargument.data, dopt, optstring,
+ longopts, progname))
+ {
+ fclose(fp);
+ return false;
+ }
+ }
+ else if (opt != 0 &&
+ !process_option(opt, optargument.data, dopt, progname))
+ {
+ fclose(fp);
+ return false;
+ }
+ }
+
+ pfree(line.data);
+ pfree(optargument.data);
+
+ if (ferror(fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m", filename);
+ return false;
+ }
+
+ if (fp != stdin)
+ fclose(fp);
+
+ return true;
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_optsfile.pl b/src/bin/pg_dump/t/004_pg_dump_optsfile.pl
new file mode 100644
index 0000000000..0955e28a85
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_optsfile.pl
@@ -0,0 +1,165 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 30;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = get_new_node('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-t table_one #comment\n";
+print $inputfile "-t table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "--exclude-table-data=table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-T table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "-N public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "--include-foreign-data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: non option arguments are not allowed in options file at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid option '-' at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-t";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '-t' requires an argument at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-a someforeignserver";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '-a' doesn't allow an argument at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "-r";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: invalid option '-r' at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "--doesnt-exists";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: unrecognized option '--doesnt-exists' at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "--data-only badparameter";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '--data-only' doesn't allow an argument at line 1/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "--table";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--options-file=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: option '--table' requires an argument at line 1/,
+ "broken format check");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-04-11 07:48 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2021-04-11 07:48 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Tom Lane <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Hi
út 16. 2. 2021 v 20:32 odesílatel Pavel Stehule <[email protected]>
napsal:
>
> Hi
>
> rebase
>
fresh rebase
Regards
Pavel
> Regards
>
> Pavel
>
Attachments:
[text/x-patch] pg_dump-options-file-4.patch (24.8K, ../../CAFj8pRBmOm9yxvOLbvYfZvAdVJ4jc084=-c3MG5_kZDk0NL5UA@mail.gmail.com/3-pg_dump-options-file-4.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 529b167c96..24bfe07ee9 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -1000,6 +1000,42 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--options-file=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read options from file (one option per line). Short or long options
+ are supported. If you use "-" as a filename, the filters are read
+ from stdin.
+ </para>
+
+ <para>
+ With the following options file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
+-t mytable1
+--include-foreign-data=some_foreign_server
+--exclude-table-data=mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The text after symbol <literal>#</literal> is ignored. This can
+ be used for comments, notes. Empty lines are ignored too.
+ </para>
+
+ <para>
+ The option <option>--options-file</option> can be used more times,
+ and the nesting is allowed. The options from options files are
+ processed first, other options from command line later. Any option
+ file is processed only one time. In next time the processing is
+ ignored.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--quote-all-identifiers</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index d0ea489614..665b719c89 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -54,9 +54,11 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -126,18 +128,35 @@ static SimpleOidList foreign_servers_include_oids = {NULL, NULL};
static SimpleStringList extension_include_patterns = {NULL, NULL};
static SimpleOidList extension_include_oids = {NULL, NULL};
+static SimpleStringList optsfilenames_processed = {NULL, NULL};
+
static const CatalogId nilCatalogId = {0, 0};
/* override for standard extra_float_digits setting */
static bool have_extra_float_digits = false;
static int extra_float_digits;
+static const char *filename = NULL;
+static const char *format = "p";
+static bool g_verbose = false;
+static const char *dumpencoding = NULL;
+static const char *dumpsnapshot = NULL;
+static char *use_role = NULL;
+static long rowsPerInsert;
+static int numWorkers = 1;
+static int compressLevel = -1;
+
/*
* The default number of rows per INSERT when
* --inserts is specified without --rows-per-insert
*/
#define DUMP_DEFAULT_ROWS_PER_INSERT 1
+/*
+ * Option's code of "options-file" option
+ */
+#define OPTIONS_FILE_OPTNUM 12
+
/*
* Macro for producing quoted, schema-qualified name of a dumpable object.
*/
@@ -304,14 +323,226 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static bool read_options_from_file(const char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname);
+
+/*
+ * It assigns the values of options to related DumpOption fields or to
+ * some global values. Options file loading is not processed here.
+ */
+static bool
+process_option(int opt,
+ char *optargstr,
+ DumpOptions *dopt,
+ const char *progname)
+{
+ char *endptr;
+
+ switch (opt)
+ {
+ case 'a': /* Dump data only */
+ dopt->dataOnly = true;
+ break;
+
+ case 'b': /* Dump blobs */
+ dopt->outputBlobs = true;
+ break;
+
+ case 'B': /* Don't dump blobs */
+ dopt->dontOutputBlobs = true;
+ break;
+
+ case 'c': /* clean (i.e., drop) schema prior to create */
+ dopt->outputClean = 1;
+ break;
+
+ case 'C': /* Create DB */
+ dopt->outputCreateDB = 1;
+ break;
+
+ case 'd': /* database name */
+ dopt->cparams.dbname = pg_strdup(optargstr);
+ break;
+
+ case 'e': /* include extension(s) */
+ simple_string_list_append(&extension_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'E': /* Dump encoding */
+ dumpencoding = pg_strdup(optargstr);
+ break;
+
+ case 'f':
+ filename = pg_strdup(optargstr);
+ break;
+
+ case 'F':
+ format = pg_strdup(optargstr);
+ break;
+
+ case 'h': /* server host */
+ dopt->cparams.pghost = pg_strdup(optargstr);
+ break;
+
+ case 'j': /* number of dump jobs */
+ numWorkers = atoi(optargstr);
+ break;
+
+ case 'n': /* include schema(s) */
+ simple_string_list_append(&schema_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'N': /* exclude schema(s) */
+ simple_string_list_append(&schema_exclude_patterns, optargstr);
+ break;
+
+ case 'O': /* Don't reconnect to match owner */
+ dopt->outputNoOwner = 1;
+ break;
+
+ case 'p': /* server port */
+ dopt->cparams.pgport = pg_strdup(optargstr);
+ break;
+
+ case 'R':
+ /* no-op, still accepted for backwards compatibility */
+ break;
+
+ case 's': /* dump schema only */
+ dopt->schemaOnly = true;
+ break;
+
+ case 'S': /* Username for superuser in plain text output */
+ dopt->outputSuperuser = pg_strdup(optargstr);
+ break;
+
+ case 't': /* include table(s) */
+ simple_string_list_append(&table_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'T': /* exclude table(s) */
+ simple_string_list_append(&table_exclude_patterns, optargstr);
+ break;
+
+ case 'U':
+ dopt->cparams.username = pg_strdup(optargstr);
+ break;
+
+ case 'v': /* verbose */
+ g_verbose = true;
+ pg_logging_increase_verbosity();
+ break;
+
+ case 'w':
+ dopt->cparams.promptPassword = TRI_NO;
+ break;
+
+ case 'W':
+ dopt->cparams.promptPassword = TRI_YES;
+ break;
+
+ case 'x': /* skip ACL dump */
+ dopt->aclsSkip = true;
+ break;
+
+ case 'Z': /* Compression Level */
+ compressLevel = atoi(optargstr);
+ if (compressLevel < 0 || compressLevel > 9)
+ {
+ pg_log_error("compression level must be in range 0..9");
+ return false;
+ }
+ break;
+
+ case 0:
+ /* This covers the long options. */
+ break;
+
+ case 2: /* lock-wait-timeout */
+ dopt->lockWaitTimeout = pg_strdup(optargstr);
+ break;
+
+ case 3: /* SET ROLE */
+ use_role = pg_strdup(optargstr);
+ break;
+
+ case 4: /* exclude table(s) data */
+ simple_string_list_append(&tabledata_exclude_patterns, optargstr);
+ break;
+
+ case 5: /* section */
+ set_dump_section(optargstr, &dopt->dumpSections);
+ break;
+
+ case 6: /* snapshot */
+ dumpsnapshot = pg_strdup(optargstr);
+ break;
+
+ case 7: /* no-sync */
+ dosync = false;
+ break;
+
+ case 8:
+ have_extra_float_digits = true;
+ extra_float_digits = atoi(optargstr);
+ if (extra_float_digits < -15 || extra_float_digits > 3)
+ {
+ pg_log_error("extra_float_digits must be in range -15..3");
+ return false;
+ }
+ break;
+
+ case 9: /* inserts */
+
+ /*
+ * dump_inserts also stores --rows-per-insert, careful not to
+ * overwrite that.
+ */
+ if (dopt->dump_inserts == 0)
+ dopt->dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
+ break;
+
+ case 10: /* rows per insert */
+ errno = 0;
+ rowsPerInsert = strtol(optargstr, &endptr, 10);
+
+ if (endptr == optargstr || *endptr != '\0' ||
+ rowsPerInsert <= 0 || rowsPerInsert > INT_MAX ||
+ errno == ERANGE)
+ {
+ pg_log_error("rows-per-insert must be in range %d..%d",
+ 1, INT_MAX);
+ return false;
+ }
+ dopt->dump_inserts = (int) rowsPerInsert;
+ break;
+
+ case 11: /* include foreign data */
+ simple_string_list_append(&foreign_servers_include_patterns,
+ optargstr);
+ break;
+
+ case OPTIONS_FILE_OPTNUM: /* reading options file */
+ break; /* should not be processed here ever */
+
+ default:
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ return false;
+ }
+ return true;
+}
int
main(int argc, char **argv)
{
int c;
- const char *filename = NULL;
- const char *format = "p";
TableInfo *tblinfo;
int numTables;
DumpableObject **dobjs;
@@ -319,20 +550,11 @@ main(int argc, char **argv)
DumpableObject *boundaryObjs;
int i;
int optindex;
- char *endptr;
RestoreOptions *ropt;
Archive *fout; /* the script file */
- bool g_verbose = false;
- const char *dumpencoding = NULL;
- const char *dumpsnapshot = NULL;
- char *use_role = NULL;
- long rowsPerInsert;
- int numWorkers = 1;
- int compressLevel = -1;
int plainText = 0;
ArchiveFormat archiveFormat = archUnknown;
ArchiveMode archiveMode;
-
static DumpOptions dopt;
static struct option long_options[] = {
@@ -399,13 +621,17 @@ main(int argc, char **argv)
{"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1},
{"no-sync", no_argument, NULL, 7},
{"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1},
+ {"options-file", required_argument, NULL, OPTIONS_FILE_OPTNUM},
{"rows-per-insert", required_argument, NULL, 10},
{"include-foreign-data", required_argument, NULL, 11},
{"index-collation-versions-unknown", no_argument, &dopt.coll_unknown, 1},
+ {"include-foreign-data-file", required_argument, NULL, 17},
{NULL, 0, NULL, 0}
};
+ const char *short_options = "abBcCd:e:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:";
+
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_dump"));
@@ -434,202 +660,33 @@ main(int argc, char **argv)
InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "abBcCd:e:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:",
+ /*
+ * We would to process options-file opts before other options.
+ * This implements higher priority of options from command line.
+ * Later processed options wins because overwrite result of
+ * early processed options.
+ */
+ while ((c = getopt_long(argc, argv, short_options,
long_options, &optindex)) != -1)
{
- switch (c)
+ if (c == OPTIONS_FILE_OPTNUM)
{
- case 'a': /* Dump data only */
- dopt.dataOnly = true;
- break;
-
- case 'b': /* Dump blobs */
- dopt.outputBlobs = true;
- break;
-
- case 'B': /* Don't dump blobs */
- dopt.dontOutputBlobs = true;
- break;
-
- case 'c': /* clean (i.e., drop) schema prior to create */
- dopt.outputClean = 1;
- break;
-
- case 'C': /* Create DB */
- dopt.outputCreateDB = 1;
- break;
-
- case 'd': /* database name */
- dopt.cparams.dbname = pg_strdup(optarg);
- break;
-
- case 'e': /* include extension(s) */
- simple_string_list_append(&extension_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'E': /* Dump encoding */
- dumpencoding = pg_strdup(optarg);
- break;
-
- case 'f':
- filename = pg_strdup(optarg);
- break;
-
- case 'F':
- format = pg_strdup(optarg);
- break;
-
- case 'h': /* server host */
- dopt.cparams.pghost = pg_strdup(optarg);
- break;
-
- case 'j': /* number of dump jobs */
- numWorkers = atoi(optarg);
- break;
-
- case 'n': /* include schema(s) */
- simple_string_list_append(&schema_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'N': /* exclude schema(s) */
- simple_string_list_append(&schema_exclude_patterns, optarg);
- break;
-
- case 'O': /* Don't reconnect to match owner */
- dopt.outputNoOwner = 1;
- break;
-
- case 'p': /* server port */
- dopt.cparams.pgport = pg_strdup(optarg);
- break;
-
- case 'R':
- /* no-op, still accepted for backwards compatibility */
- break;
-
- case 's': /* dump schema only */
- dopt.schemaOnly = true;
- break;
-
- case 'S': /* Username for superuser in plain text output */
- dopt.outputSuperuser = pg_strdup(optarg);
- break;
-
- case 't': /* include table(s) */
- simple_string_list_append(&table_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'T': /* exclude table(s) */
- simple_string_list_append(&table_exclude_patterns, optarg);
- break;
-
- case 'U':
- dopt.cparams.username = pg_strdup(optarg);
- break;
-
- case 'v': /* verbose */
- g_verbose = true;
- pg_logging_increase_verbosity();
- break;
-
- case 'w':
- dopt.cparams.promptPassword = TRI_NO;
- break;
-
- case 'W':
- dopt.cparams.promptPassword = TRI_YES;
- break;
-
- case 'x': /* skip ACL dump */
- dopt.aclsSkip = true;
- break;
-
- case 'Z': /* Compression Level */
- compressLevel = atoi(optarg);
- if (compressLevel < 0 || compressLevel > 9)
- {
- pg_log_error("compression level must be in range 0..9");
- exit_nicely(1);
- }
- break;
-
- case 0:
- /* This covers the long options. */
- break;
-
- case 2: /* lock-wait-timeout */
- dopt.lockWaitTimeout = pg_strdup(optarg);
- break;
-
- case 3: /* SET ROLE */
- use_role = pg_strdup(optarg);
- break;
-
- case 4: /* exclude table(s) data */
- simple_string_list_append(&tabledata_exclude_patterns, optarg);
- break;
-
- case 5: /* section */
- set_dump_section(optarg, &dopt.dumpSections);
- break;
-
- case 6: /* snapshot */
- dumpsnapshot = pg_strdup(optarg);
- break;
-
- case 7: /* no-sync */
- dosync = false;
- break;
-
- case 8:
- have_extra_float_digits = true;
- extra_float_digits = atoi(optarg);
- if (extra_float_digits < -15 || extra_float_digits > 3)
- {
- pg_log_error("extra_float_digits must be in range -15..3");
- exit_nicely(1);
- }
- break;
-
- case 9: /* inserts */
-
- /*
- * dump_inserts also stores --rows-per-insert, careful not to
- * overwrite that.
- */
- if (dopt.dump_inserts == 0)
- dopt.dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
- break;
-
- case 10: /* rows per insert */
- errno = 0;
- rowsPerInsert = strtol(optarg, &endptr, 10);
-
- if (endptr == optarg || *endptr != '\0' ||
- rowsPerInsert <= 0 || rowsPerInsert > INT_MAX ||
- errno == ERANGE)
- {
- pg_log_error("rows-per-insert must be in range %d..%d",
- 1, INT_MAX);
- exit_nicely(1);
- }
- dopt.dump_inserts = (int) rowsPerInsert;
- break;
-
- case 11: /* include foreign data */
- simple_string_list_append(&foreign_servers_include_patterns,
- optarg);
- break;
-
- default:
- fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ if (!read_options_from_file(optarg, &dopt, short_options,
+ long_options, progname))
exit_nicely(1);
}
}
+ /* reset getopt_long index */
+ optind = 1;
+
+ while ((c = getopt_long(argc, argv, short_options,
+ long_options, &optindex)) != -1)
+ {
+ if (!process_option(c, optarg, &dopt, progname))
+ exit_nicely(1);
+ }
+
/*
* Non-option argument specifies database name as long as it wasn't
* already specified with -d / --dbname
@@ -1082,6 +1139,7 @@ help(const char *progname)
printf(_(" --no-toast-compression do not dump toast compression methods\n"));
printf(_(" --no-unlogged-table-data do not dump unlogged table data\n"));
printf(_(" --on-conflict-do-nothing add ON CONFLICT DO NOTHING to INSERT commands\n"));
+ printf(_(" --options-file=FILENAME read options from options file\n"));
printf(_(" --quote-all-identifiers quote all identifiers, even if not key words\n"));
printf(_(" --rows-per-insert=NROWS number of rows per INSERT; implies --inserts\n"));
printf(_(" --section=SECTION dump named section (pre-data, data, or post-data)\n"));
@@ -18980,3 +19038,450 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and close input file
+ */
+static void
+invalid_optfile_format(FILE *fp,
+ char *message,
+ char *optname,
+ int optnamelen,
+ char *line,
+ int lineno)
+{
+ Assert(message);
+
+ if (optnamelen > 0)
+ {
+ Assert(optname);
+ pg_log_error(message, optnamelen, optname, lineno);
+ }
+ else
+ pg_log_error(message, lineno);
+
+ if (line)
+ fprintf(stderr, "LINE %d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+}
+
+/*
+ * Reads an option argument from file. Supports double-quoted
+ * strings. In this case multi-line strings are supported.
+ */
+static bool
+read_optarg(FILE *fp,
+ const char *filename,
+ char *str,
+ StringInfo line,
+ StringInfo optargument,
+ char *optname,
+ int optnamelen,
+ bool islongopt,
+ int *lineno)
+{
+ if (*str == '\0' || *str == '#')
+ {
+ if (islongopt)
+ invalid_optfile_format(fp,
+ "option '--%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ else
+ invalid_optfile_format(fp,
+ "option '-%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ return false;
+ }
+
+ resetStringInfo(optargument);
+
+ /* simple case */
+ if (*str != '"')
+ {
+ char *start = str;
+
+ /* read first white char */
+ while (*str != '\0' && *str != '#')
+ {
+ if (*str == ' ')
+ break;
+ str++;
+ }
+
+ appendBinaryStringInfo(optargument, start, str - start);
+ }
+ else
+ {
+ appendStringInfoChar(optargument, *str++);
+
+ while (1)
+ {
+ int c = *str++;
+
+ if (c == '\0')
+ {
+ /* multiline string, read next line */
+ if (!pg_get_line_buf(fp, line))
+ {
+ invalid_optfile_format(fp,
+ "unexpected end of line at line %d",
+ NULL, 0,
+ NULL,
+ *lineno);
+ return false;
+ }
+
+ if (ferror(fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m",
+ filename);
+ return false;
+ }
+
+ appendStringInfoChar(optargument, '\n');
+
+ str = line->data;
+ (void) pg_strip_crlf(str);
+ *lineno += 1;
+
+ c = *str++;
+ }
+
+ if (c == '\\')
+ {
+ c = *str++;
+
+ if (c == '\0')
+ {
+ invalid_optfile_format(fp,
+ "unexpected end of line at line %d",
+ NULL, 0,
+ NULL,
+ *lineno);
+ return false;
+ }
+
+ switch (c)
+ {
+ case 'n':
+ appendStringInfoChar(optargument, '\n');
+ break;
+
+ case 't':
+ appendStringInfoChar(optargument, '\t');
+ break;
+
+ case '\\':
+ appendStringInfoChar(optargument, '\\');
+ break;
+
+ default:
+ appendStringInfoChar(optargument, '\\');
+ appendStringInfoChar(optargument, c);
+ }
+ }
+ else
+ {
+ appendStringInfoChar(optargument, c);
+
+ if (c == '"')
+ {
+ if (*str == '"')
+ str++;
+ else
+ break;
+ }
+ }
+ }
+ }
+
+ /* check garbage after optarg, but ignore white-space */
+ while (isspace(*str))
+ str++;
+
+ /* at the end there should be EOL or comment symbol */
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "unexpected characters after an option's argument at line %d",
+ NULL, 0,
+ line->data,
+ *lineno);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Returns true, when long option is defined. Assign code,
+ * and check if option has required argument.
+ */
+static bool
+is_recognized_longopt(const struct option *longopts,
+ char *optname,
+ int optnamelen,
+ int *opt,
+ bool *has_arg)
+{
+ int i;
+
+ for (i = 0; longopts[i].name != NULL; i++)
+ {
+ if (strlen(longopts[i].name) == optnamelen &&
+ strncmp(optname, longopts[i].name, optnamelen) == 0)
+ {
+ *has_arg = longopts[i].has_arg == required_argument;
+
+ if (longopts[i].flag == NULL)
+ {
+ *opt = longopts[i].val;
+
+ return true;
+ }
+ else
+ {
+ *longopts[i].flag = longopts[i].val;
+ *opt = 0;
+
+ return true;
+ }
+ }
+ }
+
+ return false;
+}
+
+/*
+ * Returns true, when short option char is defined.
+ */
+static bool
+is_valid_shortopt(const char *optstring,
+ char optname,
+ bool *has_arg)
+{
+ while (*optstring != '\0')
+ {
+ if (*optstring != ':' && *optstring == optname)
+ {
+ *has_arg = optstring[1] == ':';
+ return true;
+ }
+
+ optstring++;
+ }
+
+ return false;
+}
+
+/*
+ * Read dumped object specification from file. Returns
+ * true, when processing was ok.
+ */
+static bool
+read_options_from_file(const char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+ StringInfoData optargument;
+
+ /* Ignore already processed files */
+ if (simple_string_list_member(&optsfilenames_processed,
+ filename))
+ {
+ pg_log_warning("the options file \"%s\" was processed already, skip this",
+ filename);
+ return true;
+ }
+
+ simple_string_list_append(&optsfilenames_processed, filename);
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ {
+ pg_log_error("could not open the input file \"%s\": %m",
+ filename);
+ return false;
+ }
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+ initStringInfo(&optargument);
+
+ while (pg_get_line_buf(fp, &line))
+ {
+ char *optname;
+ char *str = line.data;
+ int opt;
+ int optnamelen;
+ bool has_arg;
+
+ (void) pg_strip_crlf(str);
+
+ lineno += 1;
+
+ /* skip initial spaces */
+ while (isspace(*str))
+ str++;
+
+ /* Ignore empty lines or lines with hash symbol (comment) */
+ if (*str == '\0' || *str == '#')
+ continue;
+
+ if (*str++ != '-')
+ {
+ invalid_optfile_format(fp,
+ "non option arguments are not allowed in options file at line %d",
+ NULL, 0,
+ line.data,
+ lineno);
+ return false;
+ }
+
+ if (*str == '-')
+ {
+ /* process long option */
+ str++;
+ optname = str++;
+
+ while (!isspace(*str) && *str != '=' && *str != '\0')
+ str++;
+
+ optnamelen = str - optname;
+
+ if (is_recognized_longopt(longopts, optname, optnamelen,
+ &opt, &has_arg))
+ {
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (has_arg)
+ {
+ /* skip optional = */
+ if (*str == '=')
+ str++;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (!read_optarg(fp, filename, str, &line, &optargument,
+ optname, optnamelen, true, &lineno))
+ return false;
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "option '--%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ }
+ else
+ {
+ invalid_optfile_format(fp,
+ "unrecognized option '--%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ else
+ {
+ /* process short option */
+ optname = str++;
+ opt = *optname;
+
+ optnamelen = 1;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (is_valid_shortopt(optstring, opt, &has_arg))
+ {
+ if (has_arg)
+ {
+ if (!read_optarg(fp, filename, str, &line, &optargument,
+ optname, optnamelen, false, &lineno))
+ return false;
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "option '-%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ }
+ else
+ {
+ invalid_optfile_format(fp,
+ "invalid option '-%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+
+ /* nested options file reading */
+ if (opt == OPTIONS_FILE_OPTNUM)
+ {
+ if (!read_options_from_file(optargument.data, dopt, optstring,
+ longopts, progname))
+ {
+ fclose(fp);
+ return false;
+ }
+ }
+ else if (opt != 0 &&
+ !process_option(opt, optargument.data, dopt, progname))
+ {
+ fclose(fp);
+ return false;
+ }
+ }
+
+ pfree(line.data);
+ pfree(optargument.data);
+
+ if (ferror(fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m", filename);
+ return false;
+ }
+
+ if (fp != stdin)
+ fclose(fp);
+
+ return true;
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-05-12 06:22 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-05-12 06:22 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Tom Lane <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Hi
ne 11. 4. 2021 v 9:48 odesílatel Pavel Stehule <[email protected]>
napsal:
> Hi
>
> út 16. 2. 2021 v 20:32 odesílatel Pavel Stehule <[email protected]>
> napsal:
>
>>
>> Hi
>>
>> rebase
>>
>
>
rebase
> fresh rebase
>
> Regards
>
> Pavel
>
>
>> Regards
>>
>> Pavel
>>
>
Attachments:
[text/x-patch] pg_dump-options-file-5.patch (24.8K, ../../CAFj8pRBkHSSq4epfVBfkabBtf_ZZMcRb8uC-YgxdbOzwCnhKug@mail.gmail.com/3-pg_dump-options-file-5.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 67c2cbbec6..742e3515a3 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -1006,6 +1006,42 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--options-file=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read options from file (one option per line). Short or long options
+ are supported. If you use "-" as a filename, the filters are read
+ from stdin.
+ </para>
+
+ <para>
+ With the following options file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
+-t mytable1
+--include-foreign-data=some_foreign_server
+--exclude-table-data=mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The text after symbol <literal>#</literal> is ignored. This can
+ be used for comments, notes. Empty lines are ignored too.
+ </para>
+
+ <para>
+ The option <option>--options-file</option> can be used more times,
+ and the nesting is allowed. The options from options files are
+ processed first, other options from command line later. Any option
+ file is processed only one time. In next time the processing is
+ ignored.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--quote-all-identifiers</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e384690d94..085791895c 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -53,9 +53,11 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -125,18 +127,35 @@ static SimpleOidList foreign_servers_include_oids = {NULL, NULL};
static SimpleStringList extension_include_patterns = {NULL, NULL};
static SimpleOidList extension_include_oids = {NULL, NULL};
+static SimpleStringList optsfilenames_processed = {NULL, NULL};
+
static const CatalogId nilCatalogId = {0, 0};
/* override for standard extra_float_digits setting */
static bool have_extra_float_digits = false;
static int extra_float_digits;
+static const char *filename = NULL;
+static const char *format = "p";
+static bool g_verbose = false;
+static const char *dumpencoding = NULL;
+static const char *dumpsnapshot = NULL;
+static char *use_role = NULL;
+static long rowsPerInsert;
+static int numWorkers = 1;
+static int compressLevel = -1;
+
/*
* The default number of rows per INSERT when
* --inserts is specified without --rows-per-insert
*/
#define DUMP_DEFAULT_ROWS_PER_INSERT 1
+/*
+ * Option's code of "options-file" option
+ */
+#define OPTIONS_FILE_OPTNUM 12
+
/*
* Macro for producing quoted, schema-qualified name of a dumpable object.
*/
@@ -300,14 +319,226 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static bool read_options_from_file(const char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname);
+
+/*
+ * It assigns the values of options to related DumpOption fields or to
+ * some global values. Options file loading is not processed here.
+ */
+static bool
+process_option(int opt,
+ char *optargstr,
+ DumpOptions *dopt,
+ const char *progname)
+{
+ char *endptr;
+
+ switch (opt)
+ {
+ case 'a': /* Dump data only */
+ dopt->dataOnly = true;
+ break;
+
+ case 'b': /* Dump blobs */
+ dopt->outputBlobs = true;
+ break;
+
+ case 'B': /* Don't dump blobs */
+ dopt->dontOutputBlobs = true;
+ break;
+
+ case 'c': /* clean (i.e., drop) schema prior to create */
+ dopt->outputClean = 1;
+ break;
+
+ case 'C': /* Create DB */
+ dopt->outputCreateDB = 1;
+ break;
+
+ case 'd': /* database name */
+ dopt->cparams.dbname = pg_strdup(optargstr);
+ break;
+
+ case 'e': /* include extension(s) */
+ simple_string_list_append(&extension_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'E': /* Dump encoding */
+ dumpencoding = pg_strdup(optargstr);
+ break;
+
+ case 'f':
+ filename = pg_strdup(optargstr);
+ break;
+
+ case 'F':
+ format = pg_strdup(optargstr);
+ break;
+
+ case 'h': /* server host */
+ dopt->cparams.pghost = pg_strdup(optargstr);
+ break;
+
+ case 'j': /* number of dump jobs */
+ numWorkers = atoi(optargstr);
+ break;
+
+ case 'n': /* include schema(s) */
+ simple_string_list_append(&schema_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'N': /* exclude schema(s) */
+ simple_string_list_append(&schema_exclude_patterns, optargstr);
+ break;
+
+ case 'O': /* Don't reconnect to match owner */
+ dopt->outputNoOwner = 1;
+ break;
+
+ case 'p': /* server port */
+ dopt->cparams.pgport = pg_strdup(optargstr);
+ break;
+
+ case 'R':
+ /* no-op, still accepted for backwards compatibility */
+ break;
+
+ case 's': /* dump schema only */
+ dopt->schemaOnly = true;
+ break;
+
+ case 'S': /* Username for superuser in plain text output */
+ dopt->outputSuperuser = pg_strdup(optargstr);
+ break;
+
+ case 't': /* include table(s) */
+ simple_string_list_append(&table_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'T': /* exclude table(s) */
+ simple_string_list_append(&table_exclude_patterns, optargstr);
+ break;
+
+ case 'U':
+ dopt->cparams.username = pg_strdup(optargstr);
+ break;
+
+ case 'v': /* verbose */
+ g_verbose = true;
+ pg_logging_increase_verbosity();
+ break;
+
+ case 'w':
+ dopt->cparams.promptPassword = TRI_NO;
+ break;
+
+ case 'W':
+ dopt->cparams.promptPassword = TRI_YES;
+ break;
+
+ case 'x': /* skip ACL dump */
+ dopt->aclsSkip = true;
+ break;
+
+ case 'Z': /* Compression Level */
+ compressLevel = atoi(optargstr);
+ if (compressLevel < 0 || compressLevel > 9)
+ {
+ pg_log_error("compression level must be in range 0..9");
+ return false;
+ }
+ break;
+
+ case 0:
+ /* This covers the long options. */
+ break;
+
+ case 2: /* lock-wait-timeout */
+ dopt->lockWaitTimeout = pg_strdup(optargstr);
+ break;
+
+ case 3: /* SET ROLE */
+ use_role = pg_strdup(optargstr);
+ break;
+
+ case 4: /* exclude table(s) data */
+ simple_string_list_append(&tabledata_exclude_patterns, optargstr);
+ break;
+
+ case 5: /* section */
+ set_dump_section(optargstr, &dopt->dumpSections);
+ break;
+
+ case 6: /* snapshot */
+ dumpsnapshot = pg_strdup(optargstr);
+ break;
+
+ case 7: /* no-sync */
+ dosync = false;
+ break;
+
+ case 8:
+ have_extra_float_digits = true;
+ extra_float_digits = atoi(optargstr);
+ if (extra_float_digits < -15 || extra_float_digits > 3)
+ {
+ pg_log_error("extra_float_digits must be in range -15..3");
+ return false;
+ }
+ break;
+
+ case 9: /* inserts */
+
+ /*
+ * dump_inserts also stores --rows-per-insert, careful not to
+ * overwrite that.
+ */
+ if (dopt->dump_inserts == 0)
+ dopt->dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
+ break;
+
+ case 10: /* rows per insert */
+ errno = 0;
+ rowsPerInsert = strtol(optargstr, &endptr, 10);
+
+ if (endptr == optargstr || *endptr != '\0' ||
+ rowsPerInsert <= 0 || rowsPerInsert > INT_MAX ||
+ errno == ERANGE)
+ {
+ pg_log_error("rows-per-insert must be in range %d..%d",
+ 1, INT_MAX);
+ return false;
+ }
+ dopt->dump_inserts = (int) rowsPerInsert;
+ break;
+
+ case 11: /* include foreign data */
+ simple_string_list_append(&foreign_servers_include_patterns,
+ optargstr);
+ break;
+
+ case OPTIONS_FILE_OPTNUM: /* reading options file */
+ break; /* should not be processed here ever */
+
+ default:
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ return false;
+ }
+ return true;
+}
int
main(int argc, char **argv)
{
int c;
- const char *filename = NULL;
- const char *format = "p";
TableInfo *tblinfo;
int numTables;
DumpableObject **dobjs;
@@ -315,20 +546,11 @@ main(int argc, char **argv)
DumpableObject *boundaryObjs;
int i;
int optindex;
- char *endptr;
RestoreOptions *ropt;
Archive *fout; /* the script file */
- bool g_verbose = false;
- const char *dumpencoding = NULL;
- const char *dumpsnapshot = NULL;
- char *use_role = NULL;
- long rowsPerInsert;
- int numWorkers = 1;
- int compressLevel = -1;
int plainText = 0;
ArchiveFormat archiveFormat = archUnknown;
ArchiveMode archiveMode;
-
static DumpOptions dopt;
static struct option long_options[] = {
@@ -395,12 +617,15 @@ main(int argc, char **argv)
{"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1},
{"no-sync", no_argument, NULL, 7},
{"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1},
+ {"options-file", required_argument, NULL, OPTIONS_FILE_OPTNUM},
{"rows-per-insert", required_argument, NULL, 10},
{"include-foreign-data", required_argument, NULL, 11},
-
+ {"include-foreign-data-file", required_argument, NULL, 17},
{NULL, 0, NULL, 0}
};
+ const char *short_options = "abBcCd:e:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:";
+
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_dump"));
@@ -429,202 +654,33 @@ main(int argc, char **argv)
InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "abBcCd:e:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:",
+ /*
+ * We would to process options-file opts before other options.
+ * This implements higher priority of options from command line.
+ * Later processed options wins because overwrite result of
+ * early processed options.
+ */
+ while ((c = getopt_long(argc, argv, short_options,
long_options, &optindex)) != -1)
{
- switch (c)
+ if (c == OPTIONS_FILE_OPTNUM)
{
- case 'a': /* Dump data only */
- dopt.dataOnly = true;
- break;
-
- case 'b': /* Dump blobs */
- dopt.outputBlobs = true;
- break;
-
- case 'B': /* Don't dump blobs */
- dopt.dontOutputBlobs = true;
- break;
-
- case 'c': /* clean (i.e., drop) schema prior to create */
- dopt.outputClean = 1;
- break;
-
- case 'C': /* Create DB */
- dopt.outputCreateDB = 1;
- break;
-
- case 'd': /* database name */
- dopt.cparams.dbname = pg_strdup(optarg);
- break;
-
- case 'e': /* include extension(s) */
- simple_string_list_append(&extension_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'E': /* Dump encoding */
- dumpencoding = pg_strdup(optarg);
- break;
-
- case 'f':
- filename = pg_strdup(optarg);
- break;
-
- case 'F':
- format = pg_strdup(optarg);
- break;
-
- case 'h': /* server host */
- dopt.cparams.pghost = pg_strdup(optarg);
- break;
-
- case 'j': /* number of dump jobs */
- numWorkers = atoi(optarg);
- break;
-
- case 'n': /* include schema(s) */
- simple_string_list_append(&schema_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'N': /* exclude schema(s) */
- simple_string_list_append(&schema_exclude_patterns, optarg);
- break;
-
- case 'O': /* Don't reconnect to match owner */
- dopt.outputNoOwner = 1;
- break;
-
- case 'p': /* server port */
- dopt.cparams.pgport = pg_strdup(optarg);
- break;
-
- case 'R':
- /* no-op, still accepted for backwards compatibility */
- break;
-
- case 's': /* dump schema only */
- dopt.schemaOnly = true;
- break;
-
- case 'S': /* Username for superuser in plain text output */
- dopt.outputSuperuser = pg_strdup(optarg);
- break;
-
- case 't': /* include table(s) */
- simple_string_list_append(&table_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'T': /* exclude table(s) */
- simple_string_list_append(&table_exclude_patterns, optarg);
- break;
-
- case 'U':
- dopt.cparams.username = pg_strdup(optarg);
- break;
-
- case 'v': /* verbose */
- g_verbose = true;
- pg_logging_increase_verbosity();
- break;
-
- case 'w':
- dopt.cparams.promptPassword = TRI_NO;
- break;
-
- case 'W':
- dopt.cparams.promptPassword = TRI_YES;
- break;
-
- case 'x': /* skip ACL dump */
- dopt.aclsSkip = true;
- break;
-
- case 'Z': /* Compression Level */
- compressLevel = atoi(optarg);
- if (compressLevel < 0 || compressLevel > 9)
- {
- pg_log_error("compression level must be in range 0..9");
- exit_nicely(1);
- }
- break;
-
- case 0:
- /* This covers the long options. */
- break;
-
- case 2: /* lock-wait-timeout */
- dopt.lockWaitTimeout = pg_strdup(optarg);
- break;
-
- case 3: /* SET ROLE */
- use_role = pg_strdup(optarg);
- break;
-
- case 4: /* exclude table(s) data */
- simple_string_list_append(&tabledata_exclude_patterns, optarg);
- break;
-
- case 5: /* section */
- set_dump_section(optarg, &dopt.dumpSections);
- break;
-
- case 6: /* snapshot */
- dumpsnapshot = pg_strdup(optarg);
- break;
-
- case 7: /* no-sync */
- dosync = false;
- break;
-
- case 8:
- have_extra_float_digits = true;
- extra_float_digits = atoi(optarg);
- if (extra_float_digits < -15 || extra_float_digits > 3)
- {
- pg_log_error("extra_float_digits must be in range -15..3");
- exit_nicely(1);
- }
- break;
-
- case 9: /* inserts */
-
- /*
- * dump_inserts also stores --rows-per-insert, careful not to
- * overwrite that.
- */
- if (dopt.dump_inserts == 0)
- dopt.dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
- break;
-
- case 10: /* rows per insert */
- errno = 0;
- rowsPerInsert = strtol(optarg, &endptr, 10);
-
- if (endptr == optarg || *endptr != '\0' ||
- rowsPerInsert <= 0 || rowsPerInsert > INT_MAX ||
- errno == ERANGE)
- {
- pg_log_error("rows-per-insert must be in range %d..%d",
- 1, INT_MAX);
- exit_nicely(1);
- }
- dopt.dump_inserts = (int) rowsPerInsert;
- break;
-
- case 11: /* include foreign data */
- simple_string_list_append(&foreign_servers_include_patterns,
- optarg);
- break;
-
- default:
- fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ if (!read_options_from_file(optarg, &dopt, short_options,
+ long_options, progname))
exit_nicely(1);
}
}
+ /* reset getopt_long index */
+ optind = 1;
+
+ while ((c = getopt_long(argc, argv, short_options,
+ long_options, &optindex)) != -1)
+ {
+ if (!process_option(c, optarg, &dopt, progname))
+ exit_nicely(1);
+ }
+
/*
* Non-option argument specifies database name as long as it wasn't
* already specified with -d / --dbname
@@ -1073,6 +1129,7 @@ help(const char *progname)
printf(_(" --no-toast-compression do not dump toast compression methods\n"));
printf(_(" --no-unlogged-table-data do not dump unlogged table data\n"));
printf(_(" --on-conflict-do-nothing add ON CONFLICT DO NOTHING to INSERT commands\n"));
+ printf(_(" --options-file=FILENAME read options from options file\n"));
printf(_(" --quote-all-identifiers quote all identifiers, even if not key words\n"));
printf(_(" --rows-per-insert=NROWS number of rows per INSERT; implies --inserts\n"));
printf(_(" --section=SECTION dump named section (pre-data, data, or post-data)\n"));
@@ -18832,3 +18889,450 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and close input file
+ */
+static void
+invalid_optfile_format(FILE *fp,
+ char *message,
+ char *optname,
+ int optnamelen,
+ char *line,
+ int lineno)
+{
+ Assert(message);
+
+ if (optnamelen > 0)
+ {
+ Assert(optname);
+ pg_log_error(message, optnamelen, optname, lineno);
+ }
+ else
+ pg_log_error(message, lineno);
+
+ if (line)
+ fprintf(stderr, "LINE %d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+}
+
+/*
+ * Reads an option argument from file. Supports double-quoted
+ * strings. In this case multi-line strings are supported.
+ */
+static bool
+read_optarg(FILE *fp,
+ const char *filename,
+ char *str,
+ StringInfo line,
+ StringInfo optargument,
+ char *optname,
+ int optnamelen,
+ bool islongopt,
+ int *lineno)
+{
+ if (*str == '\0' || *str == '#')
+ {
+ if (islongopt)
+ invalid_optfile_format(fp,
+ "option '--%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ else
+ invalid_optfile_format(fp,
+ "option '-%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ return false;
+ }
+
+ resetStringInfo(optargument);
+
+ /* simple case */
+ if (*str != '"')
+ {
+ char *start = str;
+
+ /* read first white char */
+ while (*str != '\0' && *str != '#')
+ {
+ if (*str == ' ')
+ break;
+ str++;
+ }
+
+ appendBinaryStringInfo(optargument, start, str - start);
+ }
+ else
+ {
+ appendStringInfoChar(optargument, *str++);
+
+ while (1)
+ {
+ int c = *str++;
+
+ if (c == '\0')
+ {
+ /* multiline string, read next line */
+ if (!pg_get_line_buf(fp, line))
+ {
+ invalid_optfile_format(fp,
+ "unexpected end of line at line %d",
+ NULL, 0,
+ NULL,
+ *lineno);
+ return false;
+ }
+
+ if (ferror(fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m",
+ filename);
+ return false;
+ }
+
+ appendStringInfoChar(optargument, '\n');
+
+ str = line->data;
+ (void) pg_strip_crlf(str);
+ *lineno += 1;
+
+ c = *str++;
+ }
+
+ if (c == '\\')
+ {
+ c = *str++;
+
+ if (c == '\0')
+ {
+ invalid_optfile_format(fp,
+ "unexpected end of line at line %d",
+ NULL, 0,
+ NULL,
+ *lineno);
+ return false;
+ }
+
+ switch (c)
+ {
+ case 'n':
+ appendStringInfoChar(optargument, '\n');
+ break;
+
+ case 't':
+ appendStringInfoChar(optargument, '\t');
+ break;
+
+ case '\\':
+ appendStringInfoChar(optargument, '\\');
+ break;
+
+ default:
+ appendStringInfoChar(optargument, '\\');
+ appendStringInfoChar(optargument, c);
+ }
+ }
+ else
+ {
+ appendStringInfoChar(optargument, c);
+
+ if (c == '"')
+ {
+ if (*str == '"')
+ str++;
+ else
+ break;
+ }
+ }
+ }
+ }
+
+ /* check garbage after optarg, but ignore white-space */
+ while (isspace(*str))
+ str++;
+
+ /* at the end there should be EOL or comment symbol */
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "unexpected characters after an option's argument at line %d",
+ NULL, 0,
+ line->data,
+ *lineno);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Returns true, when long option is defined. Assign code,
+ * and check if option has required argument.
+ */
+static bool
+is_recognized_longopt(const struct option *longopts,
+ char *optname,
+ int optnamelen,
+ int *opt,
+ bool *has_arg)
+{
+ int i;
+
+ for (i = 0; longopts[i].name != NULL; i++)
+ {
+ if (strlen(longopts[i].name) == optnamelen &&
+ strncmp(optname, longopts[i].name, optnamelen) == 0)
+ {
+ *has_arg = longopts[i].has_arg == required_argument;
+
+ if (longopts[i].flag == NULL)
+ {
+ *opt = longopts[i].val;
+
+ return true;
+ }
+ else
+ {
+ *longopts[i].flag = longopts[i].val;
+ *opt = 0;
+
+ return true;
+ }
+ }
+ }
+
+ return false;
+}
+
+/*
+ * Returns true, when short option char is defined.
+ */
+static bool
+is_valid_shortopt(const char *optstring,
+ char optname,
+ bool *has_arg)
+{
+ while (*optstring != '\0')
+ {
+ if (*optstring != ':' && *optstring == optname)
+ {
+ *has_arg = optstring[1] == ':';
+ return true;
+ }
+
+ optstring++;
+ }
+
+ return false;
+}
+
+/*
+ * Read dumped object specification from file. Returns
+ * true, when processing was ok.
+ */
+static bool
+read_options_from_file(const char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+ StringInfoData optargument;
+
+ /* Ignore already processed files */
+ if (simple_string_list_member(&optsfilenames_processed,
+ filename))
+ {
+ pg_log_warning("the options file \"%s\" was processed already, skip this",
+ filename);
+ return true;
+ }
+
+ simple_string_list_append(&optsfilenames_processed, filename);
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ {
+ pg_log_error("could not open the input file \"%s\": %m",
+ filename);
+ return false;
+ }
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+ initStringInfo(&optargument);
+
+ while (pg_get_line_buf(fp, &line))
+ {
+ char *optname;
+ char *str = line.data;
+ int opt;
+ int optnamelen;
+ bool has_arg;
+
+ (void) pg_strip_crlf(str);
+
+ lineno += 1;
+
+ /* skip initial spaces */
+ while (isspace(*str))
+ str++;
+
+ /* Ignore empty lines or lines with hash symbol (comment) */
+ if (*str == '\0' || *str == '#')
+ continue;
+
+ if (*str++ != '-')
+ {
+ invalid_optfile_format(fp,
+ "non option arguments are not allowed in options file at line %d",
+ NULL, 0,
+ line.data,
+ lineno);
+ return false;
+ }
+
+ if (*str == '-')
+ {
+ /* process long option */
+ str++;
+ optname = str++;
+
+ while (!isspace(*str) && *str != '=' && *str != '\0')
+ str++;
+
+ optnamelen = str - optname;
+
+ if (is_recognized_longopt(longopts, optname, optnamelen,
+ &opt, &has_arg))
+ {
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (has_arg)
+ {
+ /* skip optional = */
+ if (*str == '=')
+ str++;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (!read_optarg(fp, filename, str, &line, &optargument,
+ optname, optnamelen, true, &lineno))
+ return false;
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "option '--%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ }
+ else
+ {
+ invalid_optfile_format(fp,
+ "unrecognized option '--%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ else
+ {
+ /* process short option */
+ optname = str++;
+ opt = *optname;
+
+ optnamelen = 1;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (is_valid_shortopt(optstring, opt, &has_arg))
+ {
+ if (has_arg)
+ {
+ if (!read_optarg(fp, filename, str, &line, &optargument,
+ optname, optnamelen, false, &lineno))
+ return false;
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "option '-%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ }
+ else
+ {
+ invalid_optfile_format(fp,
+ "invalid option '-%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+
+ /* nested options file reading */
+ if (opt == OPTIONS_FILE_OPTNUM)
+ {
+ if (!read_options_from_file(optargument.data, dopt, optstring,
+ longopts, progname))
+ {
+ fclose(fp);
+ return false;
+ }
+ }
+ else if (opt != 0 &&
+ !process_option(opt, optargument.data, dopt, progname))
+ {
+ fclose(fp);
+ return false;
+ }
+ }
+
+ pfree(line.data);
+ pfree(optargument.data);
+
+ if (ferror(fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m", filename);
+ return false;
+ }
+
+ if (fp != stdin)
+ fclose(fp);
+
+ return true;
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-10 15:47 Tomas Vondra <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Tomas Vondra @ 2021-07-10 15:47 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; +Cc: Tom Lane <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
Hi,
I started looking at the patch allowing to export just functions [1],
and I got pointed to this patch as an alternative approach (to adding a
separate filtering option for every possible object type).
I'm familiar with the customer that inspired Pavel to start working on
this, so I understand the use case he's trying to address - a flexible
way to filter (include/exclude) large number of objects.
IMHO it's a mistake to try to broaden the scope of the patch and require
implementing some universal pg_dump config file, particularly if it
requires "complex" structure or formats like JSON, TOML or whatever.
Maybe that's worth doing, but in my mind it's orthogonal to what this
patch aims (or aimed) to do - filtering objects using rules in a file,
not on the command line.
I believe it's much closer to .gitignore or rsync --filter than to a
full config file. Even if we end up implementing the pg_dump config
file, it'd be nice to keep the filter rules in a separate file and just
reference that file from the config file.
That also means I find it pointless to use an "advanced" format like
JSON or TOML - I think the format should be as simple as possible. Yes,
it has to support all valid identifiers, comments and so on. But I don't
quite see a point in using JSON or similar "full" format. If a simple
format is good enough for rsync or gitignore, why should we insist on
using something more complex?
OTOH I don't quite like the current approach of simply reading options
from a file, because that requires adding new command-line options for
each type of object we want to support. Which seems to contradict the
idea of "general filter" method as mentioned in [1].
So if it was up to me, I'd go back to the original format or something
close it. So something like this:
[+-] OBJECT_TYPE_PATTERN OBJECT_NAME_PATTERN
regards
[1] https://commitfest.postgresql.org/33/3051/
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-12 22:08 Daniel Gustafsson <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Daniel Gustafsson @ 2021-07-12 22:08 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 10 Jul 2021, at 17:47, Tomas Vondra <[email protected]> wrote:
> So if it was up to me, I'd go back to the original format or something close it. So something like this:
>
> [+-] OBJECT_TYPE_PATTERN OBJECT_NAME_PATTERN
That still leaves the parsing with quoting and escaping that needs to be done
less trivial and more bespoke than what meets the eye, no?
As mentioned upthread, I'm still hesitant to add a file format which deosn't
have any version information of sorts for distinguishing it from when the
inevitable "now wouldn't it be nice if we could do this too" patch which we all
know will come. The amount of selectivity switches we have for pg_dump is an
indication about just how much control users like, this will no doubt be
subject to the same.
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-12 22:47 Tomas Vondra <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Tomas Vondra @ 2021-07-12 22:47 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On 7/13/21 12:08 AM, Daniel Gustafsson wrote:
>> On 10 Jul 2021, at 17:47, Tomas Vondra <[email protected]> wrote:
>
>> So if it was up to me, I'd go back to the original format or something close it. So something like this:
>>
>> [+-] OBJECT_TYPE_PATTERN OBJECT_NAME_PATTERN
>
> That still leaves the parsing with quoting and escaping that needs to be done
> less trivial and more bespoke than what meets the eye, no?
>
Yes, it'd require proper escaping/quoting of the fields/identifiers etc.
> As mentioned upthread, I'm still hesitant to add a file format which deosn't
> have any version information of sorts for distinguishing it from when the
> inevitable "now wouldn't it be nice if we could do this too" patch which we all
> know will come. The amount of selectivity switches we have for pg_dump is an
> indication about just how much control users like, this will no doubt be
> subject to the same.
>
I'm not going to fight against some sort of versioning, but I think
keeping the scope as narrow as possible would make it unnecessary. That
is, let's stick to the original goal to allow passing filtering rules
that would not fit on the command-line, and maybe let's make it a bit
more flexible to support other object types etc.
IMHO the filtering rules are simple enough to not really need elaborate
versioning, and if a more advanced rule is proposed in the future it can
be supported in the existing format (extra field, ...).
Of course, maybe my imagination is not wild enough.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-12 22:59 Alvaro Herrera <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Alvaro Herrera @ 2021-07-12 22:59 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On 2021-Jul-13, Tomas Vondra wrote:
> I'm not going to fight against some sort of versioning, but I think keeping
> the scope as narrow as possible would make it unnecessary. That is, let's
> stick to the original goal to allow passing filtering rules that would not
> fit on the command-line, and maybe let's make it a bit more flexible to
> support other object types etc.
>
> IMHO the filtering rules are simple enough to not really need elaborate
> versioning, and if a more advanced rule is proposed in the future it can be
> supported in the existing format (extra field, ...).
I don't understand why is versioning needed for this file. Surely we
can just define some line-based grammar that's accepted by the current
pg_dump[1] and that would satisfy the current need as well as allowing
for extending the grammar in the future; even JSON or Windows-INI format
(ugh?) if that's necessary to tailor the output file in some other way
not covered by that.
[1] your proposal of "[+-] OBJTYPE OBJIDENT" plus empty lines allowed
plus lines starting with # are comments, seems plenty. Any line not
following that format would cause an error to be thrown.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-12 23:16 Tom Lane <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 2 replies; 181+ messages in thread
From: Tom Lane @ 2021-07-12 23:16 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Daniel Gustafsson <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Alvaro Herrera <[email protected]> writes:
> [1] your proposal of "[+-] OBJTYPE OBJIDENT" plus empty lines allowed
> plus lines starting with # are comments, seems plenty. Any line not
> following that format would cause an error to be thrown.
I'd like to see some kind of keyword on each line, so that we could extend
the command set by adding new keywords. As this stands, I fear we'd end
up using random punctuation characters in place of [+-], which seems
pretty horrid from a readability standpoint.
I think that this file format should be designed with an eye to allowing
every, or at least most, pg_dump options to be written in the file rather
than on the command line. I don't say we have to *implement* that right
now; but if the format spec is incapable of being extended to meet
requests like that one, I think we'll regret it. This line of thought
suggests that the initial commands ought to match the existing
include/exclude switches, at least approximately.
Hence I suggest
include table PATTERN
exclude table PATTERN
which ends up being the above but with words not [+-].
regards, tom lane
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-13 13:40 Stephen Frost <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Stephen Frost @ 2021-07-13 13:40 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Daniel Gustafsson <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Greetings,
* Tom Lane ([email protected]) wrote:
> Alvaro Herrera <[email protected]> writes:
> > [1] your proposal of "[+-] OBJTYPE OBJIDENT" plus empty lines allowed
> > plus lines starting with # are comments, seems plenty. Any line not
> > following that format would cause an error to be thrown.
>
> I'd like to see some kind of keyword on each line, so that we could extend
> the command set by adding new keywords. As this stands, I fear we'd end
> up using random punctuation characters in place of [+-], which seems
> pretty horrid from a readability standpoint.
I agree that it'd end up being bad with single characters.
> I think that this file format should be designed with an eye to allowing
> every, or at least most, pg_dump options to be written in the file rather
> than on the command line. I don't say we have to *implement* that right
> now; but if the format spec is incapable of being extended to meet
> requests like that one, I think we'll regret it. This line of thought
> suggests that the initial commands ought to match the existing
> include/exclude switches, at least approximately.
I agree that we want to have an actual config file that allows just
about every pg_dump option. I'm also fine with saying that we don't
have to implement that initially but the format should be one which can
be extended to allow that.
> Hence I suggest
>
> include table PATTERN
> exclude table PATTERN
>
> which ends up being the above but with words not [+-].
Which ends up inventing yet-another-file-format which people will end up
writing generators and parsers for. Which is exactly what I was arguing
we really should be trying to avoid doing.
I definitely feel that we should have a way to allow anything that can
be created as an object in the database to be explicitly included in the
file and that means whatever we do need to be able to handle objects
that have names that span multiple lines, etc. It's not clear how the
above would. As I recall, the proposed patch didn't have anything for
handling that, which was one of the issues I had with it and is why I
bring it up again.
Thanks,
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-13 16:14 Tomas Vondra <[email protected]>
parent: Stephen Frost <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Tomas Vondra @ 2021-07-13 16:14 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; Tom Lane <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Daniel Gustafsson <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On 7/13/21 3:40 PM, Stephen Frost wrote:
> Greetings,
>
> * Tom Lane ([email protected]) wrote:
>> Alvaro Herrera <[email protected]> writes:
>>> [1] your proposal of "[+-] OBJTYPE OBJIDENT" plus empty lines allowed
>>> plus lines starting with # are comments, seems plenty. Any line not
>>> following that format would cause an error to be thrown.
>>
>> I'd like to see some kind of keyword on each line, so that we could extend
>> the command set by adding new keywords. As this stands, I fear we'd end
>> up using random punctuation characters in place of [+-], which seems
>> pretty horrid from a readability standpoint.
>
> I agree that it'd end up being bad with single characters.
>
The [+-] format is based on what rsync does, so there's at least some
precedent for that, and IMHO it's fairly readable. I agree the rest of
the rule (object type, ...) may be a bit more verbose.
>> I think that this file format should be designed with an eye to allowing
>> every, or at least most, pg_dump options to be written in the file rather
>> than on the command line. I don't say we have to *implement* that right
>> now; but if the format spec is incapable of being extended to meet
>> requests like that one, I think we'll regret it. This line of thought
>> suggests that the initial commands ought to match the existing
>> include/exclude switches, at least approximately.
>
> I agree that we want to have an actual config file that allows just
> about every pg_dump option. I'm also fine with saying that we don't
> have to implement that initially but the format should be one which can
> be extended to allow that.
>
I understand the desire to have a config file that may contain all
pg_dump options, but I really don't see why we'd want to mix that with
the file containing filter rules.
I think those should be separate, one of the reasons being that I find
it desirable to be able to "include" the filter rules into different
pg_dump configs. That also means the format for the filter rules can be
much simpler.
It's also not clear to me whether the single-file approach would allow
filtering not supported by actual pg_dump option, for example.
>> Hence I suggest
>>
>> include table PATTERN
>> exclude table PATTERN
>>
>> which ends up being the above but with words not [+-].
>
Work for me.
> Which ends up inventing yet-another-file-format which people will end up
> writing generators and parsers for. Which is exactly what I was arguing
> we really should be trying to avoid doing.
>
People will have to write generators *in any case* because how else
would you use this? Unless we also provide tools to manipulate that file
(which seems rather futile), they'll have to do that. Even if we used
JSON/YAML/TOML/... they'd still need to deal with the semantics of the
file format.
FWIW I don't understand why would they need to write parsers. That's
something we'd need to do to process the file. I think the case when the
filter file needs to be modified is rather rare - it certainly is not
what the original use case Pavel tried to address needs. (I know that
customer and the filter would be generated and used for a single dump.)
My opinion is that the best solution (to make both generators and
parsers simple) is to keep the format itself as simple as possible.
Which is exactly why I'm arguing for only addressing the filtering, not
trying to invent a "universal" pg_dump config file format.
> I definitely feel that we should have a way to allow anything that can
> be created as an object in the database to be explicitly included in the
> file and that means whatever we do need to be able to handle objects
> that have names that span multiple lines, etc. It's not clear how the
> above would. As I recall, the proposed patch didn't have anything for
> handling that, which was one of the issues I had with it and is why I
> bring it up again.
>
I really don't understand why you think the current format can't do
escaping/quoting or handle names spanning multiple lines. The fact that
the original patch did not handle that correctly is a bug, but it does
not mean the format can't handle that.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-13 20:44 Daniel Gustafsson <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Daniel Gustafsson @ 2021-07-13 20:44 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Stephen Frost <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 13 Jul 2021, at 18:14, Tomas Vondra <[email protected]> wrote:
> FWIW I don't understand why would they need to write parsers.
It's quite common to write unit tests for VM recipes/playbooks wheen using
tools like Chef etc, parsing and checking the installed/generated files is part
of that. This would be one very real use case for writing a parser.
> I think the case when the filter file needs to be modified is rather rare - it certainly is not what the original use case Pavel tried to address needs. (I know that customer and the filter would be generated and used for a single dump.)
I'm not convinced that basing design decisions on a single customer reference
who only want to use the code once is helpful. I hear what you're saying, but
I think this will see more diverse use cases than what we can foresee here.
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-13 20:55 Daniel Gustafsson <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Daniel Gustafsson @ 2021-07-13 20:55 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 13 Jul 2021, at 00:59, Alvaro Herrera <[email protected]> wrote:
>
> On 2021-Jul-13, Tomas Vondra wrote:
>
>> I'm not going to fight against some sort of versioning, but I think keeping
>> the scope as narrow as possible would make it unnecessary. That is, let's
>> stick to the original goal to allow passing filtering rules that would not
>> fit on the command-line, and maybe let's make it a bit more flexible to
>> support other object types etc.
>>
>> IMHO the filtering rules are simple enough to not really need elaborate
>> versioning, and if a more advanced rule is proposed in the future it can be
>> supported in the existing format (extra field, ...).
>
> I don't understand why is versioning needed for this file. Surely we
> can just define some line-based grammar that's accepted by the current
> pg_dump[1] and that would satisfy the current need as well as allowing
> for extending the grammar in the future; even JSON or Windows-INI format
> (ugh?) if that's necessary to tailor the output file in some other way
> not covered by that.
I wasn't expressing myself very well; by "versioning" I mean a way to be able
to add to/change/fix the format and still be able to deterministically parse it
without having to resort to ugly heuristics and hacks. If that's achieved by
an explicit version number or if it's an inherit characteristic of the format
doesn't really matter (to me). My worry is that the very simple proposed
format might not fit that bill, but since I don't know what the future of the
feature might bring it's (mostly) a gut feeling.
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-13 20:55 Stephen Frost <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Stephen Frost @ 2021-07-13 20:55 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Tomas Vondra <[email protected]>; vignesh C <[email protected]>
Greetings,
On Tue, Jul 13, 2021 at 16:44 Daniel Gustafsson <[email protected]> wrote:
> > On 13 Jul 2021, at 18:14, Tomas Vondra <[email protected]>
> wrote:
>
> > FWIW I don't understand why would they need to write parsers.
>
> It's quite common to write unit tests for VM recipes/playbooks wheen using
> tools like Chef etc, parsing and checking the installed/generated files is
> part
> of that. This would be one very real use case for writing a parser.
Consider pgAdmin and the many other tools which essentially embed pg_dump
and pg_restore. There’s no shortage of use cases for a variety of tools to
be able to understand, read, parse, generate, rewrite, and probably do
more, with such a pg_dump/restore config file.
> I think the case when the filter file needs to be modified is rather rare
> - it certainly is not what the original use case Pavel tried to address
> needs. (I know that customer and the filter would be generated and used for
> a single dump.)
>
> I'm not convinced that basing design decisions on a single customer
> reference
> who only want to use the code once is helpful.
Agreed.
Thanks,
Stephen
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-13 21:57 Tomas Vondra <[email protected]>
parent: Stephen Frost <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Tomas Vondra @ 2021-07-13 21:57 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; Daniel Gustafsson <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; vignesh C <[email protected]>
On 7/13/21 10:55 PM, Stephen Frost wrote:
> Greetings,
>
> On Tue, Jul 13, 2021 at 16:44 Daniel Gustafsson <[email protected]
> <mailto:[email protected]>> wrote:
>
> > On 13 Jul 2021, at 18:14, Tomas Vondra
> <[email protected]
> <mailto:[email protected]>> wrote:
>
> > FWIW I don't understand why would they need to write parsers.
>
> It's quite common to write unit tests for VM recipes/playbooks wheen
> using
> tools like Chef etc, parsing and checking the installed/generated
> files is part
> of that. This would be one very real use case for writing a parser.
>
>
> Consider pgAdmin and the many other tools which essentially embed
> pg_dump and pg_restore. There’s no shortage of use cases for a variety
> of tools to be able to understand, read, parse, generate, rewrite, and
> probably do more, with such a pg_dump/restore config file.
>
Sure. Which is why I'm advocating for the simplest possible format (and
not expanding the scope of this patch beyond filtering), because that
makes this kind of processing simpler.
> > I think the case when the filter file needs to be modified is
> rather rare - it certainly is not what the original use case Pavel
> tried to address needs. (I know that customer and the filter would
> be generated and used for a single dump.)
>
> I'm not convinced that basing design decisions on a single customer
> reference
> who only want to use the code once is helpful.
>
>
> Agreed.
>
I wasn't really basing this on a single customer - that was merely an
example, of course. FWIW Justin Pryzby already stated having to use some
more complex format would likely mean they would not use the feature, so
that's another data point to consider.
FWIW I believe it's clear what my opinions on this topic are. Repeating
that seems a bit pointless, so I'll step aside and let this thread move
forward in whatever direction.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-13 22:32 Stephen Frost <[email protected]>
parent: Tomas Vondra <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Stephen Frost @ 2021-07-13 22:32 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Alvaro Herrera <[email protected]>; Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; vignesh C <[email protected]>
Greetings,
* Tomas Vondra ([email protected]) wrote:
> On 7/13/21 10:55 PM, Stephen Frost wrote:
> >On Tue, Jul 13, 2021 at 16:44 Daniel Gustafsson <[email protected]
> ><mailto:[email protected]>> wrote:
> >
> > > On 13 Jul 2021, at 18:14, Tomas Vondra
> > <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> > > FWIW I don't understand why would they need to write parsers.
> >
> > It's quite common to write unit tests for VM recipes/playbooks wheen
> > using
> > tools like Chef etc, parsing and checking the installed/generated
> > files is part
> > of that. This would be one very real use case for writing a parser.
>
> >Consider pgAdmin and the many other tools which essentially embed pg_dump
> >and pg_restore. There’s no shortage of use cases for a variety of tools
> >to be able to understand, read, parse, generate, rewrite, and probably do
> >more, with such a pg_dump/restore config file.
>
> Sure. Which is why I'm advocating for the simplest possible format (and not
> expanding the scope of this patch beyond filtering), because that makes this
> kind of processing simpler.
The simplest possible format isn't going to work with all the different
pg_dump options and it still isn't going to be 'simple' since it needs
to work with the flexibility that we have in what we support for object
names, and is still going to require people write a new parser and
generator for it instead of using something existing.
I don't know that the options that I suggested previously would
definitely work or not but they at least would allow other projects like
pgAdmin to leverage existing code for parsing and generating these
config files. I'm not completely against inventing something new, but
I'd really prefer that we at least try to make something existing work
first before inventing something new that everyone is going to have to
deal with. If we do invent a new thing for $reasons, then we should
really look at what exists today and try to design it properly instead
of just throwing something together and formally document it because
it's absolutely going to become a standard of sorts that people are
going to almost immediately write their own parsers/generators in
various languages for.
Thanks,
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-13 23:00 Alvaro Herrera <[email protected]>
parent: Stephen Frost <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Alvaro Herrera @ 2021-07-13 23:00 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Daniel Gustafsson <[email protected]>; Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; vignesh C <[email protected]>
On 2021-Jul-13, Stephen Frost wrote:
> The simplest possible format isn't going to work with all the different
> pg_dump options and it still isn't going to be 'simple' since it needs
> to work with the flexibility that we have in what we support for object
> names,
That's fine. If people want a mechanism that allows changing the other
pg_dump options that are not related to object filtering, they can
implement a configuration file for that.
> and is still going to require people write a new parser and
> generator for it instead of using something existing.
Sure. That's not part of this patch.
> I don't know that the options that I suggested previously would
> definitely work or not but they at least would allow other projects like
> pgAdmin to leverage existing code for parsing and generating these
> config files.
Keep in mind that this patch is not intended to help pgAdmin
specifically. It would be great if pgAdmin uses the functionality
implemented here, but if they decide not to, that's not terrible. They
have survived decades without a pg_dump configuration file; they still
can.
There are several votes in this thread for pg_dump to gain functionality
to filter objects based on a simple specification -- particularly one
that can be written using shell pipelines. This patch gives it.
> I'm not completely against inventing something new, but I'd really
> prefer that we at least try to make something existing work first
> before inventing something new that everyone is going to have to deal
> with.
That was discussed upthread and led nowhere.
--
Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/
"Nunca se desea ardientemente lo que solo se desea por razón" (F. Alexandre)
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-14 00:18 Stephen Frost <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Stephen Frost @ 2021-07-14 00:18 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Tomas Vondra <[email protected]>; Daniel Gustafsson <[email protected]>; Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; vignesh C <[email protected]>
Greetings,
* Alvaro Herrera ([email protected]) wrote:
> On 2021-Jul-13, Stephen Frost wrote:
> > The simplest possible format isn't going to work with all the different
> > pg_dump options and it still isn't going to be 'simple' since it needs
> > to work with the flexibility that we have in what we support for object
> > names,
>
> That's fine. If people want a mechanism that allows changing the other
> pg_dump options that are not related to object filtering, they can
> implement a configuration file for that.
It's been said multiple times that people *do* want that and that they
want it to all be part of this one file, and specifically that they
don't want to end up with a file structure that actively works against
allowing other options to be added to it.
> > I don't know that the options that I suggested previously would
> > definitely work or not but they at least would allow other projects like
> > pgAdmin to leverage existing code for parsing and generating these
> > config files.
>
> Keep in mind that this patch is not intended to help pgAdmin
> specifically. It would be great if pgAdmin uses the functionality
> implemented here, but if they decide not to, that's not terrible. They
> have survived decades without a pg_dump configuration file; they still
> can.
The adding of a config file for pg_dump should specifically be looking
at pgAdmin as the exact use-case for having such a capability.
> There are several votes in this thread for pg_dump to gain functionality
> to filter objects based on a simple specification -- particularly one
> that can be written using shell pipelines. This patch gives it.
And several votes for having a config file that supports, or at least
can support in the future, the various options which pg_dump supports-
and active voices against having a new file format that doesn't allow
for that.
> > I'm not completely against inventing something new, but I'd really
> > prefer that we at least try to make something existing work first
> > before inventing something new that everyone is going to have to deal
> > with.
>
> That was discussed upthread and led nowhere.
You're right- no one followed up on that. Instead, one group continues
to push for 'simple' and to just accept what's been proposed, while
another group counters that we should be looking at the broader design
question and work towards a solution which will work for us down the
road, and not just right now.
One thing remains clear- there's no consensus here.
Thanks,
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-14 05:00 Pavel Stehule <[email protected]>
parent: Stephen Frost <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-07-14 05:00 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Daniel Gustafsson <[email protected]>; Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; vignesh C <[email protected]>
Hi
> You're right- no one followed up on that. Instead, one group continues
> to push for 'simple' and to just accept what's been proposed, while
> another group counters that we should be looking at the broader design
> question and work towards a solution which will work for us down the
> road, and not just right now.
>
> One thing remains clear- there's no consensus here.
>
I think there should be some misunderstanding about the target of this
patch, and I am afraid so there cannot be consensus, because the people are
speaking about two very different features. And it is not possible to push
it to one thing. It cannot work I am afraid.
1. The main target of this patch is to solve the problem with the too large
command line of pg_dump when there are a lot of dumped objects. You need to
call pg_dump only once to ensure dump in one transaction. And sometimes it
is not possible to use wild characters effectively, because the state of
objects is in different databases. Enhancing the length of the command line
is not secure, and there are other production issues. In this case you need
a very simple format - just because you want to use pg_dump in pipe. This
format should be line oriented - and usually it will contain just "dump
this table, dump second table". Nothing else. Nobody will read this format,
nobody will edit this format. Because the main platform for this format is
probably the UNIX shell, the format should be simple. I really don't see
any joy in generating JSON and parsing JSON later. These data will be
processed locally. This is one purpose designed format, and it is not
designed for holding configuration. For this purpose the complex format has
not any advantage. There is not a problem with parsing JSON or other
formats on the pg_dump side, but it is pretty hard to generate valid JSON
from bash script. For a unix shell we need the most possible simple format.
Theoretically this format (this file) can hold any pg_dump's option, but
for usual streaming processing the only filter's options will be there.
Originally this feature had the name "filter file". There are a lot of
examples of successful filter's file formats in the UNIX world, and I think
so nobody doubts about sense and usability. Probably there is a consensus
so filter's files are not config files.
The format of the filter file can look like "+d tablename" or "include data
tablename". If we find a consensus so the filter file is a good thing, then
the format design and implementation is easy work. Isn't problem to invent
comment lines.
2. Is true, so there is only a small step from filter's file to option's
file. I rewrote this patch in this direction. The advantage is universality
- it can support any options without necessity to modify related code.
Still this format is not difficult for producers, and it is simple for
parsing. Now, the format should be defined by command line format: "-t
tablename" or "--table tablename" or "table tablename". There can be issues
related to different parsers in shell and in implemented code, but it can
be solved. Isn't problem to introduce comment lines. The big advantage is
simplicity of usage, simplicity of implementation - more the implementation
is generic.
3. But the option's file is just a small step to config file. I can imagine
somebody wanting to store typical configuration (and usual options) for
psql, pg_dump, pg_restore, pgAdmin, ... somewhere. The config files are
very different creatures than filter's files. Although they can be
generated, usually are edited and can be very complex. There can be shared
parts for all applications, and specific sections for psql, and specific
sections for every database. The config files can be brutally complex. The
simple text format is not good for this purpose. And some people prefer
YAML, some people hate this format. Other people prefer XML or JSON or
anything else. Sometimes the complexity of config files is too big, and
people prefer startup scripting.
Although there is an intersection between filter's files and config files,
I see very big differences in usage. Filter's files are usually temporal
and generated and non shared. Config file's are persistent, usually
manually modified and can be shared. The requests are different, and should
be different too. I don't propose any configuration's file related
features, and my proposal doesn't block the introduction of configuration's
file in any format in future. I think these features are very different,
and should be implemented differently. The filter's file or option's file
will be a pretty ugly config file, and config's file will be a pretty
impractical filter's file.
So can we talk about implementation of filter's file or option's file? And
can we talk about implementation config's files in separate topics? Without
it, I am afraid so there is no possibility of finding an agreement and
moving forward.
Regards
Pavel
> Thanks,
>
> Stephen
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-14 10:08 Tomas Vondra <[email protected]>
parent: Stephen Frost <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Tomas Vondra @ 2021-07-14 10:08 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; vignesh C <[email protected]>
On 7/14/21 2:18 AM, Stephen Frost wrote:
> Greetings,
>
> * Alvaro Herrera ([email protected]) wrote:
>> On 2021-Jul-13, Stephen Frost wrote:
>>> The simplest possible format isn't going to work with all the different
>>> pg_dump options and it still isn't going to be 'simple' since it needs
>>> to work with the flexibility that we have in what we support for object
>>> names,
>>
>> That's fine. If people want a mechanism that allows changing the other
>> pg_dump options that are not related to object filtering, they can
>> implement a configuration file for that.
>
> It's been said multiple times that people *do* want that and that they
> want it to all be part of this one file, and specifically that they
> don't want to end up with a file structure that actively works against
> allowing other options to be added to it.
>
I have no problem believing some people want to be able to specify
pg_dump parameters in a file, similarly to IMPDP/EXPDP parameter files
etc. That seems useful, but I doubt they considered the case with many
filter rules ... which is what "my people" want.
Not sure how keeping the filter rules in a separate file (which I assume
is what you mean by "file structure"), with a format tailored for filter
rules, works *actively* against adding options to the "main" config.
I'm not buying the argument that keeping some of the stuff in a separate
file is an issue - plenty of established tools do that, the concept of
"including" a config is not a radical new thing, and I don't expect we'd
have many options supported by a file.
In any case, I think user input is important, but ultimately it's up to
us to reconcile the conflicting requirements coming from various users
and come up with a reasonable compromise design.
>>> I don't know that the options that I suggested previously would
>>> definitely work or not but they at least would allow other projects like
>>> pgAdmin to leverage existing code for parsing and generating these
>>> config files.
>>
>> Keep in mind that this patch is not intended to help pgAdmin
>> specifically. It would be great if pgAdmin uses the functionality
>> implemented here, but if they decide not to, that's not terrible. They
>> have survived decades without a pg_dump configuration file; they still
>> can.
>
> The adding of a config file for pg_dump should specifically be looking
> at pgAdmin as the exact use-case for having such a capability.
>
>> There are several votes in this thread for pg_dump to gain functionality
>> to filter objects based on a simple specification -- particularly one
>> that can be written using shell pipelines. This patch gives it.
>
> And several votes for having a config file that supports, or at least
> can support in the future, the various options which pg_dump supports-
> and active voices against having a new file format that doesn't allow
> for that.
>
IMHO the whole "problem" here stems from the question whether there
should be a single universal pg_dump config file, containing everything
including the filter rules. I'm of the opinion it's better to keep the
filter rules separate, mainly because:
1) simplicity - Options (key/value) and filter rules (with more internal
structure) seem quite different, and mixing them in the same file will
just make the format more complex.
2) flexibility - Keeping the filter rules in a separate file makes it
easier to reuse the same set of rules with different pg_dump configs,
specified in (much smaller) config files.
So in principle, the "main" config could use e.g. TOML or whatever we
find most suitable for this type of key/value config file (or we could
just use the same format as for postgresql.conf et al). And the filter
rules could use something as simple as CSV (yes, I know it's not great,
but there's plenty of parsers, it handles multi-line strings etc.).
>>> I'm not completely against inventing something new, but I'd really
>>> prefer that we at least try to make something existing work first
>>> before inventing something new that everyone is going to have to deal
>>> with.
>>
>> That was discussed upthread and led nowhere.
>
> You're right- no one followed up on that. Instead, one group continues
> to push for 'simple' and to just accept what's been proposed, while
> another group counters that we should be looking at the broader design
> question and work towards a solution which will work for us down the
> road, and not just right now.
>
I have quite thick skin, but I have to admit I rather dislike how this
paints the people arguing for simplicity.
IMO simplicity is a perfectly legitimate (and desirable) design feature,
and simpler solutions often fare better in the long run. Yes, we need to
look at the broader design, no doubt about that.
> One thing remains clear- there's no consensus here.
>
True.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-28 04:18 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-07-28 04:18 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Tom Lane <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; Daniel Gustafsson <[email protected]>
st 12. 5. 2021 v 8:22 odesílatel Pavel Stehule <[email protected]>
napsal:
> Hi
>
> ne 11. 4. 2021 v 9:48 odesílatel Pavel Stehule <[email protected]>
> napsal:
>
>> Hi
>>
>> út 16. 2. 2021 v 20:32 odesílatel Pavel Stehule <[email protected]>
>> napsal:
>>
>>>
>>> Hi
>>>
>>> rebase
>>>
>>
>>
> rebase
>
>
>
>> fresh rebase
>>
>
fresh rebase
>> Regards
>>
>> Pavel
>>
>>
>>> Regards
>>>
>>> Pavel
>>>
>>
Attachments:
[text/x-patch] pg_dump-option-file-20210728.patch (24.1K, ../../CAFj8pRCVdGJsb+7hLm=KjeV=V5xiwat7DHB7aFxr9sbrz3YhJQ@mail.gmail.com/3-pg_dump-option-file-20210728.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7682226b99..94fc5aa6c2 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -1006,6 +1006,42 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--options-file=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read options from file (one option per line). Short or long options
+ are supported. If you use "-" as a filename, the filters are read
+ from stdin.
+ </para>
+
+ <para>
+ With the following options file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
+-t mytable1
+--include-foreign-data=some_foreign_server
+--exclude-table-data=mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The text after symbol <literal>#</literal> is ignored. This can
+ be used for comments, notes. Empty lines are ignored too.
+ </para>
+
+ <para>
+ The option <option>--options-file</option> can be used more times,
+ and the nesting is allowed. The options from options files are
+ processed first, other options from command line later. Any option
+ file is processed only one time. In next time the processing is
+ ignored.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--quote-all-identifiers</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 90ac445bcd..29f85ccfbf 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -55,10 +55,12 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -128,18 +130,34 @@ static SimpleOidList foreign_servers_include_oids = {NULL, NULL};
static SimpleStringList extension_include_patterns = {NULL, NULL};
static SimpleOidList extension_include_oids = {NULL, NULL};
+static SimpleStringList optsfilenames_processed = {NULL, NULL};
+
static const CatalogId nilCatalogId = {0, 0};
/* override for standard extra_float_digits setting */
static bool have_extra_float_digits = false;
static int extra_float_digits;
+static const char *filename = NULL;
+static const char *format = "p";
+static bool g_verbose = false;
+static const char *dumpencoding = NULL;
+static const char *dumpsnapshot = NULL;
+static char *use_role = NULL;
+static int numWorkers = 1;
+static int compressLevel = -1;
+
/*
* The default number of rows per INSERT when
* --inserts is specified without --rows-per-insert
*/
#define DUMP_DEFAULT_ROWS_PER_INSERT 1
+/*
+ * Option's code of "options-file" option
+ */
+#define OPTIONS_FILE_OPTNUM 12
+
/*
* Macro for producing quoted, schema-qualified name of a dumpable object.
*/
@@ -308,14 +326,212 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static bool read_options_from_file(const char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname);
+
+/*
+ * It assigns the values of options to related DumpOption fields or to
+ * some global values. Options file loading is not processed here.
+ */
+static bool
+process_option(int opt,
+ char *optargstr,
+ DumpOptions *dopt,
+ const char *progname)
+{
+ switch (opt)
+ {
+ case 'a': /* Dump data only */
+ dopt->dataOnly = true;
+ break;
+
+ case 'b': /* Dump blobs */
+ dopt->outputBlobs = true;
+ break;
+
+ case 'B': /* Don't dump blobs */
+ dopt->dontOutputBlobs = true;
+ break;
+
+ case 'c': /* clean (i.e., drop) schema prior to create */
+ dopt->outputClean = 1;
+ break;
+
+ case 'C': /* Create DB */
+ dopt->outputCreateDB = 1;
+ break;
+
+ case 'd': /* database name */
+ dopt->cparams.dbname = pg_strdup(optargstr);
+ break;
+
+ case 'e': /* include extension(s) */
+ simple_string_list_append(&extension_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'E': /* Dump encoding */
+ dumpencoding = pg_strdup(optargstr);
+ break;
+
+ case 'f':
+ filename = pg_strdup(optargstr);
+ break;
+
+ case 'F':
+ format = pg_strdup(optargstr);
+ break;
+
+ case 'h': /* server host */
+ dopt->cparams.pghost = pg_strdup(optargstr);
+ break;
+
+ case 'j': /* number of dump jobs */
+ if (!option_parse_int(optarg, "-j/--jobs", 1,
+ PG_MAX_JOBS,
+ &numWorkers))
+ exit_nicely(1);
+ break;
+
+ case 'n': /* include schema(s) */
+ simple_string_list_append(&schema_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'N': /* exclude schema(s) */
+ simple_string_list_append(&schema_exclude_patterns, optargstr);
+ break;
+
+ case 'O': /* Don't reconnect to match owner */
+ dopt->outputNoOwner = 1;
+ break;
+
+ case 'p': /* server port */
+ dopt->cparams.pgport = pg_strdup(optargstr);
+ break;
+
+ case 'R':
+ /* no-op, still accepted for backwards compatibility */
+ break;
+
+ case 's': /* dump schema only */
+ dopt->schemaOnly = true;
+ break;
+
+ case 'S': /* Username for superuser in plain text output */
+ dopt->outputSuperuser = pg_strdup(optargstr);
+ break;
+
+ case 't': /* include table(s) */
+ simple_string_list_append(&table_include_patterns, optargstr);
+ dopt->include_everything = false;
+ break;
+
+ case 'T': /* exclude table(s) */
+ simple_string_list_append(&table_exclude_patterns, optargstr);
+ break;
+
+ case 'U':
+ dopt->cparams.username = pg_strdup(optargstr);
+ break;
+
+ case 'v': /* verbose */
+ g_verbose = true;
+ pg_logging_increase_verbosity();
+ break;
+
+ case 'w':
+ dopt->cparams.promptPassword = TRI_NO;
+ break;
+
+ case 'W':
+ dopt->cparams.promptPassword = TRI_YES;
+ break;
+
+ case 'x': /* skip ACL dump */
+ dopt->aclsSkip = true;
+ break;
+
+ case 'Z': /* Compression Level */
+ if (!option_parse_int(optarg, "-Z/--compress", 0, 9,
+ &compressLevel))
+ exit_nicely(1);
+ break;
+
+ case 0:
+ /* This covers the long options. */
+ break;
+
+ case 2: /* lock-wait-timeout */
+ dopt->lockWaitTimeout = pg_strdup(optargstr);
+ break;
+
+ case 3: /* SET ROLE */
+ use_role = pg_strdup(optargstr);
+ break;
+
+ case 4: /* exclude table(s) data */
+ simple_string_list_append(&tabledata_exclude_patterns, optargstr);
+ break;
+
+ case 5: /* section */
+ set_dump_section(optargstr, &dopt->dumpSections);
+ break;
+
+ case 6: /* snapshot */
+ dumpsnapshot = pg_strdup(optargstr);
+ break;
+
+ case 7: /* no-sync */
+ dosync = false;
+ break;
+
+ case 8:
+ have_extra_float_digits = true;
+ if (!option_parse_int(optarg, "--extra-float-digits", -15, 3,
+ &extra_float_digits))
+ exit_nicely(1);
+ break;
+
+ case 9: /* inserts */
+
+ /*
+ * dump_inserts also stores --rows-per-insert, careful not to
+ * overwrite that.
+ */
+ if (dopt->dump_inserts == 0)
+ dopt->dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
+ break;
+
+ case 10: /* rows per insert */
+ if (!option_parse_int(optarg, "--rows-per-insert", 1, INT_MAX,
+ &dopt->dump_inserts))
+ exit_nicely(1);
+ break;
+
+ case 11: /* include foreign data */
+ simple_string_list_append(&foreign_servers_include_patterns,
+ optargstr);
+ break;
+
+ case OPTIONS_FILE_OPTNUM: /* reading options file */
+ break; /* should not be processed here ever */
+
+ default:
+ fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ return false;
+ }
+ return true;
+}
int
main(int argc, char **argv)
{
int c;
- const char *filename = NULL;
- const char *format = "p";
TableInfo *tblinfo;
int numTables;
DumpableObject **dobjs;
@@ -325,16 +541,9 @@ main(int argc, char **argv)
int optindex;
RestoreOptions *ropt;
Archive *fout; /* the script file */
- bool g_verbose = false;
- const char *dumpencoding = NULL;
- const char *dumpsnapshot = NULL;
- char *use_role = NULL;
- int numWorkers = 1;
- int compressLevel = -1;
int plainText = 0;
ArchiveFormat archiveFormat = archUnknown;
ArchiveMode archiveMode;
-
static DumpOptions dopt;
static struct option long_options[] = {
@@ -401,12 +610,15 @@ main(int argc, char **argv)
{"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1},
{"no-sync", no_argument, NULL, 7},
{"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1},
+ {"options-file", required_argument, NULL, OPTIONS_FILE_OPTNUM},
{"rows-per-insert", required_argument, NULL, 10},
{"include-foreign-data", required_argument, NULL, 11},
-
+ {"include-foreign-data-file", required_argument, NULL, 17},
{NULL, 0, NULL, 0}
};
+ const char *short_options = "abBcCd:e:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:";
+
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_dump"));
@@ -435,190 +647,33 @@ main(int argc, char **argv)
InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "abBcCd:e:E:f:F:h:j:n:N:Op:RsS:t:T:U:vwWxZ:",
+ /*
+ * We would to process options-file opts before other options.
+ * This implements higher priority of options from command line.
+ * Later processed options wins because overwrite result of
+ * early processed options.
+ */
+ while ((c = getopt_long(argc, argv, short_options,
long_options, &optindex)) != -1)
{
- switch (c)
+ if (c == OPTIONS_FILE_OPTNUM)
{
- case 'a': /* Dump data only */
- dopt.dataOnly = true;
- break;
-
- case 'b': /* Dump blobs */
- dopt.outputBlobs = true;
- break;
-
- case 'B': /* Don't dump blobs */
- dopt.dontOutputBlobs = true;
- break;
-
- case 'c': /* clean (i.e., drop) schema prior to create */
- dopt.outputClean = 1;
- break;
-
- case 'C': /* Create DB */
- dopt.outputCreateDB = 1;
- break;
-
- case 'd': /* database name */
- dopt.cparams.dbname = pg_strdup(optarg);
- break;
-
- case 'e': /* include extension(s) */
- simple_string_list_append(&extension_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'E': /* Dump encoding */
- dumpencoding = pg_strdup(optarg);
- break;
-
- case 'f':
- filename = pg_strdup(optarg);
- break;
-
- case 'F':
- format = pg_strdup(optarg);
- break;
-
- case 'h': /* server host */
- dopt.cparams.pghost = pg_strdup(optarg);
- break;
-
- case 'j': /* number of dump jobs */
- if (!option_parse_int(optarg, "-j/--jobs", 1,
- PG_MAX_JOBS,
- &numWorkers))
- exit_nicely(1);
- break;
-
- case 'n': /* include schema(s) */
- simple_string_list_append(&schema_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'N': /* exclude schema(s) */
- simple_string_list_append(&schema_exclude_patterns, optarg);
- break;
-
- case 'O': /* Don't reconnect to match owner */
- dopt.outputNoOwner = 1;
- break;
-
- case 'p': /* server port */
- dopt.cparams.pgport = pg_strdup(optarg);
- break;
-
- case 'R':
- /* no-op, still accepted for backwards compatibility */
- break;
-
- case 's': /* dump schema only */
- dopt.schemaOnly = true;
- break;
-
- case 'S': /* Username for superuser in plain text output */
- dopt.outputSuperuser = pg_strdup(optarg);
- break;
-
- case 't': /* include table(s) */
- simple_string_list_append(&table_include_patterns, optarg);
- dopt.include_everything = false;
- break;
-
- case 'T': /* exclude table(s) */
- simple_string_list_append(&table_exclude_patterns, optarg);
- break;
-
- case 'U':
- dopt.cparams.username = pg_strdup(optarg);
- break;
-
- case 'v': /* verbose */
- g_verbose = true;
- pg_logging_increase_verbosity();
- break;
-
- case 'w':
- dopt.cparams.promptPassword = TRI_NO;
- break;
-
- case 'W':
- dopt.cparams.promptPassword = TRI_YES;
- break;
-
- case 'x': /* skip ACL dump */
- dopt.aclsSkip = true;
- break;
-
- case 'Z': /* Compression Level */
- if (!option_parse_int(optarg, "-Z/--compress", 0, 9,
- &compressLevel))
- exit_nicely(1);
- break;
-
- case 0:
- /* This covers the long options. */
- break;
-
- case 2: /* lock-wait-timeout */
- dopt.lockWaitTimeout = pg_strdup(optarg);
- break;
-
- case 3: /* SET ROLE */
- use_role = pg_strdup(optarg);
- break;
-
- case 4: /* exclude table(s) data */
- simple_string_list_append(&tabledata_exclude_patterns, optarg);
- break;
-
- case 5: /* section */
- set_dump_section(optarg, &dopt.dumpSections);
- break;
-
- case 6: /* snapshot */
- dumpsnapshot = pg_strdup(optarg);
- break;
-
- case 7: /* no-sync */
- dosync = false;
- break;
-
- case 8:
- have_extra_float_digits = true;
- if (!option_parse_int(optarg, "--extra-float-digits", -15, 3,
- &extra_float_digits))
- exit_nicely(1);
- break;
-
- case 9: /* inserts */
-
- /*
- * dump_inserts also stores --rows-per-insert, careful not to
- * overwrite that.
- */
- if (dopt.dump_inserts == 0)
- dopt.dump_inserts = DUMP_DEFAULT_ROWS_PER_INSERT;
- break;
-
- case 10: /* rows per insert */
- if (!option_parse_int(optarg, "--rows-per-insert", 1, INT_MAX,
- &dopt.dump_inserts))
- exit_nicely(1);
- break;
-
- case 11: /* include foreign data */
- simple_string_list_append(&foreign_servers_include_patterns,
- optarg);
- break;
-
- default:
- fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
+ if (!read_options_from_file(optarg, &dopt, short_options,
+ long_options, progname))
exit_nicely(1);
}
}
+ /* reset getopt_long index */
+ optind = 1;
+
+ while ((c = getopt_long(argc, argv, short_options,
+ long_options, &optindex)) != -1)
+ {
+ if (!process_option(c, optarg, &dopt, progname))
+ exit_nicely(1);
+ }
+
/*
* Non-option argument specifies database name as long as it wasn't
* already specified with -d / --dbname
@@ -1053,6 +1108,7 @@ help(const char *progname)
printf(_(" --no-toast-compression do not dump TOAST compression methods\n"));
printf(_(" --no-unlogged-table-data do not dump unlogged table data\n"));
printf(_(" --on-conflict-do-nothing add ON CONFLICT DO NOTHING to INSERT commands\n"));
+ printf(_(" --options-file=FILENAME read options from options file\n"));
printf(_(" --quote-all-identifiers quote all identifiers, even if not key words\n"));
printf(_(" --rows-per-insert=NROWS number of rows per INSERT; implies --inserts\n"));
printf(_(" --section=SECTION dump named section (pre-data, data, or post-data)\n"));
@@ -18940,3 +18996,450 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and close input file
+ */
+static void
+invalid_optfile_format(FILE *fp,
+ char *message,
+ char *optname,
+ int optnamelen,
+ char *line,
+ int lineno)
+{
+ Assert(message);
+
+ if (optnamelen > 0)
+ {
+ Assert(optname);
+ pg_log_error(message, optnamelen, optname, lineno);
+ }
+ else
+ pg_log_error(message, lineno);
+
+ if (line)
+ fprintf(stderr, "LINE %d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+}
+
+/*
+ * Reads an option argument from file. Supports double-quoted
+ * strings. In this case multi-line strings are supported.
+ */
+static bool
+read_optarg(FILE *fp,
+ const char *filename,
+ char *str,
+ StringInfo line,
+ StringInfo optargument,
+ char *optname,
+ int optnamelen,
+ bool islongopt,
+ int *lineno)
+{
+ if (*str == '\0' || *str == '#')
+ {
+ if (islongopt)
+ invalid_optfile_format(fp,
+ "option '--%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ else
+ invalid_optfile_format(fp,
+ "option '-%.*s' requires an argument at line %d",
+ optname, optnamelen,
+ line->data,
+ *lineno);
+ return false;
+ }
+
+ resetStringInfo(optargument);
+
+ /* simple case */
+ if (*str != '"')
+ {
+ char *start = str;
+
+ /* read first white char */
+ while (*str != '\0' && *str != '#')
+ {
+ if (*str == ' ')
+ break;
+ str++;
+ }
+
+ appendBinaryStringInfo(optargument, start, str - start);
+ }
+ else
+ {
+ appendStringInfoChar(optargument, *str++);
+
+ while (1)
+ {
+ int c = *str++;
+
+ if (c == '\0')
+ {
+ /* multiline string, read next line */
+ if (!pg_get_line_buf(fp, line))
+ {
+ invalid_optfile_format(fp,
+ "unexpected end of line at line %d",
+ NULL, 0,
+ NULL,
+ *lineno);
+ return false;
+ }
+
+ if (ferror(fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m",
+ filename);
+ return false;
+ }
+
+ appendStringInfoChar(optargument, '\n');
+
+ str = line->data;
+ (void) pg_strip_crlf(str);
+ *lineno += 1;
+
+ c = *str++;
+ }
+
+ if (c == '\\')
+ {
+ c = *str++;
+
+ if (c == '\0')
+ {
+ invalid_optfile_format(fp,
+ "unexpected end of line at line %d",
+ NULL, 0,
+ NULL,
+ *lineno);
+ return false;
+ }
+
+ switch (c)
+ {
+ case 'n':
+ appendStringInfoChar(optargument, '\n');
+ break;
+
+ case 't':
+ appendStringInfoChar(optargument, '\t');
+ break;
+
+ case '\\':
+ appendStringInfoChar(optargument, '\\');
+ break;
+
+ default:
+ appendStringInfoChar(optargument, '\\');
+ appendStringInfoChar(optargument, c);
+ }
+ }
+ else
+ {
+ appendStringInfoChar(optargument, c);
+
+ if (c == '"')
+ {
+ if (*str == '"')
+ str++;
+ else
+ break;
+ }
+ }
+ }
+ }
+
+ /* check garbage after optarg, but ignore white-space */
+ while (isspace(*str))
+ str++;
+
+ /* at the end there should be EOL or comment symbol */
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "unexpected characters after an option's argument at line %d",
+ NULL, 0,
+ line->data,
+ *lineno);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Returns true, when long option is defined. Assign code,
+ * and check if option has required argument.
+ */
+static bool
+is_recognized_longopt(const struct option *longopts,
+ char *optname,
+ int optnamelen,
+ int *opt,
+ bool *has_arg)
+{
+ int i;
+
+ for (i = 0; longopts[i].name != NULL; i++)
+ {
+ if (strlen(longopts[i].name) == optnamelen &&
+ strncmp(optname, longopts[i].name, optnamelen) == 0)
+ {
+ *has_arg = longopts[i].has_arg == required_argument;
+
+ if (longopts[i].flag == NULL)
+ {
+ *opt = longopts[i].val;
+
+ return true;
+ }
+ else
+ {
+ *longopts[i].flag = longopts[i].val;
+ *opt = 0;
+
+ return true;
+ }
+ }
+ }
+
+ return false;
+}
+
+/*
+ * Returns true, when short option char is defined.
+ */
+static bool
+is_valid_shortopt(const char *optstring,
+ char optname,
+ bool *has_arg)
+{
+ while (*optstring != '\0')
+ {
+ if (*optstring != ':' && *optstring == optname)
+ {
+ *has_arg = optstring[1] == ':';
+ return true;
+ }
+
+ optstring++;
+ }
+
+ return false;
+}
+
+/*
+ * Read dumped object specification from file. Returns
+ * true, when processing was ok.
+ */
+static bool
+read_options_from_file(const char *filename,
+ DumpOptions *dopt,
+ const char *optstring,
+ const struct option *longopts,
+ const char *progname)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+ StringInfoData optargument;
+
+ /* Ignore already processed files */
+ if (simple_string_list_member(&optsfilenames_processed,
+ filename))
+ {
+ pg_log_warning("the options file \"%s\" was processed already, skip this",
+ filename);
+ return true;
+ }
+
+ simple_string_list_append(&optsfilenames_processed, filename);
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ {
+ pg_log_error("could not open the input file \"%s\": %m",
+ filename);
+ return false;
+ }
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+ initStringInfo(&optargument);
+
+ while (pg_get_line_buf(fp, &line))
+ {
+ char *optname;
+ char *str = line.data;
+ int opt;
+ int optnamelen;
+ bool has_arg;
+
+ (void) pg_strip_crlf(str);
+
+ lineno += 1;
+
+ /* skip initial spaces */
+ while (isspace(*str))
+ str++;
+
+ /* Ignore empty lines or lines with hash symbol (comment) */
+ if (*str == '\0' || *str == '#')
+ continue;
+
+ if (*str++ != '-')
+ {
+ invalid_optfile_format(fp,
+ "non option arguments are not allowed in options file at line %d",
+ NULL, 0,
+ line.data,
+ lineno);
+ return false;
+ }
+
+ if (*str == '-')
+ {
+ /* process long option */
+ str++;
+ optname = str++;
+
+ while (!isspace(*str) && *str != '=' && *str != '\0')
+ str++;
+
+ optnamelen = str - optname;
+
+ if (is_recognized_longopt(longopts, optname, optnamelen,
+ &opt, &has_arg))
+ {
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (has_arg)
+ {
+ /* skip optional = */
+ if (*str == '=')
+ str++;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (!read_optarg(fp, filename, str, &line, &optargument,
+ optname, optnamelen, true, &lineno))
+ return false;
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "option '--%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ }
+ else
+ {
+ invalid_optfile_format(fp,
+ "unrecognized option '--%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ else
+ {
+ /* process short option */
+ optname = str++;
+ opt = *optname;
+
+ optnamelen = 1;
+
+ /* skip optional spaces */
+ while (isspace(*str))
+ str++;
+
+ if (is_valid_shortopt(optstring, opt, &has_arg))
+ {
+ if (has_arg)
+ {
+ if (!read_optarg(fp, filename, str, &line, &optargument,
+ optname, optnamelen, false, &lineno))
+ return false;
+ }
+ else
+ {
+ if (*str != '\0' && *str != '#')
+ {
+ invalid_optfile_format(fp,
+ "option '-%.*s' doesn't allow an argument at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+ }
+ else
+ {
+ invalid_optfile_format(fp,
+ "invalid option '-%.*s' at line %d",
+ optname, optnamelen,
+ line.data,
+ lineno);
+ return false;
+ }
+ }
+
+ /* nested options file reading */
+ if (opt == OPTIONS_FILE_OPTNUM)
+ {
+ if (!read_options_from_file(optargument.data, dopt, optstring,
+ longopts, progname))
+ {
+ fclose(fp);
+ return false;
+ }
+ }
+ else if (opt != 0 &&
+ !process_option(opt, optargument.data, dopt, progname))
+ {
+ fclose(fp);
+ return false;
+ }
+ }
+
+ pfree(line.data);
+ pfree(optargument.data);
+
+ if (ferror(fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m", filename);
+ return false;
+ }
+
+ if (fp != stdin)
+ fclose(fp);
+
+ return true;
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-07-28 07:28 Pavel Stehule <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-07-28 07:28 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Daniel Gustafsson <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
út 13. 7. 2021 v 1:16 odesílatel Tom Lane <[email protected]> napsal:
> Alvaro Herrera <[email protected]> writes:
> > [1] your proposal of "[+-] OBJTYPE OBJIDENT" plus empty lines allowed
> > plus lines starting with # are comments, seems plenty. Any line not
> > following that format would cause an error to be thrown.
>
> I'd like to see some kind of keyword on each line, so that we could extend
> the command set by adding new keywords. As this stands, I fear we'd end
> up using random punctuation characters in place of [+-], which seems
> pretty horrid from a readability standpoint.
>
> I think that this file format should be designed with an eye to allowing
> every, or at least most, pg_dump options to be written in the file rather
> than on the command line. I don't say we have to *implement* that right
> now; but if the format spec is incapable of being extended to meet
> requests like that one, I think we'll regret it. This line of thought
> suggests that the initial commands ought to match the existing
> include/exclude switches, at least approximately.
>
> Hence I suggest
>
> include table PATTERN
> exclude table PATTERN
>
> which ends up being the above but with words not [+-].
>
Here is an updated implementation of filter's file, that implements syntax
proposed by you.
Regards
Pavel
> regards, tom lane
>
Attachments:
[text/x-patch] pg_dump-filter-20210728.patch (10.6K, ../../CAFj8pRBq9X=oSNbo+v93EwznkQ3JczEW6UsfQK37vx1ktW-2ZA@mail.gmail.com/3-pg_dump-filter-20210728.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7682226b99..d0459b385e 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -789,6 +789,56 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file.
+ If you use "-" as a filename, the filters are read from stdin.
+ The lines of this file must have the following format:
+<synopsis>
+(include|exclude)[table|schema|foreign_data|data] <replaceable class="parameter">objectname</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the object is to be included
+ or excluded, and the second keyword specifies the type of object
+ to be filtered:
+ <literal>table</literal> (table),
+ <literal>schema</literal> (schema),
+ <literal>foreign_data</literal> (foreign server),
+ <literal>data</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
+include table mytable1
+include foreign_data some_foreign_server
+exclude table mytable2
+</programlisting>
+ </para>
+
+ <para>
+ The lines starting with symbol <literal>#</literal> are ignored.
+ Previous white chars (spaces, tabs) are not allowed. These
+ lines can be used for comments, notes. Empty lines are ignored too.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 90ac445bcd..ba4c425ee6 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -55,10 +55,12 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -308,7 +310,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
-
+static void read_patterns_from_file(char *filename, DumpOptions *dopt);
int
main(int argc, char **argv)
@@ -380,6 +382,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -613,6 +616,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_patterns_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1038,6 +1045,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18940,3 +18949,281 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FILE *fp, char *filename, char *message, char *line, int lineno)
+{
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
+
+ if (fp != stdin)
+ fclose(fp);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Search keyword (can contains only ascii alphabetic characters) on line.
+ * Returns NULL, when the line is empty or first char is not alpha
+ */
+static const char *
+get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* skip initial white spaces */
+ while (isblank(*ptr))
+ ptr += 1;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr += 1;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+static bool
+is_keyword(const char *keyword, int size, const char *str)
+{
+ if (strlen(str) != size)
+ return false;
+
+ return pg_strncasecmp(keyword, str, size) == 0;
+}
+
+static bool
+isblank_line(const char *line)
+{
+ while (*line)
+ {
+ if (!isblank(*line++))
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_patterns_from_file(char *filename, DumpOptions *dopt)
+{
+ FILE *fp;
+ int lineno = 0;
+ StringInfoData line;
+ PQExpBuffer quoted_name = NULL;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fp = fopen(filename, "r");
+ if (!fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fp = stdin;
+
+ initStringInfo(&line);
+
+ while (pg_get_line_buf(fp, &line))
+ {
+ bool is_include;
+ char objecttype;
+ char *objectname;
+ char *str = line.data;
+ char *str_mark;
+ const char *keyword;
+ int size;
+
+ lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* ignore blank lines */
+ if (isblank_line(str))
+ continue;
+
+ /* when first char is hash, ignore whole line */
+ if (*str == '#')
+ continue;
+
+ keyword = get_keyword((const char **) &str, &size);
+
+ /* Now we expect sequence of two keywords */
+ if (keyword && is_keyword(keyword, size, "include"))
+ is_include = true;
+ else if (keyword && is_keyword(keyword, size, "exclude"))
+ is_include = false;
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "expected keyword \"include\" or \"exclude\"",
+ line.data,
+ lineno);
+
+ /*
+ * Save current position in parsed line. Can be used later
+ * in error message.
+ */
+ str_mark = str;
+
+ keyword = get_keyword((const char **) &str, &size);
+
+ if (keyword && is_keyword(keyword, size, "table"))
+ objecttype = 't';
+ else if (keyword && is_keyword(keyword, size, "schema"))
+ objecttype = 's';
+ else if (keyword && is_keyword(keyword, size, "foreign_data"))
+ objecttype = 'f';
+ else if (keyword && is_keyword(keyword, size, "data"))
+ objecttype = 'd';
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "expected keyword \"table\", \"schema\", \"foreign_data\" or \"data\"",
+ str_mark,
+ lineno);
+
+ objectname = str;
+
+ /* skip initial spaces */
+ while (isspace(*objectname))
+ objectname++;
+
+ if (*objectname == '\0')
+ exit_invalid_filter_format(fp,
+ filename,
+ "missing object name",
+ str,
+ lineno);
+
+ if (*objectname == '"')
+ {
+ PQExpBuffer quoted_name;
+ char *ptr = objectname + 1;
+
+ quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+
+ while (1)
+ {
+ if (*ptr == '\0')
+ {
+ if (!pg_get_line_buf(fp, &line))
+ exit_invalid_filter_format(fp,
+ filename,
+ "unexpected end of file",
+ "",
+ lineno);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ ptr = line.data;
+ lineno += 1;
+ }
+
+ appendPQExpBufferChar(quoted_name, *ptr);
+ if (*ptr++ == '"')
+ {
+ if (*ptr == '"')
+ appendPQExpBufferChar(quoted_name, *ptr++);
+ else
+ break;
+ }
+ }
+
+ /* check garbage after identifier */
+ if (!isblank_line(ptr))
+ exit_invalid_filter_format(fp,
+ filename,
+ "unexpected chars after object name",
+ ptr,
+ lineno);
+
+ objectname = quoted_name->data;
+ }
+
+ if (objecttype == 't')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 's')
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objectname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'd')
+ {
+ if (is_include)
+ exit_invalid_filter_format(fp,
+ filename,
+ "include filter is not supported for this type of object",
+ str,
+ lineno);
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objectname);
+ }
+ else if (objecttype == 'f')
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objectname);
+ else
+ exit_invalid_filter_format(fp,
+ filename,
+ "exclude filter is not supported for this type of object",
+ str,
+ lineno);
+ }
+
+ if (quoted_name)
+ {
+ destroyPQExpBuffer(quoted_name);
+ quoted_name = NULL;
+ }
+ }
+
+ pfree(line.data);
+
+ if (ferror(fp))
+ fatal("could not read from file \"%s\": %m", filename);
+
+ if (fp != stdin)
+ fclose(fp);
+}
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-13 13:01 Daniel Gustafsson <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Daniel Gustafsson @ 2021-09-13 13:01 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 28 Jul 2021, at 09:28, Pavel Stehule <[email protected]> wrote:
> út 13. 7. 2021 v 1:16 odesílatel Tom Lane <[email protected] <mailto:[email protected]>> napsal:
> Hence I suggest
>
> include table PATTERN
> exclude table PATTERN
>
> which ends up being the above but with words not [+-].
One issue with this syntax is that the include keyword can be quite misleading
as it's semantic interpretion of "include table t" can be different from
"--table=t". The former is less clear about the fact that it means "exclude
all other tables than " then the latter. It can be solved with documentation,
but I think that needs be to be made clearer.
> Here is an updated implementation of filter's file, that implements syntax proposed by you.
While it's not the format I would prefer, it does allow for most (all?) use
cases expressed in this thread with ample armtwisting applied so let's go ahead
from this point and see if we can agree on it (or a version of it).
A few notes on the patch after a first pass over it:
+(include|exclude)[table|schema|foreign_data|data] <replaceable class="parameter">objectname</replaceable>
Lacks whitespace between keyword and object type. Also, since these are
mandatory parameters, shouldn't they be within '{..}' ?
+ /* skip initial white spaces */
+ while (isblank(*ptr))
+ ptr += 1;
We don't trust isblank() as of 3fd5faed5 due to portability concerns, this
should probably use a version of the pg_isblank() we already have (and possibly
move that to src/common/string.c as there now are more consumers).
+static bool
+isblank_line(const char *line)
This could be replaced with a single call to strspn() as we already do for
parsing the TOC file.
+ /* when first char is hash, ignore whole line */
+ if (*str == '#')
+ continue;
I think we should strip leading whitespace before this to allow commentlines to
start with whitespace, it's easy enough and will make life easier for users.
+ pg_log_error("invalid format of filter file \"%s\": %s",
+ filename,
+ message);
+
+ fprintf(stderr, "%d: %s\n", lineno, line);
Can't we just include the lineno in the error logging and skip dumping the
offending line? Fast-forwarding the pointer to print the offending part is
less useful than a context marker, and in some cases suboptimal. With this
coding, if a pattern is omitted for example the below error message is given:
pg_dump: error: invalid format of filter file "filter.txt": missing object name
1:
The errormessage and the linenumber in the file should be enough for the user
to figure out what to fix.
+ if (keyword && is_keyword(keyword, size, "table"))
+ objecttype = 't';
Should this use an enum, or at least a struct translation the literal keyword
to the internal representation? Magic constants without explicit connection to
their token counterparts can easily be cause of bugs.
If I create a table called "a\nb" and try to dump it I get an error in parsing
the file. Isn't this supposed to work?
$ cat filter.txt
include table "a
b"
$ ./bin/pg_dump --filter=filter.txt
pg_dump: error: invalid format of filter file "filter.txt": unexpected chars after object name
2:
Did you consider implementing this in Bison to abstract some of the messier
parsing logic?
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-13 13:11 Justin Pryzby <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Justin Pryzby @ 2021-09-13 13:11 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Daniel Gustafsson <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Wed, Jul 28, 2021 at 09:28:17AM +0200, Pavel Stehule wrote:
> Here is an updated implementation of filter's file, that implements syntax
> proposed by you.
Thanks.
If there's any traction for this approach. I have some comments for the next
revision,
> +++ b/doc/src/sgml/ref/pg_dump.sgml
> @@ -789,6 +789,56 @@ PostgreSQL documentation
> </listitem>
> </varlistentry>
>
> + <varlistentry>
> + <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
> + <listitem>
> + <para>
> + Read objects filters from the specified file.
> + If you use "-" as a filename, the filters are read from stdin.
Say 'Specify "-" to read from stdin'
> + The lines starting with symbol <literal>#</literal> are ignored.
Remove "The" and "symbol"
> + Previous white chars (spaces, tabs) are not allowed. These
Preceding whitespace characters...
But actually, they are allowed? But if it needs to be explained, maybe they
shouldn't be - I don't see the utility of it.
> +static bool
> +isblank_line(const char *line)
> +{
> + while (*line)
> + {
> + if (!isblank(*line++))
> + return false;
> + }
> +
> + return true;
> +}
I don't think this requires nor justifies having a separate function.
Either don't support blank lines, or use get_keyword() with size==0 for that ?
> + /* Now we expect sequence of two keywords */
> + if (keyword && is_keyword(keyword, size, "include"))
> + is_include = true;
> + else if (keyword && is_keyword(keyword, size, "exclude"))
> + is_include = false;
> + else
I think this should first check "if keyword == NULL".
That could give a more specific error message like "no keyword found",
> + exit_invalid_filter_format(fp,
> + filename,
> + "expected keyword \"include\" or \"exclude\"",
> + line.data,
> + lineno);
..and then this one can say "invalid keyword".
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-15 17:31 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-09-15 17:31 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
po 13. 9. 2021 v 15:01 odesílatel Daniel Gustafsson <[email protected]>
napsal:
> > On 28 Jul 2021, at 09:28, Pavel Stehule <[email protected]> wrote:
> > út 13. 7. 2021 v 1:16 odesílatel Tom Lane <[email protected] <mailto:
> [email protected]>> napsal:
>
> > Hence I suggest
> >
> > include table PATTERN
> > exclude table PATTERN
> >
> > which ends up being the above but with words not [+-].
>
> One issue with this syntax is that the include keyword can be quite
> misleading
> as it's semantic interpretion of "include table t" can be different from
> "--table=t". The former is less clear about the fact that it means
> "exclude
> all other tables than " then the latter. It can be solved with
> documentation,
> but I think that needs be to be made clearer.
>
I invite any documentation enhancing and fixing
>
> > Here is an updated implementation of filter's file, that implements
> syntax proposed by you.
>
> While it's not the format I would prefer, it does allow for most (all?) use
> cases expressed in this thread with ample armtwisting applied so let's go
> ahead
> from this point and see if we can agree on it (or a version of it).
>
> A few notes on the patch after a first pass over it:
>
> +(include|exclude)[table|schema|foreign_data|data] <replaceable
> class="parameter">objectname</replaceable>
> Lacks whitespace between keyword and object type. Also, since these are
> mandatory parameters, shouldn't they be within '{..}' ?
>
> yes, fixed
>
> + /* skip initial white spaces */
> + while (isblank(*ptr))
> + ptr += 1;
> We don't trust isblank() as of 3fd5faed5 due to portability concerns, this
> should probably use a version of the pg_isblank() we already have (and
> possibly
> move that to src/common/string.c as there now are more consumers).
>
>
I rewrote this part, and I don't use function isblank ever
>
> +static bool
> +isblank_line(const char *line)
> This could be replaced with a single call to strspn() as we already do for
> parsing the TOC file.
>
>
> + /* when first char is hash, ignore whole line */
> + if (*str == '#')
> + continue;
> I think we should strip leading whitespace before this to allow
> commentlines to
> start with whitespace, it's easy enough and will make life easier for
> users.
>
now, the comments can be used as first non blank char or after filter
>
>
> + pg_log_error("invalid format of filter file \"%s\": %s",
> + filename,
> + message);
> +
> + fprintf(stderr, "%d: %s\n", lineno, line);
> Can't we just include the lineno in the error logging and skip dumping the
> offending line? Fast-forwarding the pointer to print the offending part is
> less useful than a context marker, and in some cases suboptimal. With this
> coding, if a pattern is omitted for example the below error message is
> given:
>
>
pg_dump: error: invalid format of filter file "filter.txt": missing
> object name
> 1:
>
> The errormessage and the linenumber in the file should be enough for the
> user
> to figure out what to fix.
>
I did it like you proposed, but still, I think the content can be useful.
More times you read dynamically generated files, or you read data from
stdin, and in complex environments it can be hard regenerate new content
for debugging.
>
> + if (keyword && is_keyword(keyword, size, "table"))
> + objecttype = 't';
> Should this use an enum, or at least a struct translation the literal
> keyword
> to the internal representation? Magic constants without explicit
> connection to
> their token counterparts can easily be cause of bugs.
>
>
fixed
> If I create a table called "a\nb" and try to dump it I get an error in
> parsing
> the file. Isn't this supposed to work?
> $ cat filter.txt
> include table "a
> b"
> $ ./bin/pg_dump --filter=filter.txt
> pg_dump: error: invalid format of filter file "filter.txt": unexpected
> chars after object name
> 2:
probably there was some issue, because it should work. I tested a new
version and this is tested in new regress tests. Please, check
>
> Did you consider implementing this in Bison to abstract some of the messier
> parsing logic?
>
Initially not, but now, when I am thinking about it, I don't think so Bison
helps. The syntax of the filter file is nicely linear. Now, the code of the
parser is a little bit larger than minimalistic, but it is due to nicer
error's messages. The raw implementation in Bison raised just "syntax
error" and positions. I did code refactoring, and now the scanning, parsing
and processing are divided into separated routines. Parsing related code
has 90 lines. In this case, I don't think using a parser grammar file can
carry any benefit. grammar is more readable, sure, but we need to include
bison, we need to handle errors, and if we want to raise more helpful
errors than just "syntax error", then the code will be longer.
please, check attached patch
Regards
Pavel
> --
> Daniel Gustafsson https://vmware.com/
>
>
Attachments:
[text/x-patch] pg_dump-filteropt-20210915.patch (17.1K, ../../CAFj8pRAZbCvM3v8jp5wkPpZHJvAopWpbF=+v+84Pns=qWtvEcg@mail.gmail.com/3-pg_dump-filteropt-20210915.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7682226b99..1b74c0eadd 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -789,6 +789,55 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file. Specify "-" to read from
+ stdin. Lines of this file must have the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the object is to be included
+ or excluded, and the second keyword specifies the type of object
+ to be filtered:
+ <literal>table</literal> (table),
+ <literal>schema</literal> (schema),
+ <literal>foreign_data</literal> (foreign server),
+ <literal>data</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
+include table mytable1
+include foreign_data some_foreign_server
+exclude table mytable2
+</programlisting>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are ignored. The comment
+ (started by <literal>#</literal>) can be placed after filter too.
+ Empty lines are ignored too.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a485fb2d07..844492c64f 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -55,10 +55,12 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -308,7 +310,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
-
+static void read_filters_from_file(char *filename, DumpOptions *dopt);
int
main(int argc, char **argv)
@@ -380,6 +382,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -613,6 +616,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_filters_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1038,6 +1045,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18979,3 +18988,343 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+typedef struct
+{
+ FILE *fp;
+ char *filename;
+ int lineno;
+} FilterStateData;
+
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_DATA
+} FilterObjectType;
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+
+ fclose(fstate->fp);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Search keyword (can contains only ascii alphabetic characters) on line.
+ * Returns NULL, when the line is empty or first char is not alpha
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* skip initial white spaces */
+ while (isspace(*ptr))
+ ptr += 1;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr += 1;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+static bool
+filter_is_keyword(const char *keyword, int size, const char *str)
+{
+ if (strlen(str) != size)
+ return false;
+
+ return pg_strncasecmp(keyword, str, size) == 0;
+}
+
+/*
+ * Sets objname to string with object identifier. The line variable holds string
+ * of last line with object identifier (object name). Returns pointer to first char
+ * after last char of object name.
+ */
+static char *
+filter_get_object_name(FilterStateData *fstate,
+ StringInfo line,
+ char *str,
+ char **objname)
+{
+ /* skip white spaces */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ exit_invalid_filter_format(fstate, "missing object name");
+
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ if (!pg_get_line_buf(fstate->fp, line))
+ {
+ if (ferror(fstate->fp))
+ fatal("could not read from file \"%s\": %m", fstate->filename);
+
+ exit_invalid_filter_format(fstate,"unexpected end of file");
+ }
+
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ str = line->data;
+ fstate->lineno += 1;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ str++;
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = quoted_name->data;
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* simple variant, read to end or to first space */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = strndup(startptr, str - startptr);
+ }
+
+ return str;
+}
+
+/*
+ * Returns true, when one filter item was successfully read and parsed.
+ * When object name contains \n chars, then more than one line from input
+ * file can be processed. Returns false when EOF. Run exit on error.
+ */
+static bool
+read_filter_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ StringInfoData line;
+
+ initStringInfo(&line);
+
+ if (pg_get_line_buf(fstate->fp, &line))
+ {
+ char *str = line.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * skip empty lines or lines when first noblank char is hash (comment)
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no keyword found (expected \"include\" or \"exclude\")");
+
+ /* Now we expect sequence of two keywords */
+ if (filter_is_keyword(keyword, size, "include"))
+ *is_include = true;
+ else if (filter_is_keyword(keyword, size, "exclude"))
+ *is_include = false;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid keyword (expected \"include\" or \"exclude\")");
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no keyword found (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ if (filter_is_keyword(keyword, size, "table"))
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (filter_is_keyword(keyword, size, "schema"))
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (filter_is_keyword(keyword, size, "foreign_data"))
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (filter_is_keyword(keyword, size, "data"))
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid keyword (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ str = filter_get_object_name(fstate, &line, str, objname);
+
+ /*
+ * check possible content after object identifier.
+ * Allow comment started by hash.
+ */
+ while (isspace(*str))
+ str++;
+
+ if (*str != '\0' && *str != '#')
+ exit_invalid_filter_format(fstate,
+ "unexpected chars after object name");
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ free(line.data);
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_fatal("could not read from file \"%s\": %m", fstate->filename);
+
+ if (fstate->fp != stdin)
+ fclose(fstate->fp);
+
+ exit_nicely(-1);
+ }
+
+ free(line.data);
+
+ return false;
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_filters_from_file(char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ fstate.filename = filename;
+ fstate.lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate.fp = fopen(filename, "r");
+ if (!fstate.fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fstate.fp = stdin;
+
+ while (read_filter_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ exit_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+
+ free(objname);
+ }
+
+ if (fstate.fp != stdin)
+ fclose(fstate.fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..505614f145
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,151 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 24;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = PostgresNode->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid keyword/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid keyword/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/missing object name/,
+ "broken format check");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-15 18:30 Pavel Stehule <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-09-15 18:30 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Daniel Gustafsson <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
po 13. 9. 2021 v 15:11 odesílatel Justin Pryzby <[email protected]>
napsal:
> On Wed, Jul 28, 2021 at 09:28:17AM +0200, Pavel Stehule wrote:
> > Here is an updated implementation of filter's file, that implements
> syntax
> > proposed by you.
>
> Thanks.
>
> If there's any traction for this approach. I have some comments for the
> next
> revision,
>
> > +++ b/doc/src/sgml/ref/pg_dump.sgml
> > @@ -789,6 +789,56 @@ PostgreSQL documentation
> > </listitem>
> > </varlistentry>
> >
> > + <varlistentry>
> > + <term><option>--filter=<replaceable
> class="parameter">filename</replaceable></option></term>
> > + <listitem>
> > + <para>
> > + Read objects filters from the specified file.
> > + If you use "-" as a filename, the filters are read from stdin.
>
> Say 'Specify "-" to read from stdin'
>
> > + The lines starting with symbol <literal>#</literal> are ignored.
>
> Remove "The" and "symbol"
>
> > + Previous white chars (spaces, tabs) are not allowed. These
>
> Preceding whitespace characters...
>
> But actually, they are allowed? But if it needs to be explained, maybe
> they
> shouldn't be - I don't see the utility of it.
>
> > +static bool
> > +isblank_line(const char *line)
> > +{
> > + while (*line)
> > + {
> > + if (!isblank(*line++))
> > + return false;
> > + }
> > +
> > + return true;
> > +}
>
> I don't think this requires nor justifies having a separate function.
> Either don't support blank lines, or use get_keyword() with size==0 for
> that ?
>
> > + /* Now we expect sequence of two keywords */
> > + if (keyword && is_keyword(keyword, size, "include"))
> > + is_include = true;
> > + else if (keyword && is_keyword(keyword, size, "exclude"))
> > + is_include = false;
> > + else
>
> I think this should first check "if keyword == NULL".
> That could give a more specific error message like "no keyword found",
>
> > + exit_invalid_filter_format(fp,
> > +
> filename,
> > +
> "expected keyword \"include\" or \"exclude\"",
> > +
> line.data,
> > +
> lineno);
>
> ..and then this one can say "invalid keyword".
>
I fixed (I hope) mentioned issues. Please check last patch
Regards
Pavel
> --
> Justin
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-16 04:45 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-09-16 04:45 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
In yesterday's patch I used strndup, which is not available on win. I am
sending update when I used pnstrdup instead.
Regards
Pavel
Attachments:
[text/x-patch] pg_dump-filteropt-20210916.patch (17.2K, ../../CAFj8pRAy+xfSj=tz_=RRx0hsn_x5dMpvTz2RJ9OzCG9GTdPxnA@mail.gmail.com/3-pg_dump-filteropt-20210916.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7682226b99..1b74c0eadd 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -789,6 +789,55 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file. Specify "-" to read from
+ stdin. Lines of this file must have the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the object is to be included
+ or excluded, and the second keyword specifies the type of object
+ to be filtered:
+ <literal>table</literal> (table),
+ <literal>schema</literal> (schema),
+ <literal>foreign_data</literal> (foreign server),
+ <literal>data</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
+include table mytable1
+include foreign_data some_foreign_server
+exclude table mytable2
+</programlisting>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are ignored. The comment
+ (started by <literal>#</literal>) can be placed after filter too.
+ Empty lines are ignored too.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a485fb2d07..81533eb144 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -55,10 +55,12 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -308,7 +310,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
-
+static void read_filters_from_file(char *filename, DumpOptions *dopt);
int
main(int argc, char **argv)
@@ -380,6 +382,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -613,6 +616,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_filters_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1038,6 +1045,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18979,3 +18988,344 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+typedef struct
+{
+ FILE *fp;
+ char *filename;
+ int lineno;
+} FilterStateData;
+
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_DATA
+} FilterObjectType;
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+
+ fclose(fstate->fp);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Search keyword (can contains only ascii alphabetic characters) on line.
+ * Returns NULL, when the line is empty or first char is not alpha
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* skip initial white spaces */
+ while (isspace(*ptr))
+ ptr += 1;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr += 1;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+static bool
+filter_is_keyword(const char *keyword, int size, const char *str)
+{
+ if (strlen(str) != size)
+ return false;
+
+ return pg_strncasecmp(keyword, str, size) == 0;
+}
+
+/*
+ * Sets objname to string with object identifier. The line variable holds string
+ * of last line with object identifier (object name). Returns pointer to first char
+ * after last char of object name.
+ */
+static char *
+filter_get_object_name(FilterStateData *fstate,
+ StringInfo line,
+ char *str,
+ char **objname)
+{
+ /* skip white spaces */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ exit_invalid_filter_format(fstate, "missing object name");
+
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ if (!pg_get_line_buf(fstate->fp, line))
+ {
+ if (ferror(fstate->fp))
+ fatal("could not read from file \"%s\": %m", fstate->filename);
+
+ exit_invalid_filter_format(fstate,"unexpected end of file");
+ }
+
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ str = line->data;
+ fstate->lineno += 1;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ str++;
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = quoted_name->data;
+ }
+ else
+ {
+ char *startptr = str++;
+ size_t objnamelen;
+
+ /* simple variant, read to end or to first space */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ return str;
+}
+
+/*
+ * Returns true, when one filter item was successfully read and parsed.
+ * When object name contains \n chars, then more than one line from input
+ * file can be processed. Returns false when EOF. Run exit on error.
+ */
+static bool
+read_filter_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ StringInfoData line;
+
+ initStringInfo(&line);
+
+ if (pg_get_line_buf(fstate->fp, &line))
+ {
+ char *str = line.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * skip empty lines or lines when first noblank char is hash (comment)
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no keyword found (expected \"include\" or \"exclude\")");
+
+ /* Now we expect sequence of two keywords */
+ if (filter_is_keyword(keyword, size, "include"))
+ *is_include = true;
+ else if (filter_is_keyword(keyword, size, "exclude"))
+ *is_include = false;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid keyword (expected \"include\" or \"exclude\")");
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no keyword found (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ if (filter_is_keyword(keyword, size, "table"))
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (filter_is_keyword(keyword, size, "schema"))
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (filter_is_keyword(keyword, size, "foreign_data"))
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (filter_is_keyword(keyword, size, "data"))
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid keyword (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ str = filter_get_object_name(fstate, &line, str, objname);
+
+ /*
+ * check possible content after object identifier.
+ * Allow comment started by hash.
+ */
+ while (isspace(*str))
+ str++;
+
+ if (*str != '\0' && *str != '#')
+ exit_invalid_filter_format(fstate,
+ "unexpected chars after object name");
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ free(line.data);
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_fatal("could not read from file \"%s\": %m", fstate->filename);
+
+ if (fstate->fp != stdin)
+ fclose(fstate->fp);
+
+ exit_nicely(-1);
+ }
+
+ free(line.data);
+
+ return false;
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_filters_from_file(char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ fstate.filename = filename;
+ fstate.lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate.fp = fopen(filename, "r");
+ if (!fstate.fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fstate.fp = stdin;
+
+ while (read_filter_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ exit_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+
+ free(objname);
+ }
+
+ if (fstate.fp != stdin)
+ fclose(fstate.fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..505614f145
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,151 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 24;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = PostgresNode->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid keyword/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid keyword/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/missing object name/,
+ "broken format check");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-17 11:18 Daniel Gustafsson <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Daniel Gustafsson @ 2021-09-17 11:18 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
As there have been a lot of differing opinions raised in this thread, I re-read
it and tried to summarize the discussion so far to try and figure out where we
agree and on what (and disagree) before we get deep into the technicalities wrt
the current patch. If anyone feel I've misrepresented them below then I
sincerely do apologize. If I missed a relevant viewpoint I also apologize,
I've tried to objectively represent the thread.
I proposed JSON in [0] which is where the format discussion to some extent
started, Justin and Pavel had up until that point discussed the format by
refining the original proposal.
In [1] Surafel Temesgen brought up --exclude-database from pg_dumpall and
--no-comments, and argued for them being handled by this patch. This was
objected against on the grounds that pg_dumpall is out of scope, and
all-or-nothing switches not being applicable in a filter option.
Stephen objected to both the proposed, and the suggestion of JSON, in [2] and
argued for a more holistic configuration file approach. TOML was suggested.
Dean then +1'd the config file approach in [3].
In [4] Tom supported the idea of a more generic config file, and remarked that
the proposed filter for table names only makes sense when the number of exclude
patterns are large enough that we might hit other problems in pg_dump.
Further, in [5] Tom commented that a format with established quoting
conventions would buy us not having to invent our own to cope with complicated
relation names.
The fact that JSON doesn't support comments is brought up in a few emails and
is a very valid point, as the need for comments regardless of format is brought
up as well.
Tomas Vondra in [6] wanted the object filter be a separate file from a config
file, and argued for a simpler format for these lists (while still supporting
multiple object types).
Alvaro agreed with Tomas on [+-] OBJTYPE OBJIDENT in [7] and Tom extended the
proposal to use [include/exclude] keywords in [8] in order to support more than
just excluding and including. Regardless of stance on format, the use of
keywords instead of [+-] is a rare point of consensus in this thread.
Stephen and myself have also expressed concern in various parts of the thread
that inventing our own format rather than using something with existing broad
library support will end up with third-parties (like pgAdmin et.al) having to
all write their own generators and parsers.
A main concern among most (all?) participants of the thread, regardless of
format supported, is that quoting is hard and must be done right for all object
names postgres support (including any not currently in scope by this patch).
Below is an attempt at summarizing and grouping the proposals so far into the
set of ideas presented:
A) A keyword+object based format to invoke with a switch to essentially
allow for more filters than the commandline can handle and nothing more.
After a set of revisions, the current proposal is:
[include|exclude] [<objtype>] [<objident>]
B) A format similar to (A) which can also be used for pg_dump configuration
C) The format in (A), or a close variant thereof, with the intention of it
being included in/referred to from a future configuration file of currently
unknown format. One reference being a .gitignore type file.
D) An existing format (JSON and TOML have been suggested, with JSON
being dismissed due to lack of comment support) which has quoting
conventions that supports postgres' object names and which can be used to
define a full pg_dump configuration file syntax.
For B), C) and D) there is implicit consensus in the thread that we don't need
to implement the full configuration file as of this patch, merely that it
*must* be possible to do so without having to paint ourselves out of a corner.
At this point it seems to me that B) and C) has the broadest support. Can the
C) option may represent the compromise between "simple" format for object
filtering and a more structured format for configuration? Are there other
options?
Thoughts?
--
Daniel Gustafsson https://vmware.com/
[0] https://postgr.es/m/[email protected]
[1] https://postgr.es/m/CALAY4q9u30L7oGhbsfY3dPECQ8SrYa8YO=H-xOn5xWUeiEneeg@mail.gmail.com
[2] https://postgr.es/m/[email protected]
[3] https://postgr.es/m/CAEZATCVKMG7+b+_5tNwrNZ-aNDBy3=FMRNea2bO9O4qGcEvSTg@mail.gmail.com
[4] https://postgr.es/m/[email protected]
[5] https://postgr.es/m/[email protected]
[6] https://postgr.es/m/[email protected]
[7] https://postgr.es/m/[email protected]
[8] https://postgr.es/m/[email protected]
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-17 11:42 Daniel Gustafsson <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 2 replies; 181+ messages in thread
From: Daniel Gustafsson @ 2021-09-17 11:42 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 15 Sep 2021, at 19:31, Pavel Stehule <[email protected]> wrote:
> po 13. 9. 2021 v 15:01 odesílatel Daniel Gustafsson <[email protected] <mailto:[email protected]>> napsal:
> One issue with this syntax is that the include keyword can be quite misleading
> as it's semantic interpretion of "include table t" can be different from
> "--table=t". The former is less clear about the fact that it means "exclude
> all other tables than " then the latter. It can be solved with documentation,
> but I think that needs be to be made clearer.
>
> I invite any documentation enhancing and fixing
Sure, that can be collabored on. This gist is though that IMO the keywords in
the filter file aren't as clear on the sideeffects as the command line params,
even though they are equal in functionality.
> + pg_log_error("invalid format of filter file \"%s\": %s",
> + filename,
> + message);
> +
> + fprintf(stderr, "%d: %s\n", lineno, line);
> Can't we just include the lineno in the error logging and skip dumping the
> offending line? Fast-forwarding the pointer to print the offending part is
> less useful than a context marker, and in some cases suboptimal. With this
> coding, if a pattern is omitted for example the below error message is given:
>
> pg_dump: error: invalid format of filter file "filter.txt": missing object name
> 1:
>
> The errormessage and the linenumber in the file should be enough for the user
> to figure out what to fix.
>
> I did it like you proposed, but still, I think the content can be useful.
Not when there is no content in the error message, printing an empty string for
a line number which isn't a blank line doesn't seem terribly helpful. If we
know the error context is empty, printing a tailored error hint seems more
useful for the user.
> More times you read dynamically generated files, or you read data from stdin, and in complex environments it can be hard regenerate new content for debugging.
That seems odd given that the arguments for this format has been that it's
likely to be handwritten.
> If I create a table called "a\nb" and try to dump it I get an error in parsing
> the file. Isn't this supposed to work?
> $ cat filter.txt
> include table "a
> b"
> $ ./bin/pg_dump --filter=filter.txt
> pg_dump: error: invalid format of filter file "filter.txt": unexpected chars after object name
> 2:
>
> probably there was some issue, because it should work. I tested a new version and this is tested in new regress tests. Please, check
That seems to work, but I am unable to write a filter statement which can
handle this relname:
CREATE TABLE "a""
""b" (a integer);
Are you able to craft one for that?
> Did you consider implementing this in Bison to abstract some of the messier
> parsing logic?
>
> Initially not, but now, when I am thinking about it, I don't think so Bison helps. The syntax of the filter file is nicely linear. Now, the code of the parser is a little bit larger than minimalistic, but it is due to nicer error's messages. The raw implementation in Bison raised just "syntax error" and positions. I did code refactoring, and now the scanning, parsing and processing are divided into separated routines. Parsing related code has 90 lines. In this case, I don't think using a parser grammar file can carry any benefit. grammar is more readable, sure, but we need to include bison, we need to handle errors, and if we want to raise more helpful errors than just "syntax error", then the code will be longer.
I'm not so concerned by code size, but rather parsing of quotations etc and
being able to reason about it's correctness. IMHO that's easier done by
reading a defined grammar than parsing a handwritten parser.
Will do a closer review on the patch shortly.
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-17 11:42 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-09-17 11:42 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
> A main concern among most (all?) participants of the thread, regardless of
> format supported, is that quoting is hard and must be done right for all
> object
> names postgres support (including any not currently in scope by this
> patch).
>
>
Just a small note - when quoting is calculated to design, then the
implementation is easy. I am sure, so my last code covers all
possibilities, and it is about 100 lines of code.
> Below is an attempt at summarizing and grouping the proposals so far into
> the
> set of ideas presented:
>
> A) A keyword+object based format to invoke with a switch to essentially
> allow for more filters than the commandline can handle and nothing
> more.
> After a set of revisions, the current proposal is:
> [include|exclude] [<objtype>] [<objident>]
>
> B) A format similar to (A) which can also be used for pg_dump
> configuration
>
> C) The format in (A), or a close variant thereof, with the intention
> of it
> being included in/referred to from a future configuration file of
> currently
> unknown format. One reference being a .gitignore type file.
>
> D) An existing format (JSON and TOML have been suggested, with JSON
> being dismissed due to lack of comment support) which has quoting
> conventions that supports postgres' object names and which can be used
> to
> define a full pg_dump configuration file syntax.
>
> For B), C) and D) there is implicit consensus in the thread that we don't
> need
> to implement the full configuration file as of this patch, merely that it
> *must* be possible to do so without having to paint ourselves out of a
> corner.
>
> At this point it seems to me that B) and C) has the broadest support. Can
> the
> C) option may represent the compromise between "simple" format for object
> filtering and a more structured format for configuration? Are there other
> options?
>
What should be a benefit of this variant?
Regards
Pavel
>
> Thoughts?
>
> --
> Daniel Gustafsson https://vmware.com/
>
> [0] https://postgr.es/m/[email protected]
> [1]
> https://postgr.es/m/CALAY4q9u30L7oGhbsfY3dPECQ8SrYa8YO=H-xOn5xWUeiEneeg@mail.gmail.com
> [2] https://postgr.es/m/[email protected]
> [3]
> https://postgr.es/m/CAEZATCVKMG7+b+_5tNwrNZ-aNDBy3=FMRNea2bO9O4qGcEvSTg@mail.gmail.com
> [4] https://postgr.es/m/[email protected]
> [5] https://postgr.es/m/[email protected]
> [6]
> https://postgr.es/m/[email protected]
> [7] https://postgr.es/m/[email protected]
> [8] https://postgr.es/m/[email protected]
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-17 11:51 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2021-09-17 11:51 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
pá 17. 9. 2021 v 13:42 odesílatel Daniel Gustafsson <[email protected]>
napsal:
> > On 15 Sep 2021, at 19:31, Pavel Stehule <[email protected]> wrote:
> > po 13. 9. 2021 v 15:01 odesílatel Daniel Gustafsson <[email protected]
> <mailto:[email protected]>> napsal:
>
> > One issue with this syntax is that the include keyword can be quite
> misleading
> > as it's semantic interpretion of "include table t" can be different from
> > "--table=t". The former is less clear about the fact that it means
> "exclude
> > all other tables than " then the latter. It can be solved with
> documentation,
> > but I think that needs be to be made clearer.
> >
> > I invite any documentation enhancing and fixing
>
> Sure, that can be collabored on. This gist is though that IMO the
> keywords in
> the filter file aren't as clear on the sideeffects as the command line
> params,
> even though they are equal in functionality.
>
> > + pg_log_error("invalid format of filter file \"%s\": %s",
> > + filename,
> > + message);
> > +
> > + fprintf(stderr, "%d: %s\n", lineno, line);
> > Can't we just include the lineno in the error logging and skip dumping
> the
> > offending line? Fast-forwarding the pointer to print the offending part
> is
> > less useful than a context marker, and in some cases suboptimal. With
> this
> > coding, if a pattern is omitted for example the below error message is
> given:
> >
> > pg_dump: error: invalid format of filter file "filter.txt": missing
> object name
> > 1:
> >
> > The errormessage and the linenumber in the file should be enough for the
> user
> > to figure out what to fix.
> >
> > I did it like you proposed, but still, I think the content can be useful.
>
> Not when there is no content in the error message, printing an empty
> string for
> a line number which isn't a blank line doesn't seem terribly helpful. If
> we
> know the error context is empty, printing a tailored error hint seems more
> useful for the user.
>
> > More times you read dynamically generated files, or you read data from
> stdin, and in complex environments it can be hard regenerate new content
> for debugging.
>
> That seems odd given that the arguments for this format has been that it's
> likely to be handwritten.
>
> > If I create a table called "a\nb" and try to dump it I get an error in
> parsing
> > the file. Isn't this supposed to work?
> > $ cat filter.txt
> > include table "a
> > b"
> > $ ./bin/pg_dump --filter=filter.txt
> > pg_dump: error: invalid format of filter file "filter.txt":
> unexpected chars after object name
> > 2:
> >
> > probably there was some issue, because it should work. I tested a new
> version and this is tested in new regress tests. Please, check
>
> That seems to work, but I am unable to write a filter statement which can
> handle this relname:
>
> CREATE TABLE "a""
> ""b" (a integer);
>
> Are you able to craft one for that?
>
I am not able to dump this directly in pg_dump. Is it possible?
> > Did you consider implementing this in Bison to abstract some of the
> messier
> > parsing logic?
> >
> > Initially not, but now, when I am thinking about it, I don't think so
> Bison helps. The syntax of the filter file is nicely linear. Now, the code
> of the parser is a little bit larger than minimalistic, but it is due to
> nicer error's messages. The raw implementation in Bison raised just "syntax
> error" and positions. I did code refactoring, and now the scanning, parsing
> and processing are divided into separated routines. Parsing related code
> has 90 lines. In this case, I don't think using a parser grammar file can
> carry any benefit. grammar is more readable, sure, but we need to include
> bison, we need to handle errors, and if we want to raise more helpful
> errors than just "syntax error", then the code will be longer.
>
> I'm not so concerned by code size, but rather parsing of quotations etc and
> being able to reason about it's correctness. IMHO that's easier done by
> reading a defined grammar than parsing a handwritten parser.
>
> Will do a closer review on the patch shortly.
>
> --
> Daniel Gustafsson https://vmware.com/
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-17 11:56 Daniel Gustafsson <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Daniel Gustafsson @ 2021-09-17 11:56 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 17 Sep 2021, at 13:51, Pavel Stehule <[email protected]> wrote:
> pá 17. 9. 2021 v 13:42 odesílatel Daniel Gustafsson <[email protected] <mailto:[email protected]>> napsal:
> I am unable to write a filter statement which can
> handle this relname:
>
> CREATE TABLE "a""
> ""b" (a integer);
>
> Are you able to craft one for that?
>
> I am not able to dump this directly in pg_dump. Is it possible?
Sure, see below:
$ ./bin/psql filter
psql (15devel)
Type "help" for help.
filter=# create table "a""
filter"# ""b" (a integer);
CREATE TABLE
filter=# select relname from pg_class order by oid desc limit 1;
relname
---------
a" +
"b
(1 row)
filter=# ^D\q
$ ./bin/pg_dump -s filter
--
-- PostgreSQL database dump
--
-- Dumped from database version 15devel
-- Dumped by pg_dump version 15devel
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
SET default_tablespace = '';
SET default_table_access_method = heap;
--
-- Name: a" "b; Type: TABLE; Schema: public; Owner: danielg
--
CREATE TABLE public."a""
""b" (
a integer
);
ALTER TABLE public."a""
""b" OWNER TO danielg;
--
-- PostgreSQL database dump complete
--
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-17 11:59 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-09-17 11:59 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
pá 17. 9. 2021 v 13:56 odesílatel Daniel Gustafsson <[email protected]>
napsal:
> > On 17 Sep 2021, at 13:51, Pavel Stehule <[email protected]> wrote:
> > pá 17. 9. 2021 v 13:42 odesílatel Daniel Gustafsson <[email protected]
> <mailto:[email protected]>> napsal:
>
> > I am unable to write a filter statement which can
> > handle this relname:
> >
> > CREATE TABLE "a""
> > ""b" (a integer);
> >
> > Are you able to craft one for that?
> >
> > I am not able to dump this directly in pg_dump. Is it possible?
>
> Sure, see below:
>
> $ ./bin/psql filter
> psql (15devel)
> Type "help" for help.
>
>
I didn't ask on this
I asked if you can use -t and some for filtering this name
?
> filter=# create table "a""
> filter"# ""b" (a integer);
> CREATE TABLE
> filter=# select relname from pg_class order by oid desc limit 1;
> relname
> ---------
> a" +
> "b
> (1 row)
>
> filter=# ^D\q
> $ ./bin/pg_dump -s filter
> --
> -- PostgreSQL database dump
> --
>
> -- Dumped from database version 15devel
> -- Dumped by pg_dump version 15devel
>
> SET statement_timeout = 0;
> SET lock_timeout = 0;
> SET idle_in_transaction_session_timeout = 0;
> SET client_encoding = 'UTF8';
> SET standard_conforming_strings = on;
> SELECT pg_catalog.set_config('search_path', '', false);
> SET check_function_bodies = false;
> SET xmloption = content;
> SET client_min_messages = warning;
> SET row_security = off;
>
> SET default_tablespace = '';
>
> SET default_table_access_method = heap;
>
> --
> -- Name: a" "b; Type: TABLE; Schema: public; Owner: danielg
> --
>
> CREATE TABLE public."a""
> ""b" (
> a integer
> );
>
>
> ALTER TABLE public."a""
> ""b" OWNER TO danielg;
>
> --
> -- PostgreSQL database dump complete
> --
>
> --
> Daniel Gustafsson https://vmware.com/
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-17 12:06 Stephen Frost <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Stephen Frost @ 2021-09-17 12:06 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Daniel Gustafsson <[email protected]>; Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; pgsql-hackers; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Tomas Vondra <[email protected]>; vignesh C <[email protected]>
Greetings,
On Fri, Sep 17, 2021 at 13:59 Pavel Stehule <[email protected]> wrote:
>
>
> pá 17. 9. 2021 v 13:56 odesílatel Daniel Gustafsson <[email protected]>
> napsal:
>
>> > On 17 Sep 2021, at 13:51, Pavel Stehule <[email protected]>
>> wrote:
>> > pá 17. 9. 2021 v 13:42 odesílatel Daniel Gustafsson <[email protected]
>> <mailto:[email protected]>> napsal:
>>
>> > I am unable to write a filter statement which can
>> > handle this relname:
>> >
>> > CREATE TABLE "a""
>> > ""b" (a integer);
>> >
>> > Are you able to craft one for that?
>> >
>> > I am not able to dump this directly in pg_dump. Is it possible?
>>
>> Sure, see below:
>>
>> $ ./bin/psql filter
>> psql (15devel)
>> Type "help" for help.
>>
>>
> I didn't ask on this
>
> I asked if you can use -t and some for filtering this name
>
> ?
>
For my part, at least, I don’t see that this particularly matters.. for a
new feature that’s being developed to allow users to export specific
tables, I would think we’d want to support any table names which can exist.
Thanks,
Stephen
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-17 12:07 Daniel Gustafsson <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 2 replies; 181+ messages in thread
From: Daniel Gustafsson @ 2021-09-17 12:07 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 17 Sep 2021, at 13:59, Pavel Stehule <[email protected]> wrote:
> pá 17. 9. 2021 v 13:56 odesílatel Daniel Gustafsson <[email protected] <mailto:[email protected]>> napsal:
> > On 17 Sep 2021, at 13:51, Pavel Stehule <[email protected] <mailto:[email protected]>> wrote:
> > pá 17. 9. 2021 v 13:42 odesílatel Daniel Gustafsson <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> napsal:
> > I am unable to write a filter statement which can
> > handle this relname:
> >
> > CREATE TABLE "a""
> > ""b" (a integer);
> >
> > Are you able to craft one for that?
> >
> > I am not able to dump this directly in pg_dump. Is it possible?
>
> Sure, see below:
>
> $ ./bin/psql filter
> psql (15devel)
> Type "help" for help.
>
> I didn't ask on this
>
> I asked if you can use -t and some for filtering this name?
I didn't try as I don't see how that's relevant? Surely we're not limiting the
capabilities of a filtering file format based on the quoting semantics of a
shell?
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-17 12:09 Stephen Frost <[email protected]>
parent: Daniel Gustafsson <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Stephen Frost @ 2021-09-17 12:09 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Dean Rasheed <[email protected]>; Justin Pryzby <[email protected]>; Pavel Stehule <[email protected]>; pgsql-hackers; Surafel Temesgen <[email protected]>; Tom Lane <[email protected]>; Tomas Vondra <[email protected]>; vignesh C <[email protected]>
Greetings,
On Fri, Sep 17, 2021 at 14:07 Daniel Gustafsson <[email protected]> wrote:
> > On 17 Sep 2021, at 13:59, Pavel Stehule <[email protected]> wrote:
> > pá 17. 9. 2021 v 13:56 odesílatel Daniel Gustafsson <[email protected]
> <mailto:[email protected]>> napsal:
> > > On 17 Sep 2021, at 13:51, Pavel Stehule <[email protected]
> <mailto:[email protected]>> wrote:
> > > pá 17. 9. 2021 v 13:42 odesílatel Daniel Gustafsson <[email protected]
> <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>
> napsal:
>
> > > I am unable to write a filter statement which can
> > > handle this relname:
> > >
> > > CREATE TABLE "a""
> > > ""b" (a integer);
> > >
> > > Are you able to craft one for that?
> > >
> > > I am not able to dump this directly in pg_dump. Is it possible?
> >
> > Sure, see below:
> >
> > $ ./bin/psql filter
> > psql (15devel)
> > Type "help" for help.
> >
> > I didn't ask on this
> >
> > I asked if you can use -t and some for filtering this name?
>
> I didn't try as I don't see how that's relevant? Surely we're not
> limiting the
> capabilities of a filtering file format based on the quoting semantics of a
> shell?
Yeah, agreed. I would think that a DBA might specifically want to be able
to use a config file to get away from having to deal with shell quoting, in
fact…
Thanks,
Stephen
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-17 12:16 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-09-17 12:16 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
pá 17. 9. 2021 v 14:07 odesílatel Daniel Gustafsson <[email protected]>
napsal:
> > On 17 Sep 2021, at 13:59, Pavel Stehule <[email protected]> wrote:
> > pá 17. 9. 2021 v 13:56 odesílatel Daniel Gustafsson <[email protected]
> <mailto:[email protected]>> napsal:
> > > On 17 Sep 2021, at 13:51, Pavel Stehule <[email protected]
> <mailto:[email protected]>> wrote:
> > > pá 17. 9. 2021 v 13:42 odesílatel Daniel Gustafsson <[email protected]
> <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>
> napsal:
>
> > > I am unable to write a filter statement which can
> > > handle this relname:
> > >
> > > CREATE TABLE "a""
> > > ""b" (a integer);
> > >
> > > Are you able to craft one for that?
> > >
> > > I am not able to dump this directly in pg_dump. Is it possible?
> >
> > Sure, see below:
> >
> > $ ./bin/psql filter
> > psql (15devel)
> > Type "help" for help.
> >
> > I didn't ask on this
> >
> > I asked if you can use -t and some for filtering this name?
>
> I didn't try as I don't see how that's relevant? Surely we're not
> limiting the
> capabilities of a filtering file format based on the quoting semantics of a
> shell?
>
this patch just use existing functionality, that can be buggy too.
but I had a bug in this part - if I detect double double quotes on input I
have to send double quotes to output too.
It should be fixed in attached patch
[pavel@localhost pg_dump]$ echo 'include table "a""\n""b"' | ./pg_dump
--filter=-
--
-- PostgreSQL database dump
--
-- Dumped from database version 15devel
-- Dumped by pg_dump version 15devel
> --
> Daniel Gustafsson https://vmware.com/
>
>
Attachments:
[text/x-patch] pg_dump-filteropt-20210917.patch (17.2K, ../../CAFj8pRDkEg8d5-++ZZfqS7w4U074WamLPfuosx259jAHTVJBbw@mail.gmail.com/3-pg_dump-filteropt-20210917.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7682226b99..1b74c0eadd 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -789,6 +789,55 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file. Specify "-" to read from
+ stdin. Lines of this file must have the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the object is to be included
+ or excluded, and the second keyword specifies the type of object
+ to be filtered:
+ <literal>table</literal> (table),
+ <literal>schema</literal> (schema),
+ <literal>foreign_data</literal> (foreign server),
+ <literal>data</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
+include table mytable1
+include foreign_data some_foreign_server
+exclude table mytable2
+</programlisting>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are ignored. The comment
+ (started by <literal>#</literal>) can be placed after filter too.
+ Empty lines are ignored too.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
+ tables, and both forms may be combined. Note that there are no options
+ to exclude a specific foreign table or to include a specific table's
+ data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a485fb2d07..0e8072a782 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -55,10 +55,12 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -308,7 +310,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
-
+static void read_filters_from_file(char *filename, DumpOptions *dopt);
int
main(int argc, char **argv)
@@ -380,6 +382,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -613,6 +616,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_filters_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1038,6 +1045,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18979,3 +18988,346 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+typedef struct
+{
+ FILE *fp;
+ char *filename;
+ int lineno;
+} FilterStateData;
+
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_DATA
+} FilterObjectType;
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+
+ fclose(fstate->fp);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Search keyword (can contains only ascii alphabetic characters) on line.
+ * Returns NULL, when the line is empty or first char is not alpha
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* skip initial white spaces */
+ while (isspace(*ptr))
+ ptr += 1;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr += 1;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+static bool
+filter_is_keyword(const char *keyword, int size, const char *str)
+{
+ if (strlen(str) != size)
+ return false;
+
+ return pg_strncasecmp(keyword, str, size) == 0;
+}
+
+/*
+ * Sets objname to string with object identifier. The line variable holds string
+ * of last line with object identifier (object name). Returns pointer to first char
+ * after last char of object name.
+ */
+static char *
+filter_get_object_name(FilterStateData *fstate,
+ StringInfo line,
+ char *str,
+ char **objname)
+{
+ /* skip white spaces */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ exit_invalid_filter_format(fstate, "missing object name");
+
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ if (!pg_get_line_buf(fstate->fp, line))
+ {
+ if (ferror(fstate->fp))
+ fatal("could not read from file \"%s\": %m", fstate->filename);
+
+ exit_invalid_filter_format(fstate,"unexpected end of file");
+ }
+
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ str = line->data;
+ fstate->lineno += 1;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = quoted_name->data;
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* simple variant, read to end or to first space */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ return str;
+}
+
+/*
+ * Returns true, when one filter item was successfully read and parsed.
+ * When object name contains \n chars, then more than one line from input
+ * file can be processed. Returns false when EOF. Run exit on error.
+ */
+static bool
+read_filter_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ StringInfoData line;
+
+ initStringInfo(&line);
+
+ if (pg_get_line_buf(fstate->fp, &line))
+ {
+ char *str = line.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * skip empty lines or lines when first noblank char is hash (comment)
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no keyword found (expected \"include\" or \"exclude\")");
+
+ /* Now we expect sequence of two keywords */
+ if (filter_is_keyword(keyword, size, "include"))
+ *is_include = true;
+ else if (filter_is_keyword(keyword, size, "exclude"))
+ *is_include = false;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid keyword (expected \"include\" or \"exclude\")");
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no keyword found (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ if (filter_is_keyword(keyword, size, "table"))
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (filter_is_keyword(keyword, size, "schema"))
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (filter_is_keyword(keyword, size, "foreign_data"))
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (filter_is_keyword(keyword, size, "data"))
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid keyword (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ str = filter_get_object_name(fstate, &line, str, objname);
+
+ /*
+ * check possible content after object identifier.
+ * Allow comment started by hash.
+ */
+ while (isspace(*str))
+ str++;
+
+ if (*str != '\0' && *str != '#')
+ exit_invalid_filter_format(fstate,
+ "unexpected chars after object name");
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ free(line.data);
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_fatal("could not read from file \"%s\": %m", fstate->filename);
+
+ if (fstate->fp != stdin)
+ fclose(fstate->fp);
+
+ exit_nicely(-1);
+ }
+
+ free(line.data);
+
+ return false;
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_filters_from_file(char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ fstate.filename = filename;
+ fstate.lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate.fp = fopen(filename, "r");
+ if (!fstate.fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fstate.fp = stdin;
+
+ while (read_filter_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ exit_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+
+ free(objname);
+ }
+
+ if (fstate.fp != stdin)
+ fclose(fstate.fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..505614f145
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,151 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 24;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = PostgresNode->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid keyword/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid keyword/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/missing object name/,
+ "broken format check");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-17 13:06 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2021-09-17 13:06 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> >
> > Initially not, but now, when I am thinking about it, I don't think so
> Bison helps. The syntax of the filter file is nicely linear. Now, the code
> of the parser is a little bit larger than minimalistic, but it is due to
> nicer error's messages. The raw implementation in Bison raised just "syntax
> error" and positions. I did code refactoring, and now the scanning, parsing
> and processing are divided into separated routines. Parsing related code
> has 90 lines. In this case, I don't think using a parser grammar file can
> carry any benefit. grammar is more readable, sure, but we need to include
> bison, we need to handle errors, and if we want to raise more helpful
> errors than just "syntax error", then the code will be longer.
>
> I'm not so concerned by code size, but rather parsing of quotations etc and
> being able to reason about it's correctness. IMHO that's easier done by
> reading a defined grammar than parsing a handwritten parser.
>
>
In this case the complex part is not a parser, but the scanner is complex
and writing this in flex is not too easy. I wrote so the grammar file can
be more readable, but the usual error from Bison is "syntax error" and
position, so it does not win from the user perspective. When a parser is
not linear, then a generated parser can help a lot, but using it at this
moment is premature.
> Will do a closer review on the patch shortly.
>
> --
> Daniel Gustafsson https://vmware.com/
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-20 12:10 Daniel Gustafsson <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Daniel Gustafsson @ 2021-09-20 12:10 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> Will do a closer review on the patch shortly.
Had a read through, and tested, the latest posted version today:
+ Read objects filters from the specified file. Specify "-" to read from
+ stdin. Lines of this file must have the following format:
I think this should be <filename>-</filename> and <literal>STDIN</literal> to
match the rest of the docs.
+ <para>
+ With the following filter file, the dump would include table
+ <literal>mytable1</literal> and data from foreign tables of
+ <literal>some_foreign_server</literal> foreign server, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
+include table mytable1
+include foreign_data some_foreign_server
+exclude table mytable2
+</programlisting>
+ </para>
This example is highlighting the issue I've previously raised with the UX/doc
of this feature. The "exclude table mytable2" is totally pointless in the
above since the exact match of "mytable1" will remove all other objects. What
we should be doing instead is use the pattern matching aspect along the lines
of the below:
include table mytable*
exclude table mytable2
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables, schemas, table data, or foreign
This should refer to the actual options by name to make it clear which we are
talking about.
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
Before we settle on --filter I think we need to conclude whether this file is
intended to be included from a config file, or used on it's own. If we gow tih
the former then we might not want a separate option for just --filter.
+ if (filter_is_keyword(keyword, size, "include"))
I would prefer if this function call was replaced by just the pg_strcasecmp()
call in filter_is_keyword() and the strlen optimization there removed. The is
not a hot-path, we can afford the string comparison in case of errors. Having
the string comparison done inline here will improve readability saving the
reading from jumping to another function to see what it does.
+ initStringInfo(&line);
Why is this using a StringInfo rather than a PQExpBuffer as the rest of pg_dump
does?
+typedef struct
I think these should be at the top of the file with the other typedefs.
When testing strange object names, I was unable to express this name in the filter file:
$ ./bin/psql
psql (15devel)
Type "help" for help.
danielg=# create table "
danielg"# t
danielg"# t
danielg"# " (a integer);
CREATE TABLE
danielg=# select relname from pg_class order by oid desc limit 1;
relname
---------
+
t +
t +
(1 row)
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-21 06:50 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2021-09-21 06:50 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
po 20. 9. 2021 v 14:10 odesílatel Daniel Gustafsson <[email protected]>
napsal:
> > Will do a closer review on the patch shortly.
>
> Had a read through, and tested, the latest posted version today:
>
> + Read objects filters from the specified file. Specify "-" to read from
> + stdin. Lines of this file must have the following format:
> I think this should be <filename>-</filename> and <literal>STDIN</literal>
> to
> match the rest of the docs.
>
>
> + <para>
> + With the following filter file, the dump would include table
> + <literal>mytable1</literal> and data from foreign tables of
> + <literal>some_foreign_server</literal> foreign server, but
> exclude data
> + from table <literal>mytable2</literal>.
> +<programlisting>
> +include table mytable1
> +include foreign_data some_foreign_server
> +exclude table mytable2
> +</programlisting>
> + </para>
> This example is highlighting the issue I've previously raised with the
> UX/doc
> of this feature. The "exclude table mytable2" is totally pointless in the
> above since the exact match of "mytable1" will remove all other objects.
> What
> we should be doing instead is use the pattern matching aspect along the
> lines
> of the below:
>
> include table mytable*
> exclude table mytable2
>
> + The <option>--filter</option> option works just like the other
> + options to include or exclude tables, schemas, table data, or
> foreign
> This should refer to the actual options by name to make it clear which we
> are
> talking about.
>
fixed
>
> + printf(_(" --filter=FILENAME dump objects and data
> based on the filter expressions\n"
> + " from the filter
> file\n"));
> Before we settle on --filter I think we need to conclude whether this file
> is
> intended to be included from a config file, or used on it's own. If we
> gow tih
> the former then we might not want a separate option for just --filter.
>
I prefer to separate two files. Although there is some intersection, I
think it is good to have two simple separate files for two really different
tasks.
It does filtering, and it should be controlled by option "--filter". When
the implementation will be changed, then this option can be changed too.
Filtering is just a pg_dump related feature. Revision of client application
configuration is a much more generic task, and if we mix it to one, we can
be
in a trap. It can be hard to find one good format for large script
generated content, and possibly hand written structured content. For
practical
reasons it can be good to have two files too. Filters and configurations
can have different life cycles.
>
> + if (filter_is_keyword(keyword, size, "include"))
> I would prefer if this function call was replaced by just the
> pg_strcasecmp()
> call in filter_is_keyword() and the strlen optimization there removed.
> The is
> not a hot-path, we can afford the string comparison in case of errors.
> Having
> the string comparison done inline here will improve readability saving the
> reading from jumping to another function to see what it does.
>
I agree that this is not a hot-path, just I don't feel well if I need to
make a zero end string just for comparison pg_strcasecmp. Current design
reduces malloc/free cycles. It is used in more places, when Postgres parses
strings - SQL parser, plpgsql parser. I am not sure about the benefits and
costs - pg_strcasecmp can be more readable, but for any keyword I have to
call pstrdup and pfree. Is it necessary? My opinion in this part is not too
strong - it is a minor issue, maybe I have a little bit different feelings
about benefits and costs in this specific case, and if you really think the
benefits of rewriting are higher, I'll do it.
>
> + initStringInfo(&line);
> Why is this using a StringInfo rather than a PQExpBuffer as the rest of
> pg_dump
> does?
>
The StringInfo is used because I use the pg_get_line_buf function, and this
function uses this API.
>
> +typedef struct
> I think these should be at the top of the file with the other typedefs.
>
>
done
>
> When testing strange object names, I was unable to express this name in
> the filter file:
>
> $ ./bin/psql
> psql (15devel)
> Type "help" for help.
>
> danielg=# create table "
> danielg"# t
> danielg"# t
> danielg"# " (a integer);
> CREATE TABLE
> danielg=# select relname from pg_class order by oid desc limit 1;
> relname
> ---------
> +
> t +
> t +
>
> (1 row)
>
>
Good catch - I had badly placed pg_strip_crlf function, fixed and regress
tests enhanced
Please check assigned patch
Regards
Pavel
> --
> Daniel Gustafsson https://vmware.com/
>
>
Attachments:
[text/x-patch] pg_dump-filteropt-20210921.patch (18.4K, ../../CAFj8pRDFwyGmyefihxnk4ZCVedPp=Gs3H=CxbPj2-hBguvmoXw@mail.gmail.com/3-pg_dump-filteropt-20210921.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7682226b99..d692f4230e 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -789,6 +789,55 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Read objects filters from the specified file. Specify <filename>-</filename> to read from
+ <literal>STDIN</literal>. Lines of this file must have the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the object is to be included
+ or excluded, and the second keyword specifies the type of object
+ to be filtered:
+ <literal>table</literal> (table),
+ <literal>schema</literal> (schema),
+ <literal>foreign_data</literal> (foreign server),
+ <literal>data</literal> (table data).
+ </para>
+
+ <para>
+ With the following filter file, the dump would include tables with
+ name starting by <literal>mytable</literal>, but exclude data
+ from table <literal>mytable2</literal>.
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are ignored. The comment
+ (started by <literal>#</literal>) can be placed after filter too.
+ Empty lines are ignored too.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables (<option>-t</option> or
+ <option>--table</option>), schemas (<option>-n</option> or
+ <option>--schema</option>), table data (<option>--exclude-table-data</option>),
+ or foreign tables (<option>--include-foreign-data</option>.
+ Isn't possible to exclude a specific foreign table and isn't possible
+ to include a specific table's data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a485fb2d07..4da5e6c771 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -55,10 +55,12 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -90,6 +92,28 @@ typedef enum OidOptions
zeroAsNone = 4
} OidOptions;
+/*
+ * State data for reading filter items from stream.
+ */
+typedef struct
+{
+ FILE *fp;
+ char *filename;
+ int lineno;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_DATA
+} FilterObjectType;
+
/* global decls */
static bool dosync = true; /* Issue fsync() to make dump durable on disk. */
@@ -308,7 +332,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
-
+static void read_filters_from_file(char *filename, DumpOptions *dopt);
int
main(int argc, char **argv)
@@ -380,6 +404,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -613,6 +638,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_filters_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1038,6 +1067,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " from the filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18979,3 +19010,330 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+
+ fclose(fstate->fp);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ exit_nicely(-1);
+}
+
+/*
+ * Search keyword (can contains only ascii alphabetic characters) on line.
+ * Returns NULL, when the line is empty or first char is not alpha
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* skip initial white spaces */
+ while (isspace(*ptr))
+ ptr += 1;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr += 1;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+static bool
+filter_is_keyword(const char *keyword, int size, const char *str)
+{
+ if (strlen(str) != size)
+ return false;
+
+ return pg_strncasecmp(keyword, str, size) == 0;
+}
+
+/*
+ * Sets objname to string with object identifier. The line variable holds string
+ * of last line with object identifier (object name). Returns pointer to first char
+ * after last char of object name.
+ */
+static char *
+filter_get_object_name(FilterStateData *fstate,
+ StringInfo line,
+ char *str,
+ char **objname)
+{
+ /* skip white spaces */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ exit_invalid_filter_format(fstate, "missing object name");
+
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ if (!pg_get_line_buf(fstate->fp, line))
+ {
+ if (ferror(fstate->fp))
+ fatal("could not read from file \"%s\": %m", fstate->filename);
+
+ exit_invalid_filter_format(fstate,"unexpected end of file");
+ }
+
+ str = line->data;
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ fstate->lineno += 1;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = quoted_name->data;
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* simple variant, read to end or to first space */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ return str;
+}
+
+/*
+ * Returns true, when one filter item was successfully read and parsed.
+ * When object name contains \n chars, then more than one line from input
+ * file can be processed. Returns false when EOF. Run exit on error.
+ */
+static bool
+read_filter_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ StringInfoData line;
+
+ initStringInfo(&line);
+
+ if (pg_get_line_buf(fstate->fp, &line))
+ {
+ char *str = line.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * skip empty lines or lines when first noblank char is hash (comment)
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no keyword found (expected \"include\" or \"exclude\")");
+
+ /* Now we expect sequence of two keywords */
+ if (filter_is_keyword(keyword, size, "include"))
+ *is_include = true;
+ else if (filter_is_keyword(keyword, size, "exclude"))
+ *is_include = false;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid keyword (expected \"include\" or \"exclude\")");
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no keyword found (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ if (filter_is_keyword(keyword, size, "table"))
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (filter_is_keyword(keyword, size, "schema"))
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (filter_is_keyword(keyword, size, "foreign_data"))
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (filter_is_keyword(keyword, size, "data"))
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid keyword (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ str = filter_get_object_name(fstate, &line, str, objname);
+
+ /*
+ * check possible content after object identifier.
+ * Allow comment started by hash.
+ */
+ while (isspace(*str))
+ str++;
+
+ if (*str != '\0' && *str != '#')
+ exit_invalid_filter_format(fstate,
+ "unexpected chars after object name");
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ free(line.data);
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_fatal("could not read from file \"%s\": %m", fstate->filename);
+
+ if (fstate->fp != stdin)
+ fclose(fstate->fp);
+
+ exit_nicely(-1);
+ }
+
+ free(line.data);
+
+ return false;
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_filters_from_file(char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ fstate.filename = filename;
+ fstate.lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate.fp = fopen(filename, "r");
+ if (!fstate.fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fstate.fp = stdin;
+
+ while (read_filter_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ exit_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+
+ free(objname);
+ }
+
+ if (fstate.fp != stdin)
+ fclose(fstate.fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..85b84db855
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,185 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 28;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = PostgresNode->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "include table \"
+t
+t
+\"";
+
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms, "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "include table \"\\nt\\nt\\n\"";
+
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms, "dump table with multiline strange name");
+
+#########################################
+# For test of +f option we need created foreign server or accept
+# fail and check error
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid keyword/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid keyword/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/missing object name/,
+ "broken format check");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-21 12:37 Daniel Gustafsson <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 4 replies; 181+ messages in thread
From: Daniel Gustafsson @ 2021-09-21 12:37 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 21 Sep 2021, at 08:50, Pavel Stehule <[email protected]> wrote:
> po 20. 9. 2021 v 14:10 odesílatel Daniel Gustafsson <[email protected] <mailto:[email protected]>> napsal:
> + printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
> + " from the filter file\n"));
> Before we settle on --filter I think we need to conclude whether this file is
> intended to be included from a config file, or used on it's own. If we gow tih
> the former then we might not want a separate option for just --filter.
>
> I prefer to separate two files. Although there is some intersection, I think it is good to have two simple separate files for two really different tasks.
> It does filtering, and it should be controlled by option "--filter". When the implementation will be changed, then this option can be changed too.
> Filtering is just a pg_dump related feature. Revision of client application configuration is a much more generic task, and if we mix it to one, we can be
> in a trap. It can be hard to find one good format for large script generated content, and possibly hand written structured content. For practical
> reasons it can be good to have two files too. Filters and configurations can have different life cycles.
I'm not convinced that we can/should change or remove a commandline parameter
in a coming version when there might be scripts expecting it to work in a
specific way. Having a --filter as well as a --config where the configfile can
refer to the filterfile also passed via --filter sounds like problem waiting to
happen, so I think we need to settle how we want to interact with this file
before anything goes in.
Any thoughts from those in the thread who have had strong opinions on config
files etc?
> + if (filter_is_keyword(keyword, size, "include"))
> I would prefer if this function call was replaced by just the pg_strcasecmp()
> call in filter_is_keyword() and the strlen optimization there removed. The is
> not a hot-path, we can afford the string comparison in case of errors. Having
> the string comparison done inline here will improve readability saving the
> reading from jumping to another function to see what it does.
>
> I agree that this is not a hot-path, just I don't feel well if I need to make a zero end string just for comparison pg_strcasecmp. Current design reduces malloc/free cycles. It is used in more places, when Postgres parses strings - SQL parser, plpgsql parser. I am not sure about the benefits and costs - pg_strcasecmp can be more readable, but for any keyword I have to call pstrdup and pfree. Is it necessary? My opinion in this part is not too strong - it is a minor issue, maybe I have a little bit different feelings about benefits and costs in this specific case, and if you really think the benefits of rewriting are higher, I'll do it
Sorry, I typoed my response. What I meant was to move the pg_strncasecmp call
inline and not do the strlen check, to save readers from jumping around. So
basically end up with the below in read_filter_item():
+ /* Now we expect sequence of two keywords */
+ if (pg_strncasecmp(keyword, "include", size) == 0)
+ *is_include = true;
> + initStringInfo(&line);
> Why is this using a StringInfo rather than a PQExpBuffer as the rest of pg_dump
> does?
>
> The StringInfo is used because I use the pg_get_line_buf function, and this function uses this API.
Ah, of course.
A few other comments from another pass over this:
+ exit_nicely(-1);
Why -1? pg_dump (and all other binaries) exits with 1 on IMO even more serious
errors so I think this should use 1 as well.
+ if (!pg_get_line_buf(fstate->fp, line))
+ {
+ if (ferror(fstate->fp))
+ fatal("could not read from file \"%s\": %m", fstate->filename);
+
+ exit_invalid_filter_format(fstate,"unexpected end of file");
+ }
In the ferror() case this codepath isn't running fclose() on the file pointer
(unless stdin) which we do elsewhere, so this should use pg_log_error and
exit_nicely instead.
+ pg_log_fatal("could not read from file \"%s\": %m", fstate->filename);
Based on how other errors are treated in pg_dump I think this should be
downgraded to a pg_log_error.
The above comments are fixed in the attached, as well as a pass over the docs
and extended tests to actually test matching a foreign server. What do think
about this version? I'm still not convinced that there aren't more quoting
bugs in the parser, but I've left that intact for now.
--
Daniel Gustafsson https://vmware.com/
Attachments:
[application/octet-stream] 0001-Implement-filter-for-pg_dump.patch (20.0K, ../../[email protected]/2-0001-Implement-filter-for-pg_dump.patch)
download | inline diff:
From 7fcc71c322bc700119c2fafa15884bad2a43f0ff Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Tue, 21 Sep 2021 14:26:35 +0200
Subject: [PATCH] Implement --filter for pg_dump
Author: Pavel Stehule
---
doc/src/sgml/ref/pg_dump.sgml | 61 ++++
src/bin/pg_dump/pg_dump.c | 356 ++++++++++++++++++++
src/bin/pg_dump/t/004_pg_dump_filterfile.pl | 198 +++++++++++
3 files changed, 615 insertions(+)
create mode 100644 src/bin/pg_dump/t/004_pg_dump_filterfile.pl
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7682226b99..6e9a56f68c 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -789,6 +789,67 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Filtering rules for which objects to dump are read from the specified
+ file. Specify <filename>-</filename> to read from
+ <literal>STDIN</literal>. The file has the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the object is to be included or
+ excluded, and the second keyword specifies the type of object to be
+ filtered:
+ <itemizedlist>
+ <listitem>
+ <para><literal>table</literal>: table</para>
+ </listitem>
+ <listitem>
+ <para><literal>schema</literal>: schema</para>
+ </listitem>
+ <listitem>
+ <para><literal>foreign_data</literal>: foreign server</para>
+ </listitem>
+ <listitem>
+ <para><literal>data</literal>: table data</para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Example:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+ With the this filter file, the dump would include all tables with
+ name starting by <literal>mytable</literal>, but exclude table
+ <literal>mytable2</literal>.
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ are ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables (<option>-t</option> or
+ <option>--table</option>), schemas (<option>-n</option> or
+ <option>--schema</option>), table data (<option>--exclude-table-data</option>),
+ or foreign tables (<option>--include-foreign-data</option>).
+ It isn't possible to exclude a specific foreign table or
+ to include a specific table's data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a485fb2d07..45b1626a85 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -55,10 +55,12 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -90,6 +92,28 @@ typedef enum OidOptions
zeroAsNone = 4
} OidOptions;
+/*
+ * State data for reading filter items from stream.
+ */
+typedef struct
+{
+ FILE *fp;
+ char *filename;
+ int lineno;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_DATA
+} FilterObjectType;
+
/* global decls */
static bool dosync = true; /* Issue fsync() to make dump durable on disk. */
@@ -308,6 +332,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void read_filters_from_file(char *filename, DumpOptions *dopt);
int
@@ -380,6 +405,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -613,6 +639,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_filters_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1038,6 +1068,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18979,3 +19011,327 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+
+ fclose(fstate->fp);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ exit_nicely(1);
+}
+
+/*
+ * Search keyword (can contains only ascii alphabetic characters) on line.
+ * Returns NULL, when the line is empty or first char is not alpha
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* skip initial white spaces */
+ while (isspace(*ptr))
+ ptr += 1;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr += 1;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * Sets objname to string with object identifier. The line variable holds
+ * string of last line with object identifier (object name). Returns pointer to
+ * first char after object name.
+ */
+static char *
+filter_get_object_name(FilterStateData *fstate,
+ StringInfo line,
+ char *str,
+ char **objname)
+{
+ /* skip white spaces */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ exit_invalid_filter_format(fstate, "missing object name");
+
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ if (!pg_get_line_buf(fstate->fp, line))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m", fstate->filename);
+ if (fstate->fp != stdin)
+ fclose(fstate->fp);
+
+ exit_nicely(1);
+ }
+
+ exit_invalid_filter_format(fstate,"unexpected end of file");
+ }
+
+ str = line->data;
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ fstate->lineno += 1;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = quoted_name->data;
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* simple variant, read to end or to first space */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ return str;
+}
+
+/*
+ * Returns true, when one filter item was successfully read and parsed.
+ * When object name contains \n chars, then more than one line from input
+ * file can be processed. Returns false when EOF. Run exit on error.
+ */
+static bool
+read_filter_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ StringInfoData line;
+
+ initStringInfo(&line);
+
+ if (pg_get_line_buf(fstate->fp, &line))
+ {
+ char *str = line.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * skip empty lines or lines when first noblank char is hash (comment)
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no keyword found (expected \"include\" or \"exclude\")");
+
+ /* Now we expect sequence of two keywords */
+ if (pg_strncasecmp(keyword, "include", size) == 0)
+ *is_include = true;
+ else if (pg_strncasecmp(keyword, "exclude", size) == 0)
+ *is_include = false;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid keyword (expected \"include\" or \"exclude\")");
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no keyword found (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ if (pg_strncasecmp(keyword, "table", size) == 0)
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (pg_strncasecmp(keyword, "schema", size) == 0)
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (pg_strncasecmp(keyword, "foreign_data", size) == 0)
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (pg_strncasecmp(keyword, "data", size) == 0)
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid keyword (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ str = filter_get_object_name(fstate, &line, str, objname);
+
+ /*
+ * check possible content after object identifier.
+ * Allow comment started by hash.
+ */
+ while (isspace(*str))
+ str++;
+
+ if (*str != '\0' && *str != '#')
+ exit_invalid_filter_format(fstate,
+ "unexpected chars after object name");
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ free(line.data);
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m", fstate->filename);
+
+ if (fstate->fp != stdin)
+ fclose(fstate->fp);
+
+ exit_nicely(1);
+ }
+
+ free(line.data);
+
+ return false;
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_filters_from_file(char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ fstate.filename = filename;
+ fstate.lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate.fp = fopen(filename, "r");
+ if (!fstate.fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fstate.fp = stdin;
+
+ while (read_filter_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ exit_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+
+ free(objname);
+ }
+
+ if (fstate.fp != stdin)
+ fclose(fstate.fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..ccb28c6b8c
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,198 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 29;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = PostgresNode->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres', 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms, "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms, "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummy*\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump foreign_data with filter");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid keyword/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid keyword/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/missing object name/,
+ "broken format check");
--
2.24.3 (Apple Git-128)
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-21 12:46 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
3 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-09-21 12:46 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
út 21. 9. 2021 v 14:37 odesílatel Daniel Gustafsson <[email protected]>
napsal:
> > On 21 Sep 2021, at 08:50, Pavel Stehule <[email protected]> wrote:
> > po 20. 9. 2021 v 14:10 odesílatel Daniel Gustafsson <[email protected]
> <mailto:[email protected]>> napsal:
>
> > + printf(_(" --filter=FILENAME dump objects and data
> based on the filter expressions\n"
> > + " from the filter
> file\n"));
> > Before we settle on --filter I think we need to conclude whether this
> file is
> > intended to be included from a config file, or used on it's own. If we
> gow tih
> > the former then we might not want a separate option for just --filter.
> >
> > I prefer to separate two files. Although there is some intersection, I
> think it is good to have two simple separate files for two really different
> tasks.
> > It does filtering, and it should be controlled by option "--filter".
> When the implementation will be changed, then this option can be changed
> too.
> > Filtering is just a pg_dump related feature. Revision of client
> application configuration is a much more generic task, and if we mix it to
> one, we can be
> > in a trap. It can be hard to find one good format for large script
> generated content, and possibly hand written structured content. For
> practical
> > reasons it can be good to have two files too. Filters and configurations
> can have different life cycles.
>
> I'm not convinced that we can/should change or remove a commandline
> parameter
> in a coming version when there might be scripts expecting it to work in a
> specific way. Having a --filter as well as a --config where the
> configfile can
> refer to the filterfile also passed via --filter sounds like problem
> waiting to
> happen, so I think we need to settle how we want to interact with this file
> before anything goes in.
>
> Any thoughts from those in the thread who have had strong opinions on
> config
> files etc?
>
> > + if (filter_is_keyword(keyword, size, "include"))
> > I would prefer if this function call was replaced by just the
> pg_strcasecmp()
> > call in filter_is_keyword() and the strlen optimization there removed.
> The is
> > not a hot-path, we can afford the string comparison in case of errors.
> Having
> > the string comparison done inline here will improve readability saving
> the
> > reading from jumping to another function to see what it does.
> >
> > I agree that this is not a hot-path, just I don't feel well if I need to
> make a zero end string just for comparison pg_strcasecmp. Current design
> reduces malloc/free cycles. It is used in more places, when Postgres parses
> strings - SQL parser, plpgsql parser. I am not sure about the benefits and
> costs - pg_strcasecmp can be more readable, but for any keyword I have to
> call pstrdup and pfree. Is it necessary? My opinion in this part is not too
> strong - it is a minor issue, maybe I have a little bit different feelings
> about benefits and costs in this specific case, and if you really think the
> benefits of rewriting are higher, I'll do it
>
> Sorry, I typoed my response. What I meant was to move the pg_strncasecmp
> call
> inline and not do the strlen check, to save readers from jumping around.
> So
> basically end up with the below in read_filter_item():
>
> + /* Now we expect sequence of two keywords */
> + if (pg_strncasecmp(keyword, "include", size) == 0)
> + *is_include = true;
>
>
I don't think so it is safe (strict). Only pg_strncasecmp(..) is true for
keywords "includex", "includedsss", ... You should to compare the size
Regards
Pavel
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-21 13:28 Alvaro Herrera <[email protected]>
parent: Daniel Gustafsson <[email protected]>
3 siblings, 1 reply; 181+ messages in thread
From: Alvaro Herrera @ 2021-09-21 13:28 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Pavel Stehule <[email protected]>; Tom Lane <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
I definitely agree that we should have two files, one for config and
another one for filter, since their purposes are orthogonal and their
formats are likely different; trying to cram the filter specification in
the config file seems unfriendly because it'd force users to write the
filter in whatever alien grammar used for the config file. Also, this
would make it easier to use a single config file with a bunch of
different filter files.
On 2021-Sep-21, Daniel Gustafsson wrote:
> I'm not convinced that we can/should change or remove a commandline parameter
> in a coming version when there might be scripts expecting it to work in a
> specific way. Having a --filter as well as a --config where the configfile can
> refer to the filterfile also passed via --filter sounds like problem waiting to
> happen, so I think we need to settle how we want to interact with this file
> before anything goes in.
I think both the filter and the hypothetical config file are going to
interact (be redundant) with almost all already existing switches, and
there's no need to talk about removing anything (e.g., nobody would
argue for the removal of "-t" even though that's redundant with the
filter file).
I see no problem with the config file specifying a filter file.
AFAICS if the config file specifies a filter and the user also specifies
a filter in the command line, we have two easy options: raise an error
about the redundant option, or have the command line option supersede
the one in the config file. The latter strikes me as the more useful
behavior, and it's in line with what other tools do in similar cases, so
that's what I propose doing.
(There might be less easy options too, such as somehow combining the two
filters, but offhand I don't see any reason why this is real-world
useful, so I don't propose doing that.)
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
"How amazing is that? I call it a night and come back to find that a bug has
been identified and patched while I sleep." (Robert Davidson)
http://archives.postgresql.org/pgsql-sql/2006-03/msg00378.php
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-21 16:06 Tomas Vondra <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Tomas Vondra @ 2021-09-21 16:06 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; Daniel Gustafsson <[email protected]>; +Cc: Pavel Stehule <[email protected]>; Tom Lane <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On 9/21/21 3:28 PM, Alvaro Herrera wrote:
> I definitely agree that we should have two files, one for config and
> another one for filter, since their purposes are orthogonal and their
> formats are likely different; trying to cram the filter specification in
> the config file seems unfriendly because it'd force users to write the
> filter in whatever alien grammar used for the config file. Also, this
> would make it easier to use a single config file with a bunch of
> different filter files.
>
+1, that is pretty much excatly what I argued for not too long ago.
> On 2021-Sep-21, Daniel Gustafsson wrote:
>
>> I'm not convinced that we can/should change or remove a commandline parameter
>> in a coming version when there might be scripts expecting it to work in a
>> specific way. Having a --filter as well as a --config where the configfile can
>> refer to the filterfile also passed via --filter sounds like problem waiting to
>> happen, so I think we need to settle how we want to interact with this file
>> before anything goes in.
>
> I think both the filter and the hypothetical config file are going to
> interact (be redundant) with almost all already existing switches, and
> there's no need to talk about removing anything (e.g., nobody would
> argue for the removal of "-t" even though that's redundant with the
> filter file).
>
> I see no problem with the config file specifying a filter file.
>
> AFAICS if the config file specifies a filter and the user also specifies
> a filter in the command line, we have two easy options: raise an error
> about the redundant option, or have the command line option supersede
> the one in the config file. The latter strikes me as the more useful
> behavior, and it's in line with what other tools do in similar cases, so
> that's what I propose doing.
>
> (There might be less easy options too, such as somehow combining the two
> filters, but offhand I don't see any reason why this is real-world
> useful, so I don't propose doing that.)
>
Well, I think we already have to do decisions like that, because you can
do e.g. this:
pg_dump -T t -t t
So we already do combine the switches, and we do this:
When both -t and -T are given, the behavior is to dump just the
tables that match at least one -t switch but no -T switches. If -T
appears without -t, then tables matching -T are excluded from what
is otherwise a normal dump.
That seems fairly reasonable, and I don't see why not to use the same
logic for combining patterns no matter where we got them (filter file,
command-line option, etc.).
Just combine everything, and then check if there's any "exclude" rule.
If yes, we're done - exclude. If not, check if there's "include" rule.
If not, still exclude. Otherwise include.
Seems reasonable and consistent to me, and I don't see why not to allow
multiple --filter parameters.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-21 16:20 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
3 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-09-21 16:20 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
> The above comments are fixed in the attached, as well as a pass over the
> docs
> and extended tests to actually test matching a foreign server. What do
> think
> about this version? I'm still not convinced that there aren't more quoting
> bugs in the parser, but I've left that intact for now.
>
The problematic points are double quotes and new line char. Any other is
just in sequence of bytes.
I have just one note to your patch. When you use pg_strncasecmp, then you
have to check the size too
char *xxx = "incl";
int xxx_size = 4;
elog(NOTICE, ">>>>%d<<<<",
pg_strncasecmp(xxx, "include", xxx_size) == 0);
result is NOTICE: >>>>1<<<<
"incl" is not keyword "include"
Regards
Pavel
>
> --
> Daniel Gustafsson https://vmware.com/
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-09-24 03:59 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
3 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2021-09-24 03:59 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
> The above comments are fixed in the attached, as well as a pass over the
> docs
> and extended tests to actually test matching a foreign server. What do
> think
> about this version? I'm still not convinced that there aren't more quoting
> bugs in the parser, but I've left that intact for now.
>
This patch is based on the version that you sent 21.9. Just I modified
string comparison in keyword detection. If we don't allow support
abbreviations of keywords (and I dislike it), then the check of size is
necessary. Any other is without change.
Regards
Pavel
>
> --
> Daniel Gustafsson https://vmware.com/
>
>
Attachments:
[text/x-patch] pg_dump-filteropt-20210924.patch (19.6K, ../../CAFj8pRD-p3dHzByfo_ZiGvff10k3Vk4ee2HtHkpH13s0Hf3bhA@mail.gmail.com/3-pg_dump-filteropt-20210924.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7682226b99..6e9a56f68c 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -789,6 +789,67 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Filtering rules for which objects to dump are read from the specified
+ file. Specify <filename>-</filename> to read from
+ <literal>STDIN</literal>. The file has the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the object is to be included or
+ excluded, and the second keyword specifies the type of object to be
+ filtered:
+ <itemizedlist>
+ <listitem>
+ <para><literal>table</literal>: table</para>
+ </listitem>
+ <listitem>
+ <para><literal>schema</literal>: schema</para>
+ </listitem>
+ <listitem>
+ <para><literal>foreign_data</literal>: foreign server</para>
+ </listitem>
+ <listitem>
+ <para><literal>data</literal>: table data</para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Example:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+ With the this filter file, the dump would include all tables with
+ name starting by <literal>mytable</literal>, but exclude table
+ <literal>mytable2</literal>.
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ are ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables (<option>-t</option> or
+ <option>--table</option>), schemas (<option>-n</option> or
+ <option>--schema</option>), table data (<option>--exclude-table-data</option>),
+ or foreign tables (<option>--include-foreign-data</option>).
+ It isn't possible to exclude a specific foreign table or
+ to include a specific table's data.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a485fb2d07..ec979b9f6f 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -55,10 +55,12 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -90,6 +92,28 @@ typedef enum OidOptions
zeroAsNone = 4
} OidOptions;
+/*
+ * State data for reading filter items from stream.
+ */
+typedef struct
+{
+ FILE *fp;
+ char *filename;
+ int lineno;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_DATA
+} FilterObjectType;
+
/* global decls */
static bool dosync = true; /* Issue fsync() to make dump durable on disk. */
@@ -308,6 +332,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void read_filters_from_file(char *filename, DumpOptions *dopt);
int
@@ -380,6 +405,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -613,6 +639,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* filter implementation */
+ read_filters_from_file(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1038,6 +1068,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18979,3 +19011,327 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * Print error message and exit.
+ */
+static void
+exit_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+
+ fclose(fstate->fp);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ exit_nicely(1);
+}
+
+/*
+ * Search keyword (can contains only ascii alphabetic characters) on line.
+ * Returns NULL, when the line is empty or first char is not alpha
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* skip initial white spaces */
+ while (isspace(*ptr))
+ ptr += 1;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr += 1;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * Sets objname to string with object identifier. The line variable holds
+ * string of last line with object identifier (object name). Returns pointer to
+ * first char after object name.
+ */
+static char *
+filter_get_object_name(FilterStateData *fstate,
+ StringInfo line,
+ char *str,
+ char **objname)
+{
+ /* skip white spaces */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ exit_invalid_filter_format(fstate, "missing object name");
+
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ if (!pg_get_line_buf(fstate->fp, line))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m", fstate->filename);
+ if (fstate->fp != stdin)
+ fclose(fstate->fp);
+
+ exit_nicely(1);
+ }
+
+ exit_invalid_filter_format(fstate,"unexpected end of file");
+ }
+
+ str = line->data;
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ fstate->lineno += 1;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = quoted_name->data;
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* simple variant, read to end or to first space */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ return str;
+}
+
+/*
+ * Returns true, when one filter item was successfully read and parsed.
+ * When object name contains \n chars, then more than one line from input
+ * file can be processed. Returns false when EOF. Run exit on error.
+ */
+static bool
+read_filter_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ StringInfoData line;
+
+ initStringInfo(&line);
+
+ if (pg_get_line_buf(fstate->fp, &line))
+ {
+ char *str = line.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno += 1;
+
+ (void) pg_strip_crlf(str);
+
+ /* skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * skip empty lines or lines when first noblank char is hash (comment)
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no keyword found (expected \"include\" or \"exclude\")");
+
+ /* Now we expect sequence of two keywords */
+ if (size == 7 && pg_strncasecmp(keyword, "include", 7) == 0)
+ *is_include = true;
+ else if (size == 7 && pg_strncasecmp(keyword, "exclude", 7) == 0)
+ *is_include = false;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid keyword (expected \"include\" or \"exclude\")");
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no keyword found (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ if (size == 5 && pg_strncasecmp(keyword, "table", 5) == 0)
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (size == 6 && pg_strncasecmp(keyword, "schema", 6) == 0)
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (size == 12 && pg_strncasecmp(keyword, "foreign_data", 12) == 0)
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (size == 4 && pg_strncasecmp(keyword, "data", 4) == 0)
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid keyword (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ str = filter_get_object_name(fstate, &line, str, objname);
+
+ /*
+ * check possible content after object identifier.
+ * Allow comment started by hash.
+ */
+ while (isspace(*str))
+ str++;
+
+ if (*str != '\0' && *str != '#')
+ exit_invalid_filter_format(fstate,
+ "unexpected chars after object name");
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ free(line.data);
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from file \"%s\": %m", fstate->filename);
+
+ if (fstate->fp != stdin)
+ fclose(fstate->fp);
+
+ exit_nicely(1);
+ }
+
+ free(line.data);
+
+ return false;
+}
+
+/*
+ * Read dumped object specification from file
+ */
+static void
+read_filters_from_file(char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ fstate.filename = filename;
+ fstate.lineno = 0;
+
+ /* use "-" as symbol for stdin */
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate.fp = fopen(filename, "r");
+ if (!fstate.fp)
+ fatal("could not open the input file \"%s\": %m",
+ filename);
+ }
+ else
+ fstate.fp = stdin;
+
+ while (read_filter_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ exit_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+
+ free(objname);
+ }
+
+ if (fstate.fp != stdin)
+ fclose(fstate.fp);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..adc5675e40
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,198 @@
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 29;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+
+my $node = PostgresNode->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres', 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+my ($cmd, $stdout, $stderr, $result);
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public.table_three/m, "table three not dumped");
+ok($dump !~ qr/^COPY public.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public.table_two/m, "content of table two is included");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public.table_three/m, "dumped table three");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms, "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms, "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummy*\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump foreign_data with filter");
+
+#########################################
+# Test broken input format
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid keyword/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid keyword/,
+ "broken format check");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/missing object name/,
+ "broken format check");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-10-01 13:19 Daniel Gustafsson <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 3 replies; 181+ messages in thread
From: Daniel Gustafsson @ 2021-10-01 13:19 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
I took another pass over this today and touched up the documentation (docs and
code) as well as tweaked the code a bit here and there to both make it fit the
pg_dump style better and to clean up a few small things. I've also added a set
of additional tests to cover more of the functionality.
I'm still not happy with the docs, I need to take another look there and see if
I make them more readable but otherwise I don't think there are any open issues
with this.
As has been discussed upthread, this format strikes a compromise wrt simplicity
and doesn't preclude adding a more structured config file in the future should
we want that. I think this takes care of most comments and opinions made in
this thread.
--
Daniel Gustafsson https://vmware.com/
Attachments:
[application/octet-stream] pg_dump-filteropt-20211001.patch (27.4K, ../../[email protected]/2-pg_dump-filteropt-20211001.patch)
download | inline diff:
From 6d2edf9144685436702829acd60b802fb52c3306 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Fri, 1 Oct 2021 15:03:15 +0200
Subject: [PATCH v2] Add --filter option for reading object patterns from file
This adds a --filter=FILENAME option to pg_dump for specifying a set of
inclusion/exclusion patterns in a file. In situations where the wanted
filterset exceeds the commandline capabilities, this can be a handy way
to still be able to filter objects. The format of the file is line based
with each pattern on its own line:
<command> <object_type> <pattern>
When object names contain whitespace the pattern must be quoted.
Author: Pavel Stehule <[email protected]>
Discussion: https://postgr.es/m/CAFj8pRB10wvW0CC9Xq=1XDs=zCQxer3cbLcNZa+qiX4cUH-G_A@mail.gmail.com
---
doc/src/sgml/ref/pg_dump.sgml | 63 +++
src/bin/pg_dump/pg_dump.c | 419 ++++++++++++++++++++
src/bin/pg_dump/t/004_pg_dump_filterfile.pl | 294 ++++++++++++++
3 files changed, 776 insertions(+)
create mode 100644 src/bin/pg_dump/t/004_pg_dump_filterfile.pl
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7682226b99..2c266f60aa 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -789,6 +789,69 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Filtering rules for which objects to dump are read from the specified
+ file. Specify <filename>-</filename> to read from
+ <literal>STDIN</literal>. The file has the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para><literal>table</literal>: table</para>
+ </listitem>
+ <listitem>
+ <para><literal>schema</literal>: schema</para>
+ </listitem>
+ <listitem>
+ <para><literal>foreign_data</literal>: foreign server</para>
+ </listitem>
+ <listitem>
+ <para><literal>data</literal>: table data</para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Example:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+ With this filter file, the dump would include all tables with
+ name starting by <literal>mytable</literal>, except for table
+ <literal>mytable2</literal> which is explicitly excluded.
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ are ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored.
+ </para>
+
+ <para>
+ The <option>--filter</option> option works just like the other
+ options to include or exclude tables (<option>-t</option> or
+ <option>--table</option>), schemas (<option>-n</option> or
+ <option>--schema</option>), table data (<option>--exclude-table-data</option>),
+ or foreign tables (<option>--include-foreign-data</option>).
+ It isn't possible to exclude a specific foreign table or
+ to include a specific table's data. The <option>--filter</option>
+ option can be specified in conjunction with these options, and can
+ also be specified more than once if there are multiple filter files.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a485fb2d07..f074c363b7 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -55,10 +55,12 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -90,6 +92,28 @@ typedef enum OidOptions
zeroAsNone = 4
} OidOptions;
+/*
+ * State data for reading filter items from stream
+ */
+typedef struct
+{
+ FILE *fp;
+ const char *filename;
+ int lineno;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_DATA
+} FilterObjectType;
+
/* global decls */
static bool dosync = true; /* Issue fsync() to make dump durable on disk. */
@@ -308,6 +332,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void getFiltersFromFile(const char *filename, DumpOptions *dopt);
int
@@ -380,6 +405,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -613,6 +639,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ getFiltersFromFile(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1038,6 +1068,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18979,3 +19011,390 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * exit_invalid_filter_format - Emit error message, close the file and exit
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+static void
+exit_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+
+ if (fclose(fstate->fp) != 0)
+ fatal("could not close filter file \"%s\": %m", fstate->filename);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ exit_nicely(1);
+}
+
+/*
+ * filter_get_keyword - read the next filter keyword from buffer
+ *
+ * Search for keywords (limited to containing ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL, when the buffer is empty or first
+ * char is not alpha. The length of the found keyword is returned in the size
+ * parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* Set returnlength preemptively in case no keyword is found */
+ *size = 0;
+
+ /* Skip initial whitespace */
+ while (isspace(*ptr))
+ ptr++;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr++;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * filter_get_pattern - Read an object identifier pattern from the buffer
+ *
+ * Parses an object identifier pattern from the passed in buffer and sets
+ * objname to a string with object identifier pattern. Returns pointer to the
+ * first character after the pattern.
+ */
+static char *
+filter_get_pattern(FilterStateData *fstate,
+ char *str,
+ char **objname)
+{
+ StringInfoData line;
+
+ /* We only allocate the buffer if we need it */
+ line.data = NULL;
+
+ /* Skip whitespace */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ exit_invalid_filter_format(fstate, "missing object name pattern");
+
+ /*
+ * If the object name pattern has been quoted we must take care parse out
+ * the entire quoted pattern, which may contain whitespace and can span
+ * over many lines.
+ */
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ if (line.data == NULL)
+ initStringInfo(&line);
+
+ if (!pg_get_line_buf(fstate->fp, &line))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m",
+ fstate->filename);
+ if (fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ fatal("could not close filter file \"%s\": %m",
+ fstate->filename);
+ }
+
+ exit_nicely(1);
+ }
+
+ exit_invalid_filter_format(fstate, "unexpected end of file");
+ }
+
+ str = line.data;
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ fstate->lineno++;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = pg_strdup(quoted_name->data);
+ destroyPQExpBuffer(quoted_name);
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* Simple variant, read to EOL or to first whitespace */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ if (line.data != NULL)
+ pg_free(line.data);
+
+ return str;
+}
+
+/*
+ * read_filter_item - Read command/type/pattern triplet from filter file
+ *
+ * This will parse one filter item from the filter file, and while it is a row
+ * based format a pattern may span more than one line due to how object names
+ * can be constructed. The expected format of the filter file is:
+ *
+ * <command> <object_type> <pattern>
+ *
+ * Where command is "include" or "exclude", and object_type is one of: "table",
+ * "schema", "foreign_data" or "data". The pattern is either simple without any
+ * whitespace, or properly quoted in case there is whitespace in the object
+ * name. The pattern handling follows the same rules as other object include
+ * and exclude functions; it can use wildcards Returns true, when one filter
+ * item was successfully read and parsed. When object name contains \n chars,
+ * then more than one line from input file can be processed. Returns false when
+ * the filter file reaches EOF. In case of errors, the function wont return
+ * but will exit with an appropriate error message.
+ */
+static bool
+read_filter_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ StringInfoData line;
+
+ initStringInfo(&line);
+
+ if (pg_get_line_buf(fstate->fp, &line))
+ {
+ char *str = line.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno++;
+
+ (void) pg_strip_crlf(str);
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Skip empty lines or lines where the first non-whitespace character
+ * is a hash indicating a comment.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ /*
+ * First we expect sequence of two keywords, {include|exclude}
+ * followed by the object type to operate on.
+ */
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no filtercommand found (expected \"include\" or \"exclude\")");
+
+ if (size == 7 && pg_strncasecmp(keyword, "include", 7) == 0)
+ *is_include = true;
+ else if (size == 7 && pg_strncasecmp(keyword, "exclude", 7) == 0)
+ *is_include = false;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid filtercommand (expected \"include\" or \"exclude\")");
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no object type found (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ if (size == 5 && pg_strncasecmp(keyword, "table", 5) == 0)
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (size == 6 && pg_strncasecmp(keyword, "schema", 6) == 0)
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (size == 12 && pg_strncasecmp(keyword, "foreign_data", 12) == 0)
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (size == 4 && pg_strncasecmp(keyword, "data", 4) == 0)
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid object type (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ str = filter_get_pattern(fstate, str, objname);
+
+ /*
+ * Look for any content after the object identifier. Comments and
+ * whitespace are allowed, other content may indicate that the user
+ * needed to quote the object name so exit with an invalid format
+ * error.
+ */
+ while (isspace(*str))
+ str++;
+
+ if (*str != '\0' && *str != '#')
+ exit_invalid_filter_format(fstate,
+ "unexpected extra data after pattern");
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ free(line.data);
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m", fstate->filename);
+
+ if (fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ fatal("could not close filter file \"%s\": %m", fstate->filename);
+ }
+
+ exit_nicely(1);
+ }
+
+ free(line.data);
+
+ return false;
+}
+
+/*
+ * getFiltersFromFile - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getFiltersFromFile(const char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ fstate.filename = filename;
+ fstate.lineno = 0;
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate.fp = fopen(filename, "r");
+ if (!fstate.fp)
+ fatal("could not open filter file \"%s\": %m", filename);
+ }
+ else
+ fstate.fp = stdin;
+
+ while (read_filter_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ exit_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ if (fstate.fp != stdin)
+ {
+ if (fclose(fstate.fp) != 0)
+ fatal("could not close filter file \"%s\": %m", fstate.filename);
+ }
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..ec4ee9d8a7
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,294 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 45;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+my $node = PostgresNode->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres', 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres', "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+$node->safe_psql('postgres', "INSERT INTO table_three_one VALUES('*** TABLE THREE_ONE ***')");
+
+#
+# Test interaction of correctly specified filter file
+#
+my ($cmd, $stdout, $stderr, $result);
+
+# Empty filterfile
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "\n # a comment and nothing more\n\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, '-f', $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "filter file without patterns");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_(one|two|three|three_one)/m, "tables dumped");
+
+# Test various combinations of whitespace, comments and correct filters
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "\t\n";
+print $inputfile " \t# another comment\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter patterns as well as comments and whitespace");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public\.table_three/m, "table three not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_three_one/m, "table three_one not dumped");
+ok($dump !~ qr/^COPY public\.table_one/m, "content of table one is not included");
+ok($dump =~ qr/^COPY public\.table_two/m, "content of table two is included");
+
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with exclusion of a single table");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m, "dumped table three_one");
+
+# Test dumping tables with a wildcard pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_thre*\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with wildcard in pattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_two/m, "table two not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m, "dumped table three_one");
+
+# Test dumping table with multiline quoted tablename
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with multiline names requiring quoting");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+# Test excluding multiline quoted tablename from dump
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m, "dump table with new line in name");
+
+# Test excluding an entire schema
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "exclude the public schema");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test including and excluding an entire schema by multiple filterfiles
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema public\n";
+close $inputfile;
+
+open my $alt_inputfile, '>', "$tempdir/inputfile2.txt"
+ or die "unable to open filterfile for writing";
+print $alt_inputfile "exclude schema public\n";
+close $alt_inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", "--filter=$tempdir/inputfile2.txt", 'postgres' ],
+ "exclude the public schema with multiple filters");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test dumping a table with a single leading newline on a row
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, '-f', $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms, "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms, "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, '-f', $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump nonexisting foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummyserver\n";
+close $inputfile;
+
+command_ok(
+ [ "pg_dump", '-p', $port, '-f', $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ "dump foreign_data with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE SERVER dummyserver/m, "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude foreign_data dummy*\n";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, '-f', $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/exclude filter is not allowed/,
+ "erroneously exclude foreign server");
+
+#########################################
+# Test broken input format
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid filtercommand/,
+ "invalid syntax: incorrect filtercommand");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/invalid object type/,
+ "invalid syntax: invalid object type specified, should be table, schema, foreign_data or data");
+
+# Test missing object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/missing object name/,
+ "invalid syntax: missing object identifier pattern");
+
+# Test adding extra content after the object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table one";
+close $inputfile;
+
+command_fails_like(
+ [ "pg_dump", '-p', $port, "-f", $plainfile, "--filter=$tempdir/inputfile.txt", 'postgres' ],
+ qr/unexpected extra data/,
+ "invalid syntax: extra content after object identifier pattern");
--
2.24.3 (Apple Git-128)
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-10-01 16:00 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
2 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-10-01 16:00 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
pá 1. 10. 2021 v 15:19 odesílatel Daniel Gustafsson <[email protected]>
napsal:
> I took another pass over this today and touched up the documentation (docs
> and
> code) as well as tweaked the code a bit here and there to both make it fit
> the
> pg_dump style better and to clean up a few small things. I've also added
> a set
> of additional tests to cover more of the functionality.
>
> I'm still not happy with the docs, I need to take another look there and
> see if
> I make them more readable but otherwise I don't think there are any open
> issues
> with this.
>
> As has been discussed upthread, this format strikes a compromise wrt
> simplicity
> and doesn't preclude adding a more structured config file in the future
> should
> we want that. I think this takes care of most comments and opinions made
> in
> this thread.
>
It looks well.
Thank you
Pavel
> --
> Daniel Gustafsson https://vmware.com/
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-10-01 16:19 Erik Rijkers <[email protected]>
parent: Daniel Gustafsson <[email protected]>
2 siblings, 1 reply; 181+ messages in thread
From: Erik Rijkers @ 2021-10-01 16:19 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On 10/1/21 3:19 PM, Daniel Gustafsson wrote:
>
> As has been discussed upthread, this format strikes a compromise wrt simplicity
> and doesn't preclude adding a more structured config file in the future should
> we want that. I think this takes care of most comments and opinions made in
> this thread.
>
> --
> Daniel Gustafsson https://vmware.com/
>
Hi,
If you try to dump/restore a foreign file from a file_fdw server, the
restore step will complain and thus leave the returnvalue nonzero. The
foreign table will be there, with complete 'data'.
A complete runnable exampe is a lot of work; I hope the below bits of
input and output makes the problem clear. Main thing: the pg_restore
contains 2 ERROR lines like:
pg_restore: error: COPY failed for table "ireise1": ERROR: cannot
insert into foreign table "ireise1"
----------------------
From the test bash:
echo "
include table table0 # ok public
include table test.table1 #
include foreign_data goethe # foreign server 'goethe' (file_fdw)
include table gutenberg.ireise1 # foreign table
include table gutenberg.ireise2 # foreign table
" > inputfile1.txt
pg_dump --create -Fc -c -p $port -d $db1 -f dump1 --filter=inputfile1.txt
echo
# prepare for restore
server_name=goethe
echo "create schema if not exists test;" | psql -qaXd $db2
echo "create schema if not exists gutenberg;" | psql -qaXd $db2
echo "create server if not exists $server_name foreign data wrapper
file_fdw " \
| psql -qaXd $db2
echo "-- pg_restore --if-exists -cvd $db2 dump1 "
pg_restore --if-exists -cvd $db2 dump1
rc=$?
echo "-- rc [$rc]" -
echo
----------------------
from the output:
-- pg_dump --create -Fc -c -p 6969 -d testdb1 -f dump1
--filter=inputfile1.txt
-- pg_restore --if-exists -cvd testdb2 dump1
pg_restore: connecting to database for restore
pg_restore: dropping TABLE table1
pg_restore: dropping TABLE table0
pg_restore: dropping FOREIGN TABLE ireise2
pg_restore: dropping FOREIGN TABLE ireise1
pg_restore: creating FOREIGN TABLE "gutenberg.ireise1"
pg_restore: creating COMMENT "gutenberg.FOREIGN TABLE ireise1"
pg_restore: creating FOREIGN TABLE "gutenberg.ireise2"
pg_restore: creating COMMENT "gutenberg.FOREIGN TABLE ireise2"
pg_restore: creating TABLE "public.table0"
pg_restore: creating TABLE "test.table1"
pg_restore: processing data for table "gutenberg.ireise1"
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 5570; 0 23625 TABLE DATA ireise1 aardvark
pg_restore: error: COPY failed for table "ireise1": ERROR: cannot
insert into foreign table "ireise1"
pg_restore: processing data for table "gutenberg.ireise2"
pg_restore: from TOC entry 5571; 0 23628 TABLE DATA ireise2 aardvark
pg_restore: error: COPY failed for table "ireise2": ERROR: cannot
insert into foreign table "ireise2"
pg_restore: processing data for table "public.table0"
pg_restore: processing data for table "test.table1"
pg_restore: warning: errors ignored on restore: 2
-- rc [1]
---------
A second, separate practical hickup is that schema's are not restored
from the dumped $schema.$table includes -- but this can be worked
around; for my inputfile1.txt I had to run separately (as seen above,
before running the pg_restore):
create schema if not exists test;
create schema if not exists gutenberg;
create server if not exists goethe foreign data wrapper file_fdw;
A bit annoying but still maybe all right.
Thanks,
Erik Rijkers
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-10-02 06:18 Erik Rijkers <[email protected]>
parent: Erik Rijkers <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Erik Rijkers @ 2021-10-02 06:18 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On 10/1/21 6:19 PM, Erik Rijkers wrote:
> On 10/1/21 3:19 PM, Daniel Gustafsson wrote:
>>
>> As has been discussed upthread, this format strikes a compromise wrt
>> simplicity
>> and doesn't preclude adding a more structured config file in the
>> future should
>
> If you try to dump/restore a foreign file from a file_fdw server, the
> restore step will complain and thus leave the returnvalue nonzero. The
> foreign table will be there, with complete 'data'.
>
> A complete runnable exampe is a lot of work; I hope the below bits of
> input and output makes the problem clear. Main thing: the pg_restore
> contains 2 ERROR lines like:
>
> pg_restore: error: COPY failed for table "ireise1": ERROR: cannot
> insert into foreign table "ireise1"
Further testing makes clear that the file_fdw-addressing line
include foreign_data goethe
was the culprit: it causes a COPY which of course fails in a readonly
wrapper like file_fdw. Without that line it works (because I run the
restore on the same machine so the underlying file_fdw .txt files are
there for testdb2 too)
So the issue is not as serious as it seemed. The complaint remaining is
only that this could somehow be documented better.
I attach a running example (careful, it deletes stuff) of the original
ERROR-producing bash (remove the 'include foreign_data' line from the
input file to run it without error).
thanks,
Erik Rijkers
Attachments:
[application/x-shellscript] dump_restore_foreign_table.sh (2.6K, ../../[email protected]/2-dump_restore_foreign_table.sh)
download
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-10-04 12:54 Daniel Gustafsson <[email protected]>
parent: Erik Rijkers <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Daniel Gustafsson @ 2021-10-04 12:54 UTC (permalink / raw)
To: Erik Rijkers <[email protected]>; +Cc: Pavel Stehule <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 2 Oct 2021, at 08:18, Erik Rijkers <[email protected]> wrote:
> So the issue is not as serious as it seemed.
This is also not related to this patch in any way, or am I missing a point
here? This can just as well be achieved without this patch.
> The complaint remaining is only that this could somehow be documented better.
The pg_dump documentation today have a large highlighted note about this:
"When --include-foreign-data is specified, pg_dump does not check that the
foreign table is writable. Therefore, there is no guarantee that the
results of a foreign table dump can be successfully restored."
This was extensively discussed [0] when this went in, is there additional
documentation you'd like to see for this?
--
Daniel Gustafsson https://vmware.com/
[0] https://postgr.es/m/LEJPR01MB0185483C0079D2F651B16231E7FC0@LEJPR01MB0185.DEUPRD01.PROD.OUTLOOK.DE
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-10-05 12:30 Daniel Gustafsson <[email protected]>
parent: Daniel Gustafsson <[email protected]>
2 siblings, 3 replies; 181+ messages in thread
From: Daniel Gustafsson @ 2021-10-05 12:30 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 1 Oct 2021, at 15:19, Daniel Gustafsson <[email protected]> wrote:
> I'm still not happy with the docs, I need to take another look there and see if
> I make them more readable but otherwise I don't think there are any open issues
> with this.
Attached is a rebased version which has rewritten docs which I think are more
in line with the pg_dump documentation. I've also added tests for
--strict-names operation, as well subjected it to pgindent and pgperltidy.
Unless there are objections, I think this is pretty much ready to go in.
--
Daniel Gustafsson https://vmware.com/
Attachments:
[application/octet-stream] pg_dump-filteropt-20211005.patch (30.4K, ../../[email protected]/2-pg_dump-filteropt-20211005.patch)
download | inline diff:
From 052f234e53626ca8906a972c89a1b55f67e22beb Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Fri, 1 Oct 2021 15:03:15 +0200
Subject: [PATCH] Add --filter option for reading object patterns from file
This adds the --filter=FILENAME option to pg_dump for specifying a set of
inclusion/exclusion patterns in a file. In situations where the wanted
filterset exceeds the commandline capabilities, this can be a handy way
to still be able to filter objects. The format of the file is line based
with each pattern on its own line:
<command> <object_type> <pattern>
When object names contain whitespace the pattern must be quoted.
The object_types table, schema, foreign_data and data each correspond
to the existing command line parameters -t/--table, -n/--schema,
--include-foreign-data and --exclude-table-data.
Author: Pavel Stehule <[email protected]>
Discussion: https://postgr.es/m/CAFj8pRB10wvW0CC9Xq=1XDs=zCQxer3cbLcNZa+qiX4cUH-G_A@mail.gmail.com
fixup
---
doc/src/sgml/ref/pg_dump.sgml | 88 ++++
src/bin/pg_dump/pg_dump.c | 419 ++++++++++++++++++++
src/bin/pg_dump/t/004_pg_dump_filterfile.pl | 399 +++++++++++++++++++
3 files changed, 906 insertions(+)
create mode 100644 src/bin/pg_dump/t/004_pg_dump_filterfile.pl
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7682226b99..1c60f24b1d 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -789,6 +789,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal> use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ are ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1122,6 +1196,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1531,6 +1606,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a485fb2d07..796c431aaa 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -55,10 +55,12 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -90,6 +92,28 @@ typedef enum OidOptions
zeroAsNone = 4
} OidOptions;
+/*
+ * State data for reading filter items from stream
+ */
+typedef struct
+{
+ FILE *fp;
+ const char *filename;
+ int lineno;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_DATA
+} FilterObjectType;
+
/* global decls */
static bool dosync = true; /* Issue fsync() to make dump durable on disk. */
@@ -308,6 +332,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void getFiltersFromFile(const char *filename, DumpOptions *dopt);
int
@@ -380,6 +405,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -613,6 +639,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ getFiltersFromFile(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1038,6 +1068,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18979,3 +19011,390 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * exit_invalid_filter_format - Emit error message, close the file and exit
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+static void
+exit_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+
+ if (fclose(fstate->fp) != 0)
+ fatal("could not close filter file \"%s\": %m", fstate->filename);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ exit_nicely(1);
+}
+
+/*
+ * filter_get_keyword - read the next filter keyword from buffer
+ *
+ * Search for keywords (limited to containing ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL, when the buffer is empty or first
+ * char is not alpha. The length of the found keyword is returned in the size
+ * parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* Set returnlength preemptively in case no keyword is found */
+ *size = 0;
+
+ /* Skip initial whitespace */
+ while (isspace(*ptr))
+ ptr++;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr++;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * filter_get_pattern - Read an object identifier pattern from the buffer
+ *
+ * Parses an object identifier pattern from the passed in buffer and sets
+ * objname to a string with object identifier pattern. Returns pointer to the
+ * first character after the pattern.
+ */
+static char *
+filter_get_pattern(FilterStateData *fstate,
+ char *str,
+ char **objname)
+{
+ StringInfoData line;
+
+ /* We only allocate the buffer if we need it */
+ line.data = NULL;
+
+ /* Skip whitespace */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ exit_invalid_filter_format(fstate, "missing object name pattern");
+
+ /*
+ * If the object name pattern has been quoted we must take care parse out
+ * the entire quoted pattern, which may contain whitespace and can span
+ * over many lines.
+ */
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ if (line.data == NULL)
+ initStringInfo(&line);
+
+ if (!pg_get_line_buf(fstate->fp, &line))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m",
+ fstate->filename);
+ if (fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ fatal("could not close filter file \"%s\": %m",
+ fstate->filename);
+ }
+
+ exit_nicely(1);
+ }
+
+ exit_invalid_filter_format(fstate, "unexpected end of file");
+ }
+
+ str = line.data;
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ fstate->lineno++;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = pg_strdup(quoted_name->data);
+ destroyPQExpBuffer(quoted_name);
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* Simple variant, read to EOL or to first whitespace */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ if (line.data != NULL)
+ pg_free(line.data);
+
+ return str;
+}
+
+/*
+ * read_filter_item - Read command/type/pattern triplet from filter file
+ *
+ * This will parse one filter item from the filter file, and while it is a row
+ * based format a pattern may span more than one line due to how object names
+ * can be constructed. The expected format of the filter file is:
+ *
+ * <command> <object_type> <pattern>
+ *
+ * Where command is "include" or "exclude", and object_type is one of: "table",
+ * "schema", "foreign_data" or "data". The pattern is either simple without any
+ * whitespace, or properly quoted in case there is whitespace in the object
+ * name. The pattern handling follows the same rules as other object include
+ * and exclude functions; it can use wildcards. Returns true, when one filter
+ * item was successfully read and parsed. When object name contains \n chars,
+ * then more than one line from input file can be processed. Returns false when
+ * the filter file reaches EOF. In case of errors, the function wont return
+ * but will exit with an appropriate error message.
+ */
+static bool
+read_filter_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ StringInfoData line;
+
+ initStringInfo(&line);
+
+ if (pg_get_line_buf(fstate->fp, &line))
+ {
+ char *str = line.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno++;
+
+ (void) pg_strip_crlf(str);
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Skip empty lines or lines where the first non-whitespace character
+ * is a hash indicating a comment.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ /*
+ * First we expect sequence of two keywords, {include|exclude}
+ * followed by the object type to operate on.
+ */
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no filtercommand found (expected \"include\" or \"exclude\")");
+
+ if (size == 7 && pg_strncasecmp(keyword, "include", 7) == 0)
+ *is_include = true;
+ else if (size == 7 && pg_strncasecmp(keyword, "exclude", 7) == 0)
+ *is_include = false;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid filtercommand (expected \"include\" or \"exclude\")");
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no object type found (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ if (size == 5 && pg_strncasecmp(keyword, "table", 5) == 0)
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (size == 6 && pg_strncasecmp(keyword, "schema", 6) == 0)
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (size == 12 && pg_strncasecmp(keyword, "foreign_data", 12) == 0)
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (size == 4 && pg_strncasecmp(keyword, "data", 4) == 0)
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid object type (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ str = filter_get_pattern(fstate, str, objname);
+
+ /*
+ * Look for any content after the object identifier. Comments and
+ * whitespace are allowed, other content may indicate that the
+ * user needed to quote the object name so exit with an invalid
+ * format error.
+ */
+ while (isspace(*str))
+ str++;
+
+ if (*str != '\0' && *str != '#')
+ exit_invalid_filter_format(fstate,
+ "unexpected extra data after pattern");
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ free(line.data);
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m", fstate->filename);
+
+ if (fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ fatal("could not close filter file \"%s\": %m", fstate->filename);
+ }
+
+ exit_nicely(1);
+ }
+
+ free(line.data);
+
+ return false;
+}
+
+/*
+ * getFiltersFromFile - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getFiltersFromFile(const char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ fstate.filename = filename;
+ fstate.lineno = 0;
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate.fp = fopen(filename, "r");
+ if (!fstate.fp)
+ fatal("could not open filter file \"%s\": %m", filename);
+ }
+ else
+ fstate.fp = stdin;
+
+ while (read_filter_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ exit_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ if (fstate.fp != stdin)
+ {
+ if (fclose(fstate.fp) != 0)
+ fatal("could not close filter file \"%s\": %m", fstate.filename);
+ }
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..28ae6f1db8
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,399 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+
+use Config;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 49;
+
+my $tempdir = TestLib::tempdir;
+my $inputfile;
+
+my $node = PostgresNode->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres',
+ 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three_one(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres',
+ "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three_one VALUES('*** TABLE THREE_ONE ***')");
+
+#
+# Test interaction of correctly specified filter file
+#
+my ($cmd, $stdout, $stderr, $result);
+
+# Empty filterfile
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "\n # a comment and nothing more\n\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_(one|two|three|three_one)/m,
+ "tables dumped");
+
+# Test various combinations of whitespace, comments and correct filters
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "\t\n";
+print $inputfile " \t# another comment\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter patterns as well as comments and whitespace");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public\.table_three/m, "table three not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_three_one/m,
+ "table three_one not dumped");
+ok( $dump !~ qr/^COPY public\.table_one/m,
+ "content of table one is not included");
+ok($dump =~ qr/^COPY public\.table_two/m, "content of table two is included");
+
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with exclusion of a single table");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping tables with a wildcard pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_thre*\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with wildcard in pattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_two/m, "table two not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping table with multiline quoted tablename
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with multiline names requiring quoting");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding multiline quoted tablename from dump
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding an entire schema
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "exclude the public schema");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test including and excluding an entire schema by multiple filterfiles
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema public\n";
+close $inputfile;
+
+open my $alt_inputfile, '>', "$tempdir/inputfile2.txt"
+ or die "unable to open filterfile for writing";
+print $alt_inputfile "exclude schema public\n";
+close $alt_inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "--filter=$tempdir/inputfile2.txt", 'postgres'
+ ],
+ "exclude the public schema with multiple filters");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test dumping a table with a single leading newline on a row
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump nonexisting foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummyserver\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump foreign_data with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE SERVER dummyserver/m, "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude foreign_data dummy*\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/exclude filter is not allowed/,
+ "erroneously exclude foreign server");
+
+#########################################
+# Test broken input format
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid filtercommand/,
+ "invalid syntax: incorrect filtercommand");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid object type/,
+ "invalid syntax: invalid object type specified, should be table, schema, foreign_data or data"
+);
+
+# Test missing object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/missing object name/,
+ "invalid syntax: missing object identifier pattern");
+
+# Test adding extra content after the object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table one";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unexpected extra data/,
+ "invalid syntax: extra content after object identifier pattern");
+
+#########################################
+# Combined with --strict-names
+
+# First ensure that a matching filter works
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ "strict names with matching mattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "no table dumped");
+
+# Now append a pattern to the filter file which doesn't resolve
+open $inputfile, '>>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_nonexisting_name";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "inclusion of non-existing objects with --strict names");
--
2.24.3 (Apple Git-128)
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-10-05 12:37 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
2 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-10-05 12:37 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
út 5. 10. 2021 v 14:30 odesílatel Daniel Gustafsson <[email protected]>
napsal:
> > On 1 Oct 2021, at 15:19, Daniel Gustafsson <[email protected]> wrote:
>
> > I'm still not happy with the docs, I need to take another look there and
> see if
> > I make them more readable but otherwise I don't think there are any open
> issues
> > with this.
>
> Attached is a rebased version which has rewritten docs which I think are
> more
> in line with the pg_dump documentation. I've also added tests for
> --strict-names operation, as well subjected it to pgindent and pgperltidy.
>
> Unless there are objections, I think this is pretty much ready to go in.
>
great, thank you
Pavel
> --
> Daniel Gustafsson https://vmware.com/
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-10-05 14:39 Erik Rijkers <[email protected]>
parent: Daniel Gustafsson <[email protected]>
2 siblings, 0 replies; 181+ messages in thread
From: Erik Rijkers @ 2021-10-05 14:39 UTC (permalink / raw)
To: [email protected]
Op 05-10-2021 om 14:30 schreef Daniel Gustafsson:
>
> Unless there are objections, I think this is pretty much ready to go in.
Agreed. One typo:
'This keyword can only be with the exclude keyword.' should be
'This keyword can only be used with the exclude keyword.'
thanks,
Erik Rijkers
>
> --
> Daniel Gustafsson https://vmware.com/
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2021-10-27 09:15 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
2 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2021-10-27 09:15 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
út 5. 10. 2021 v 14:30 odesílatel Daniel Gustafsson <[email protected]>
napsal:
> > On 1 Oct 2021, at 15:19, Daniel Gustafsson <[email protected]> wrote:
>
> > I'm still not happy with the docs, I need to take another look there and
> see if
> > I make them more readable but otherwise I don't think there are any open
> issues
> > with this.
>
> Attached is a rebased version which has rewritten docs which I think are
> more
> in line with the pg_dump documentation. I've also added tests for
> --strict-names operation, as well subjected it to pgindent and pgperltidy.
>
> Unless there are objections, I think this is pretty much ready to go in.
>
I am sending a rebased version of patch pg_dump-filteropt-20211005.patch
with fixed regress tests and fixed documentation (reported by Erik).
I found another issue - the stringinfo line used in filter_get_pattern was
released too early - the line (memory) was used later in check of unexpected
chars after pattern string. I fixed it by moving this stringinfo buffer to
fstate structure. It can be shared by all routines, and it can be safely
released at
an end of filter processing, where we are sure, so these data can be free.
Regards
Pavel
> --
> Daniel Gustafsson https://vmware.com/
>
>
Attachments:
[text/x-patch] pg_dump-filteropt-20211027.patch (29.1K, ../../CAFj8pRB94jRM6GgqbEqYbadsG8=9E2wHLXEiUW5GkOvSiGuf0g@mail.gmail.com/3-pg_dump-filteropt-20211027.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 7682226b99..9245355686 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -789,6 +789,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal> use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ are ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1122,6 +1196,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1531,6 +1606,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index d1842edde0..f4f84bdce0 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -55,10 +55,12 @@
#include "catalog/pg_trigger_d.h"
#include "catalog/pg_type_d.h"
#include "common/connect.h"
+#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
#include "getopt_long.h"
+#include "lib/stringinfo.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
#include "pg_backup_db.h"
@@ -90,6 +92,29 @@ typedef enum OidOptions
zeroAsNone = 4
} OidOptions;
+/*
+ * State data for reading filter items from stream
+ */
+typedef struct
+{
+ FILE *fp;
+ const char *filename;
+ int lineno;
+ StringInfoData linebuff;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_DATA
+} FilterObjectType;
+
/* global decls */
static bool dosync = true; /* Issue fsync() to make dump durable on disk. */
@@ -308,6 +333,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void getFiltersFromFile(const char *filename, DumpOptions *dopt);
int
@@ -380,6 +406,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -613,6 +640,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ getFiltersFromFile(optarg, &dopt);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit_nicely(1);
@@ -1038,6 +1069,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18822,3 +18855,376 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse reloptions array");
}
+
+/*
+ * exit_invalid_filter_format - Emit error message, close the file and exit
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+static void
+exit_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+
+ if (fclose(fstate->fp) != 0)
+ fatal("could not close filter file \"%s\": %m", fstate->filename);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ exit_nicely(1);
+}
+
+/*
+ * filter_get_keyword - read the next filter keyword from buffer
+ *
+ * Search for keywords (limited to containing ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL, when the buffer is empty or first
+ * char is not alpha. The length of the found keyword is returned in the size
+ * parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* Set returnlength preemptively in case no keyword is found */
+ *size = 0;
+
+ /* Skip initial whitespace */
+ while (isspace(*ptr))
+ ptr++;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr++;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * filter_get_pattern - Read an object identifier pattern from the buffer
+ *
+ * Parses an object identifier pattern from the passed in buffer and sets
+ * objname to a string with object identifier pattern. Returns pointer to the
+ * first character after the pattern.
+ */
+static char *
+filter_get_pattern(FilterStateData *fstate,
+ char *str,
+ char **objname)
+{
+ /* Skip whitespace */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ exit_invalid_filter_format(fstate, "missing object name pattern");
+
+ /*
+ * If the object name pattern has been quoted we must take care parse out
+ * the entire quoted pattern, which may contain whitespace and can span
+ * over many lines.
+ */
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ Assert(fstate->linebuff.data);
+
+ if (!pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m",
+ fstate->filename);
+ if (fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ fatal("could not close filter file \"%s\": %m",
+ fstate->filename);
+ }
+
+ exit_nicely(1);
+ }
+
+ exit_invalid_filter_format(fstate, "unexpected end of file");
+ }
+
+ str = fstate->linebuff.data;
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ fstate->lineno++;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = pg_strdup(quoted_name->data);
+ destroyPQExpBuffer(quoted_name);
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* Simple variant, read to EOL or to first whitespace */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ return str;
+}
+
+/*
+ * read_filter_item - Read command/type/pattern triplet from filter file
+ *
+ * This will parse one filter item from the filter file, and while it is a row
+ * based format a pattern may span more than one line due to how object names
+ * can be constructed. The expected format of the filter file is:
+ *
+ * <command> <object_type> <pattern>
+ *
+ * Where command is "include" or "exclude", and object_type is one of: "table",
+ * "schema", "foreign_data" or "data". The pattern is either simple without any
+ * whitespace, or properly quoted in case there is whitespace in the object
+ * name. The pattern handling follows the same rules as other object include
+ * and exclude functions; it can use wildcards. Returns true, when one filter
+ * item was successfully read and parsed. When object name contains \n chars,
+ * then more than one line from input file can be processed. Returns false when
+ * the filter file reaches EOF. In case of errors, the function wont return
+ * but will exit with an appropriate error message.
+ */
+static bool
+read_filter_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ if (pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ char *str = fstate->linebuff.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno++;
+
+ (void) pg_strip_crlf(str);
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Skip empty lines or lines where the first non-whitespace character
+ * is a hash indicating a comment.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ /*
+ * First we expect sequence of two keywords, {include|exclude}
+ * followed by the object type to operate on.
+ */
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no filtercommand found (expected \"include\" or \"exclude\")");
+
+ if (size == 7 && pg_strncasecmp(keyword, "include", 7) == 0)
+ *is_include = true;
+ else if (size == 7 && pg_strncasecmp(keyword, "exclude", 7) == 0)
+ *is_include = false;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid filtercommand (expected \"include\" or \"exclude\")");
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no object type found (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ if (size == 5 && pg_strncasecmp(keyword, "table", 5) == 0)
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (size == 6 && pg_strncasecmp(keyword, "schema", 6) == 0)
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (size == 12 && pg_strncasecmp(keyword, "foreign_data", 12) == 0)
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (size == 4 && pg_strncasecmp(keyword, "data", 4) == 0)
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid object type (expected \"table\", \"schema\", \"foreign_data\" or \"data\")");
+
+ str = filter_get_pattern(fstate, str, objname);
+
+ /*
+ * Look for any content after the object identifier. Comments and
+ * whitespace are allowed, other content may indicate that the
+ * user needed to quote the object name so exit with an invalid
+ * format error.
+ */
+ while (isspace(*str))
+ str++;
+
+ if (*str != '\0' && *str != '#')
+ exit_invalid_filter_format(fstate,
+ "unexpected extra data after pattern");
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m", fstate->filename);
+
+ if (fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ fatal("could not close filter file \"%s\": %m", fstate->filename);
+ }
+
+ exit_nicely(1);
+ }
+
+ return false;
+}
+
+/*
+ * getFiltersFromFile - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getFiltersFromFile(const char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ fstate.filename = filename;
+ fstate.lineno = 0;
+ initStringInfo(&fstate.linebuff);
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate.fp = fopen(filename, "r");
+ if (!fstate.fp)
+ fatal("could not open filter file \"%s\": %m", filename);
+ }
+ else
+ fstate.fp = stdin;
+
+ while (read_filter_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ exit_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ free(fstate.linebuff.data);
+
+ if (fstate.fp != stdin)
+ {
+ if (fclose(fstate.fp) != 0)
+ fatal("could not close filter file \"%s\": %m", fstate.filename);
+ }
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..44bc3e8e96
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,398 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 49;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;;
+my $inputfile;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres',
+ 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three_one(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres',
+ "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three_one VALUES('*** TABLE THREE_ONE ***')");
+
+#
+# Test interaction of correctly specified filter file
+#
+my ($cmd, $stdout, $stderr, $result);
+
+# Empty filterfile
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "\n # a comment and nothing more\n\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_(one|two|three|three_one)/m,
+ "tables dumped");
+
+# Test various combinations of whitespace, comments and correct filters
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "\t\n";
+print $inputfile " \t# another comment\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter patterns as well as comments and whitespace");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public\.table_three/m, "table three not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_three_one/m,
+ "table three_one not dumped");
+ok( $dump !~ qr/^COPY public\.table_one/m,
+ "content of table one is not included");
+ok($dump =~ qr/^COPY public\.table_two/m, "content of table two is included");
+
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with exclusion of a single table");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping tables with a wildcard pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_thre*\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with wildcard in pattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_two/m, "table two not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping table with multiline quoted tablename
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with multiline names requiring quoting");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding multiline quoted tablename from dump
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding an entire schema
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "exclude the public schema");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test including and excluding an entire schema by multiple filterfiles
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema public\n";
+close $inputfile;
+
+open my $alt_inputfile, '>', "$tempdir/inputfile2.txt"
+ or die "unable to open filterfile for writing";
+print $alt_inputfile "exclude schema public\n";
+close $alt_inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "--filter=$tempdir/inputfile2.txt", 'postgres'
+ ],
+ "exclude the public schema with multiple filters");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test dumping a table with a single leading newline on a row
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump nonexisting foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummyserver\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump foreign_data with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE SERVER dummyserver/m, "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude foreign_data dummy*\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/exclude filter is not allowed/,
+ "erroneously exclude foreign server");
+
+#########################################
+# Test broken input format
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid filtercommand/,
+ "invalid syntax: incorrect filtercommand");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid object type/,
+ "invalid syntax: invalid object type specified, should be table, schema, foreign_data or data"
+);
+
+# Test missing object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/missing object name/,
+ "invalid syntax: missing object identifier pattern");
+
+# Test adding extra content after the object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table one";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unexpected extra data/,
+ "invalid syntax: extra content after object identifier pattern");
+
+#########################################
+# Combined with --strict-names
+
+# First ensure that a matching filter works
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ "strict names with matching mattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "no table dumped");
+
+# Now append a pattern to the filter file which doesn't resolve
+open $inputfile, '>>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_nonexisting_name";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "inclusion of non-existing objects with --strict names");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-07-13 20:49 Andrew Dunstan <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Andrew Dunstan @ 2022-07-13 20:49 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; Daniel Gustafsson <[email protected]>; +Cc: Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On 2022-04-25 Mo 13:39, Pavel Stehule wrote:
> Hi
>
> fresh rebase
>
>
If we're going to do this for pg_dump's include/exclude options,
shouldn't we also provide an equivalent facility for pg_dumpall's
--exclude-database option?
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-07-14 04:54 Pavel Stehule <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2022-07-14 04:54 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
st 13. 7. 2022 v 22:49 odesílatel Andrew Dunstan <[email protected]>
napsal:
>
> On 2022-04-25 Mo 13:39, Pavel Stehule wrote:
> > Hi
> >
> > fresh rebase
> >
> >
>
>
> If we're going to do this for pg_dump's include/exclude options,
> shouldn't we also provide an equivalent facility for pg_dumpall's
> --exclude-database option?
>
>
It has sense
Regards
Pavel
>
> cheers
>
>
> andrew
>
>
> --
> Andrew Dunstan
> EDB: https://www.enterprisedb.com
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-07-17 06:20 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2022-07-17 06:20 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Justin Pryzby <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
čt 14. 7. 2022 v 6:54 odesílatel Pavel Stehule <[email protected]>
napsal:
>
>
> st 13. 7. 2022 v 22:49 odesílatel Andrew Dunstan <[email protected]>
> napsal:
>
>>
>> On 2022-04-25 Mo 13:39, Pavel Stehule wrote:
>> > Hi
>> >
>> > fresh rebase
>> >
>> >
>>
>>
>> If we're going to do this for pg_dump's include/exclude options,
>> shouldn't we also provide an equivalent facility for pg_dumpall's
>> --exclude-database option?
>>
>>
> It has sense
>
The attached patch implements the --filter option for pg_dumpall and for
pg_restore too.
Regards
Pavel
>>
Attachments:
[text/x-patch] pg_dump-filteropt.patch (44.8K, ../../CAFj8pRDQmz86g03emFHqjHejB+gG_4KqrLij=L1L6Z-59dzL0g@mail.gmail.com/3-pg_dump-filteropt.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 5efb442b44..ba2920dbee 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal> use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ are ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 8a081f0080..137491340c 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,29 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpretted according to the same rules
+ like <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal> use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 526986eadb..5f16c4a333 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules like <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal> use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 2f524b09bf..373c4caa86 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -25,6 +25,7 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filter.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -42,8 +43,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o pg_backup_utils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o pg_backup_utils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/filter.c b/src/bin/pg_dump/filter.c
new file mode 100644
index 0000000000..b09caef1f3
--- /dev/null
+++ b/src/bin/pg_dump/filter.c
@@ -0,0 +1,369 @@
+/*-------------------------------------------------------------------------
+ *
+ * Implementation of simple filter file parser
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include "filter.h"
+
+#include "common/logging.h"
+#include "common/fe_memutils.h"
+#include "common/string.h"
+#include "lib/stringinfo.h"
+#include "pg_backup_utils.h"
+#include "pqexpbuffer.h"
+
+/*
+ * exit_invalid_filter_format - Emit error message, close the file and exit
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+exit_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+
+ if (fclose(fstate->fp) != 0)
+ pg_fatal("could not close filter file \"%s\": %m", fstate->filename);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ exit_nicely(1);
+}
+
+static const char *
+filter_object_type_name(FilterObjectType fot)
+{
+ switch (fot)
+ {
+ case FILTER_OBJECT_TYPE_NONE:
+ return "comment or empty line";
+ case FILTER_OBJECT_TYPE_DATA:
+ return "data";
+ case FILTER_OBJECT_TYPE_DATABASE:
+ return "database";
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ return "foreign data";
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ return "function";
+ case FILTER_OBJECT_TYPE_INDEX:
+ return "index";
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ return "schema";
+ case FILTER_OBJECT_TYPE_TABLE:
+ return "table";
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ return "trigger";
+ }
+
+ return "unknown object type";
+}
+
+/*
+ * Helper routine to reduce duplicated code
+ */
+void
+exit_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname,
+ FilterObjectType fot)
+{
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str,
+ "The application \"%s\" doesn't support filter for object type \"%s\".",
+ appname,
+ filter_object_type_name(fot));
+
+ exit_invalid_filter_format(fstate, str->data);
+}
+
+
+/*
+ * filter_get_keyword - read the next filter keyword from buffer
+ *
+ * Search for keywords (limited to containing ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL, when the buffer is empty or first
+ * char is not alpha. The length of the found keyword is returned in the size
+ * parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* Set returnlength preemptively in case no keyword is found */
+ *size = 0;
+
+ /* Skip initial whitespace */
+ while (isspace(*ptr))
+ ptr++;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr++;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * filter_get_pattern - Read an object identifier pattern from the buffer
+ *
+ * Parses an object identifier pattern from the passed in buffer and sets
+ * objname to a string with object identifier pattern. Returns pointer to the
+ * first character after the pattern.
+ */
+static char *
+filter_get_pattern(FilterStateData *fstate,
+ char *str,
+ char **objname)
+{
+ /* Skip whitespace */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ exit_invalid_filter_format(fstate, "missing object name pattern");
+
+ /*
+ * If the object name pattern has been quoted we must take care parse out
+ * the entire quoted pattern, which may contain whitespace and can span
+ * over many lines.
+ */
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ Assert(fstate->linebuff.data);
+
+ if (!pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m",
+ fstate->filename);
+ if (fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ pg_fatal("could not close filter file \"%s\": %m",
+ fstate->filename);
+ }
+
+ exit_nicely(1);
+ }
+
+ exit_invalid_filter_format(fstate, "unexpected end of file");
+ }
+
+ str = fstate->linebuff.data;
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ fstate->lineno++;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = pg_strdup(quoted_name->data);
+ destroyPQExpBuffer(quoted_name);
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* Simple variant, read to EOL or to first whitespace */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ return str;
+}
+
+/*
+ * read_filter_item - Read command/type/pattern triplet from filter file
+ *
+ * This will parse one filter item from the filter file, and while it is a row
+ * based format a pattern may span more than one line due to how object names
+ * can be constructed. The expected format of the filter file is:
+ *
+ * <command> <object_type> <pattern>
+ *
+ * Where command is "include" or "exclude", and object_type is one of: "table",
+ * "schema", "foreign_data", "data", "database", "function", "trigger" or "index".
+ * The pattern is either simple without any whitespace, or properly quoted
+ * in case there is whitespace in the object name. The pattern handling follows
+ * the same rules as other object include and exclude functions; it can use
+ * wildcards. Returns true, when one filter item was successfully read and
+ * parsed. When object name contains \n chars, then more than one line from
+ * input file can be processed. Returns false when the filter file reaches EOF.
+ * In case of errors, the function wont return but will exit with an appropriate
+ * error message.
+ */
+bool
+read_filter_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ if (pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ char *str = fstate->linebuff.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno++;
+
+ (void) pg_strip_crlf(str);
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Skip empty lines or lines where the first non-whitespace character
+ * is a hash indicating a comment.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ /*
+ * First we expect sequence of two keywords, {include|exclude}
+ * followed by the object type to operate on.
+ */
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate,
+ "no filter command found (expected \"include\" or \"exclude\")");
+
+ if (size == 7 && pg_strncasecmp(keyword, "include", 7) == 0)
+ *is_include = true;
+ else if (size == 7 && pg_strncasecmp(keyword, "exclude", 7) == 0)
+ *is_include = false;
+ else
+ exit_invalid_filter_format(fstate,
+ "invalid filter command (expected \"include\" or \"exclude\")");
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ exit_invalid_filter_format(fstate, "missing filter object type");
+
+ if (size == 4 && pg_strncasecmp(keyword, "data", 4) == 0)
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else if (size == 8 && pg_strncasecmp(keyword, "database", 8) == 0)
+ *objtype = FILTER_OBJECT_TYPE_DATABASE;
+ else if (size == 12 && pg_strncasecmp(keyword, "foreign_data", 12) == 0)
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (size == 8 && pg_strncasecmp(keyword, "function", 8) == 0)
+ *objtype = FILTER_OBJECT_TYPE_FUNCTION;
+ else if (size == 5 && pg_strncasecmp(keyword, "index", 5) == 0)
+ *objtype = FILTER_OBJECT_TYPE_INDEX;
+ else if (size == 6 && pg_strncasecmp(keyword, "schema", 6) == 0)
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (size == 5 && pg_strncasecmp(keyword, "table", 5) == 0)
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (size == 7 && pg_strncasecmp(keyword, "trigger", 7) == 0)
+ *objtype = FILTER_OBJECT_TYPE_TRIGGER;
+ else
+ {
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str, "unsupported filter object type: \"%.*s\"", size, keyword);
+ exit_invalid_filter_format(fstate, str->data);
+ }
+
+ str = filter_get_pattern(fstate, str, objname);
+
+ /*
+ * Look for any content after the object identifier. Comments and
+ * whitespace are allowed, other content may indicate that the
+ * user needed to quote the object name so exit with an invalid
+ * format error.
+ */
+ while (isspace(*str))
+ str++;
+
+ if (*str != '\0' && *str != '#')
+ exit_invalid_filter_format(fstate,
+ "unexpected extra data after pattern");
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m", fstate->filename);
+
+ if (fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ pg_fatal("could not close filter file \"%s\": %m", fstate->filename);
+ }
+
+ exit_nicely(1);
+ }
+
+ return false;
+}
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..e4a1a74b10
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,51 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+#include "c.h"
+
+#include "lib/stringinfo.h"
+
+/*
+ * State data for reading filter items from stream
+ */
+typedef struct
+{
+ FILE *fp;
+ const char *filename;
+ int lineno;
+ StringInfoData linebuff;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern void exit_invalid_filter_format(FilterStateData *fstate, char *message);
+extern void exit_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname, FilterObjectType fot);
+extern bool read_filter_item(FilterStateData *fstate, bool *is_include, char **objname, FilterObjectType *objtype);
+
+#endif
\ No newline at end of file
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e4fdb6b75b..cbe712226a 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -58,6 +58,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -317,6 +318,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void getFiltersFromFile(const char *filename, DumpOptions *dopt);
int
@@ -389,6 +391,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -622,6 +625,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ getFiltersFromFile(optarg, &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1027,6 +1034,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18130,3 +18139,93 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+/*
+ * getFiltersFromFile - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getFiltersFromFile(const char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ fstate.filename = filename;
+ fstate.lineno = 0;
+ initStringInfo(&fstate.linebuff);
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate.fp = fopen(filename, "r");
+ if (!fstate.fp)
+ pg_fatal("could not open filter file \"%s\": %m", filename);
+ }
+ else
+ fstate.fp = stdin;
+
+ while (read_filter_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments and empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ exit_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else
+ exit_unsupported_filter_object_type(&fstate, "pg_dump", objtype);
+
+ if (objname)
+ free(objname);
+ }
+
+ free(fstate.linebuff.data);
+
+ if (fstate.fp != stdin)
+ {
+ if (fclose(fstate.fp) != 0)
+ pg_fatal("could not close filter file \"%s\": %m", fstate.filename);
+ }
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 26d3d53809..1f75e40b49 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -25,6 +25,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -56,6 +57,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -133,6 +135,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -335,6 +338,11 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ getDatabaseExcludeFiltersFromFile(optarg,
+ &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -628,6 +636,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1748,3 +1757,60 @@ dumpTimestamp(const char *msg)
if (strftime(buf, sizeof(buf), PGDUMP_STRFTIME_FMT, localtime(&now)) != 0)
fprintf(OPF, "-- %s %s\n\n", msg, buf);
}
+
+/*
+ * getDatabaseFiltersFromFile - retrieve database identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *pattern)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ fstate.filename = filename;
+ fstate.lineno = 0;
+ initStringInfo(&fstate.linebuff);
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate.fp = fopen(filename, "r");
+ if (!fstate.fp)
+ pg_fatal("could not open filter file \"%s\": %m", filename);
+ }
+ else
+ fstate.fp = stdin;
+
+ while (read_filter_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATABASE)
+ {
+ if (!is_include)
+ simple_string_list_append(pattern, objname);
+ else
+ exit_invalid_filter_format(&fstate,
+ "include database filter is not suppported");
+ }
+ else
+ exit_unsupported_filter_object_type(&fstate, "pg_dumpall", objtype);
+
+ if (objname)
+ free(objname);
+ }
+
+ free(fstate.linebuff.data);
+
+ if (fstate.fp != stdin)
+ {
+ if (fclose(fstate.fp) != 0)
+ pg_fatal("could not close filter file \"%s\": %m", fstate.filename);
+ }
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..31634c2396 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,11 +47,13 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
static void usage(const char *progname);
+static void getFiltersFromFile(const char *filename, RestoreOptions *dopt);
int
main(int argc, char **argv)
@@ -123,6 +125,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +289,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ getFiltersFromFile(optarg, opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +470,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +502,108 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * getFiltersFromFile - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getFiltersFromFile(const char *filename, RestoreOptions *opts)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ fstate.filename = filename;
+ fstate.lineno = 0;
+ initStringInfo(&fstate.linebuff);
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate.fp = fopen(filename, "r");
+ if (!fstate.fp)
+ pg_fatal("could not open filter file \"%s\": %m", filename);
+ }
+ else
+ fstate.fp = stdin;
+
+ while (read_filter_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments or empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_FUNCTION)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ }
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_INDEX)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ }
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ simple_string_list_append(&opts->schemaNames, objname);
+ else
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ }
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TRIGGER)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, optarg);
+ }
+ else
+ exit_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ }
+ else
+ exit_unsupported_filter_object_type(&fstate, "pg_restore", objtype);
+
+ if (objname)
+ free(objname);
+ }
+
+ free(fstate.linebuff.data);
+
+ if (fstate.fp != stdin)
+ {
+ if (fclose(fstate.fp) != 0)
+ pg_fatal("could not close filter file \"%s\": %m", fstate.filename);
+ }
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..246880f6ec
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,398 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 49;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;;
+my $inputfile;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres',
+ 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three_one(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres',
+ "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three_one VALUES('*** TABLE THREE_ONE ***')");
+
+#
+# Test interaction of correctly specified filter file
+#
+my ($cmd, $stdout, $stderr, $result);
+
+# Empty filterfile
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "\n # a comment and nothing more\n\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_(one|two|three|three_one)/m,
+ "tables dumped");
+
+# Test various combinations of whitespace, comments and correct filters
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "\t\n";
+print $inputfile " \t# another comment\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter patterns as well as comments and whitespace");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public\.table_three/m, "table three not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_three_one/m,
+ "table three_one not dumped");
+ok( $dump !~ qr/^COPY public\.table_one/m,
+ "content of table one is not included");
+ok($dump =~ qr/^COPY public\.table_two/m, "content of table two is included");
+
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with exclusion of a single table");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping tables with a wildcard pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_thre*\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with wildcard in pattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_two/m, "table two not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping table with multiline quoted tablename
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with multiline names requiring quoting");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding multiline quoted tablename from dump
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding an entire schema
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "exclude the public schema");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test including and excluding an entire schema by multiple filterfiles
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema public\n";
+close $inputfile;
+
+open my $alt_inputfile, '>', "$tempdir/inputfile2.txt"
+ or die "unable to open filterfile for writing";
+print $alt_inputfile "exclude schema public\n";
+close $alt_inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "--filter=$tempdir/inputfile2.txt", 'postgres'
+ ],
+ "exclude the public schema with multiple filters");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test dumping a table with a single leading newline on a row
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump nonexisting foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummyserver\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump foreign_data with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE SERVER dummyserver/m, "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude foreign_data dummy*\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/exclude filter is not allowed/,
+ "erroneously exclude foreign server");
+
+#########################################
+# Test broken input format
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: invalid object type specified, should be table, schema, foreign_data or data"
+);
+
+# Test missing object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/missing object name/,
+ "invalid syntax: missing object identifier pattern");
+
+# Test adding extra content after the object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table one";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unexpected extra data/,
+ "invalid syntax: extra content after object identifier pattern");
+
+#########################################
+# Combined with --strict-names
+
+# First ensure that a matching filter works
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ "strict names with matching mattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "no table dumped");
+
+# Now append a pattern to the filter file which doesn't resolve
+open $inputfile, '>>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_nonexisting_name";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "inclusion of non-existing objects with --strict names");
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-07-17 14:01 Justin Pryzby <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Justin Pryzby @ 2022-07-17 14:01 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Thanks for updating the patch.
This failed to build on windows.
http://cfbot.cputube.org/pavel-stehule.html
Some more comments inline.
On Sun, Jul 17, 2022 at 08:20:47AM +0200, Pavel Stehule wrote:
> The attached patch implements the --filter option for pg_dumpall and for
> pg_restore too.
> diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
> index 5efb442b44..ba2920dbee 100644
> --- a/doc/src/sgml/ref/pg_dump.sgml
> +++ b/doc/src/sgml/ref/pg_dump.sgml
> @@ -779,6 +779,80 @@ PostgreSQL documentation
> </listitem>
> </varlistentry>
>
> + <varlistentry>
> + <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
> + <listitem>
> + <para>
> + Specify a filename from which to read patterns for objects to include
> + or exclude from the dump. The patterns are interpreted according to the
> + same rules as the corresponding options:
> + <option>-t</option>/<option>--table</option> for tables,
> + <option>-n</option>/<option>--schema</option> for schemas,
> + <option>--include-foreign-data</option> for data on foreign servers and
> + <option>--exclude-table-data</option> for table data.
> + To read from <literal>STDIN</literal> use <filename>-</filename> as the
STDIN comma
> + <para>
> + Lines starting with <literal>#</literal> are considered comments and
> + are ignored. Comments can be placed after filter as well. Blank lines
change "are ignored" to "ignored", I think.
> diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
> index 8a081f0080..137491340c 100644
> --- a/doc/src/sgml/ref/pg_dumpall.sgml
> +++ b/doc/src/sgml/ref/pg_dumpall.sgml
> @@ -122,6 +122,29 @@ PostgreSQL documentation
> </listitem>
> </varlistentry>
>
> + <varlistentry>
> + <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
> + <listitem>
> + <para>
> + Specify a filename from which to read patterns for databases excluded
> + from dump. The patterns are interpretted according to the same rules
> + like <option>--exclude-database</option>.
same rules *as*
> + To read from <literal>STDIN</literal> use <filename>-</filename> as the
comma
> + filename. The <option>--filter</option> option can be specified in
> + conjunction with the above listed options for including or excluding
For dumpall, remove "for including or"
change "above listed options" to "exclude-database" ?
> diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
> index 526986eadb..5f16c4a333 100644
> --- a/doc/src/sgml/ref/pg_restore.sgml
> +++ b/doc/src/sgml/ref/pg_restore.sgml
> @@ -188,6 +188,31 @@ PostgreSQL documentation
> </listitem>
> </varlistentry>
>
> + <varlistentry>
> + <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
> + <listitem>
> + <para>
> + Specify a filename from which to read patterns for objects excluded
> + or included from restore. The patterns are interpretted according to the
> + same rules like <option>--schema</option>, <option>--exclude-schema</option>,
s/like/as/
> + <option>--function</option>, <option>--index</option>, <option>--table</option>
> + or <option>--trigger</option>.
> + To read from <literal>STDIN</literal> use <filename>-</filename> as the
STDIN comma
> +/*
> + * filter_get_keyword - read the next filter keyword from buffer
> + *
> + * Search for keywords (limited to containing ascii alphabetic characters) in
remove "containing"
> + /*
> + * If the object name pattern has been quoted we must take care parse out
> + * the entire quoted pattern, which may contain whitespace and can span
> + * over many lines.
quoted comma
*to parse
remove "over"
> + * The pattern is either simple without any whitespace, or properly quoted
double space
> + * in case there is whitespace in the object name. The pattern handling follows
s/is/may be/
> + if (size == 7 && pg_strncasecmp(keyword, "include", 7) == 0)
> + *is_include = true;
> + else if (size == 7 && pg_strncasecmp(keyword, "exclude", 7) == 0)
> + *is_include = false;
Can't you write strncasecmp(keyword, "include", size) to avoid hardcoding "7" ?
> +
> + if (size == 4 && pg_strncasecmp(keyword, "data", 4) == 0)
> + *objtype = FILTER_OBJECT_TYPE_DATA;
> + else if (size == 8 && pg_strncasecmp(keyword, "database", 8) == 0)
> + *objtype = FILTER_OBJECT_TYPE_DATABASE;
> + else if (size == 12 && pg_strncasecmp(keyword, "foreign_data", 12) == 0)
> + *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
> + else if (size == 8 && pg_strncasecmp(keyword, "function", 8) == 0)
> + *objtype = FILTER_OBJECT_TYPE_FUNCTION;
> + else if (size == 5 && pg_strncasecmp(keyword, "index", 5) == 0)
> + *objtype = FILTER_OBJECT_TYPE_INDEX;
> + else if (size == 6 && pg_strncasecmp(keyword, "schema", 6) == 0)
> + *objtype = FILTER_OBJECT_TYPE_SCHEMA;
> + else if (size == 5 && pg_strncasecmp(keyword, "table", 5) == 0)
> + *objtype = FILTER_OBJECT_TYPE_TABLE;
> + else if (size == 7 && pg_strncasecmp(keyword, "trigger", 7) == 0)
> + *objtype = FILTER_OBJECT_TYPE_TRIGGER;
Avoid hardcoding these constants.
> diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
> new file mode 100644
> index 0000000000..e4a1a74b10
> --- /dev/null
> +++ b/src/bin/pg_dump/filter.h
...
> \ No newline at end of file
:(
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-07-18 18:48 Pavel Stehule <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2022-07-18 18:48 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
ne 17. 7. 2022 v 16:01 odesílatel Justin Pryzby <[email protected]>
napsal:
> Thanks for updating the patch.
>
> This failed to build on windows.
> http://cfbot.cputube.org/pavel-stehule.html
>
>
Yes, there was a significant problem with the function exit_nicely, that is
differently implemented in pg_dump and pg_dumpall.
> Some more comments inline.
>
> On Sun, Jul 17, 2022 at 08:20:47AM +0200, Pavel Stehule wrote:
> > The attached patch implements the --filter option for pg_dumpall and for
> > pg_restore too.
>
> > diff --git a/doc/src/sgml/ref/pg_dump.sgml
> b/doc/src/sgml/ref/pg_dump.sgml
> > index 5efb442b44..ba2920dbee 100644
> > --- a/doc/src/sgml/ref/pg_dump.sgml
> > +++ b/doc/src/sgml/ref/pg_dump.sgml
> > @@ -779,6 +779,80 @@ PostgreSQL documentation
> > </listitem>
> > </varlistentry>
> >
> > + <varlistentry>
> > + <term><option>--filter=<replaceable
> class="parameter">filename</replaceable></option></term>
> > + <listitem>
> > + <para>
> > + Specify a filename from which to read patterns for objects to
> include
> > + or exclude from the dump. The patterns are interpreted
> according to the
> > + same rules as the corresponding options:
> > + <option>-t</option>/<option>--table</option> for tables,
> > + <option>-n</option>/<option>--schema</option> for schemas,
> > + <option>--include-foreign-data</option> for data on foreign
> servers and
> > + <option>--exclude-table-data</option> for table data.
> > + To read from <literal>STDIN</literal> use
> <filename>-</filename> as the
STDIN comma
>
fixed
>
> > + <para>
> > + Lines starting with <literal>#</literal> are considered
> comments and
> > + are ignored. Comments can be placed after filter as well. Blank
> lines
>
> change "are ignored" to "ignored", I think.
>
changed
>
> > diff --git a/doc/src/sgml/ref/pg_dumpall.sgml
> b/doc/src/sgml/ref/pg_dumpall.sgml
> > index 8a081f0080..137491340c 100644
> > --- a/doc/src/sgml/ref/pg_dumpall.sgml
> > +++ b/doc/src/sgml/ref/pg_dumpall.sgml
> > @@ -122,6 +122,29 @@ PostgreSQL documentation
> > </listitem>
> > </varlistentry>
> >
> > + <varlistentry>
> > + <term><option>--filter=<replaceable
> class="parameter">filename</replaceable></option></term>
> > + <listitem>
> > + <para>
> > + Specify a filename from which to read patterns for databases
> excluded
> > + from dump. The patterns are interpretted according to the same
> rules
> > + like <option>--exclude-database</option>.
>
> same rules *as*
>
fixed
>
> > + To read from <literal>STDIN</literal> use
> <filename>-</filename> as the
>
> comma
>
fixed
>
> > + filename. The <option>--filter</option> option can be
> specified in
> > + conjunction with the above listed options for including or
> excluding
>
> For dumpall, remove "for including or"
> change "above listed options" to "exclude-database" ?
>
fixed
>
> > diff --git a/doc/src/sgml/ref/pg_restore.sgml
> b/doc/src/sgml/ref/pg_restore.sgml
> > index 526986eadb..5f16c4a333 100644
> > --- a/doc/src/sgml/ref/pg_restore.sgml
> > +++ b/doc/src/sgml/ref/pg_restore.sgml
> > @@ -188,6 +188,31 @@ PostgreSQL documentation
> > </listitem>
> > </varlistentry>
> >
> > + <varlistentry>
> > + <term><option>--filter=<replaceable
> class="parameter">filename</replaceable></option></term>
> > + <listitem>
> > + <para>
> > + Specify a filename from which to read patterns for objects
> excluded
> > + or included from restore. The patterns are interpretted
> according to the
> > + same rules like <option>--schema</option>,
> <option>--exclude-schema</option>,
>
> s/like/as/
>
changed
>
> > + <option>--function</option>, <option>--index</option>,
> <option>--table</option>
> > + or <option>--trigger</option>.
> > + To read from <literal>STDIN</literal> use
> <filename>-</filename> as the
>
> STDIN comma
>
fixed
>
> > +/*
> > + * filter_get_keyword - read the next filter keyword from buffer
> > + *
> > + * Search for keywords (limited to containing ascii alphabetic
> characters) in
>
> remove "containing"
>
fixed
>
> > + /*
> > + * If the object name pattern has been quoted we must take care
> parse out
> > + * the entire quoted pattern, which may contain whitespace and can
> span
> > + * over many lines.
>
> quoted comma
> *to parse
> remove "over"
>
fixed
>
> > + * The pattern is either simple without any whitespace, or properly
> quoted
>
> double space
>
fixed
>
> > + * in case there is whitespace in the object name. The pattern handling
> follows
>
> s/is/may be/
>
fixed
>
> > + if (size == 7 && pg_strncasecmp(keyword,
> "include", 7) == 0)
> > + *is_include = true;
> > + else if (size == 7 && pg_strncasecmp(keyword,
> "exclude", 7) == 0)
> > + *is_include = false;
>
> Can't you write strncasecmp(keyword, "include", size) to avoid hardcoding
> "7" ?
>
I need to compare the size of the keyword with expected size, but I can use
strlen(conststr). I wrote new macro is_keyword_str to fix this issue
fixed
>
> > +
> > + if (size == 4 && pg_strncasecmp(keyword, "data",
> 4) == 0)
> > + *objtype = FILTER_OBJECT_TYPE_DATA;
> > + else if (size == 8 && pg_strncasecmp(keyword,
> "database", 8) == 0)
> > + *objtype = FILTER_OBJECT_TYPE_DATABASE;
> > + else if (size == 12 && pg_strncasecmp(keyword,
> "foreign_data", 12) == 0)
> > + *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
> > + else if (size == 8 && pg_strncasecmp(keyword,
> "function", 8) == 0)
> > + *objtype = FILTER_OBJECT_TYPE_FUNCTION;
> > + else if (size == 5 && pg_strncasecmp(keyword,
> "index", 5) == 0)
> > + *objtype = FILTER_OBJECT_TYPE_INDEX;
> > + else if (size == 6 && pg_strncasecmp(keyword,
> "schema", 6) == 0)
> > + *objtype = FILTER_OBJECT_TYPE_SCHEMA;
> > + else if (size == 5 && pg_strncasecmp(keyword,
> "table", 5) == 0)
> > + *objtype = FILTER_OBJECT_TYPE_TABLE;
> > + else if (size == 7 && pg_strncasecmp(keyword,
> "trigger", 7) == 0)
> > + *objtype = FILTER_OBJECT_TYPE_TRIGGER;
>
> Avoid hardcoding these constants.
>
fixed
>
> > diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
> > new file mode 100644
> > index 0000000000..e4a1a74b10
> > --- /dev/null
> > +++ b/src/bin/pg_dump/filter.h
> ...
> > \ No newline at end of file
>
fixed
updated patch attached
Regards
Pavel
>
> :(
>
Attachments:
[text/x-patch] pg_dump-filteropt.patch (45.8K, ../../CAFj8pRDBvqsw=Y3UwLce=JRS9fce3Wg-r8YmxA6HzPqqM5sREA@mail.gmail.com/3-pg_dump-filteropt.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 5efb442b44..5bcbcf17eb 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 8a081f0080..799910a487 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpretted according to the same rules
+ as <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for excluding databases,
+ and can also be specified more than once for multiple filter files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 526986eadb..d3ca0b36d5 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules as <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 2f524b09bf..d8c1babc26 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -25,6 +25,7 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filter.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -42,8 +43,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/filter.c b/src/bin/pg_dump/filter.c
new file mode 100644
index 0000000000..5a76fd57f1
--- /dev/null
+++ b/src/bin/pg_dump/filter.c
@@ -0,0 +1,431 @@
+/*-------------------------------------------------------------------------
+ *
+ * Implementation of simple filter file parser
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include "filter.h"
+
+#include "common/logging.h"
+#include "common/fe_memutils.h"
+#include "common/string.h"
+#include "lib/stringinfo.h"
+#include "pqexpbuffer.h"
+
+#define is_keyword_str(cstr, str, bytes) \
+ ((strlen(cstr) == bytes) && (pg_strncasecmp(cstr, str, bytes) == 0))
+
+/*
+ * Following routines are called from pg_dump, pg_dumpall and pg_restore.
+ * Unfortunatelly, implementation of exit_nicely in pg_dump and pg_restore
+ * is different from implementation of this rutine in pg_dumpall. So instead
+ * direct calling exit_nicely we have to return some error flag (in this
+ * case NULL), and exit_nicelly will be executed from caller's routine.
+ */
+
+/*
+ * Simple routines - just don't repeat same code
+ *
+ * Returns true, when filter's file is opened
+ */
+bool
+filter_init(FilterStateData *fstate, const char *filename)
+{
+ fstate->filename = filename;
+ fstate->lineno = 0;
+ initStringInfo(&fstate->linebuff);
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate->fp = fopen(filename, "r");
+ if (!fstate->fp)
+ {
+ pg_log_error("could not open filter file \"%s\": %m", filename);
+ return false;
+ }
+ }
+ else
+ fstate->fp = stdin;
+
+ fstate->is_error = false;
+
+ return true;
+}
+
+/*
+ * Release allocated sources for filter
+ */
+void
+filter_free_sources(FilterStateData *fstate)
+{
+ free(fstate->linebuff.data);
+ fstate->linebuff.data = NULL;
+
+ if (fstate->fp && fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ pg_log_error("could not close filter file \"%s\": %m", fstate->filename);
+
+ fstate->fp = NULL;
+ }
+}
+
+/*
+ * log_format_error - Emit error message
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ fstate->is_error = true;
+}
+
+static const char *
+filter_object_type_name(FilterObjectType fot)
+{
+ switch (fot)
+ {
+ case FILTER_OBJECT_TYPE_NONE:
+ return "comment or empty line";
+ case FILTER_OBJECT_TYPE_DATA:
+ return "data";
+ case FILTER_OBJECT_TYPE_DATABASE:
+ return "database";
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ return "foreign data";
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ return "function";
+ case FILTER_OBJECT_TYPE_INDEX:
+ return "index";
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ return "schema";
+ case FILTER_OBJECT_TYPE_TABLE:
+ return "table";
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ return "trigger";
+ }
+
+ return "unknown object type";
+}
+
+/*
+ * Helper routine to reduce duplicated code
+ */
+void
+log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname,
+ FilterObjectType fot)
+{
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str,
+ "The application \"%s\" doesn't support filter for object type \"%s\".",
+ appname,
+ filter_object_type_name(fot));
+
+ log_invalid_filter_format(fstate, str->data);
+}
+
+/*
+ * filter_get_keyword - read the next filter keyword from buffer
+ *
+ * Search for keywords (limited to ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL, when the buffer is empty or first
+ * char is not alpha. The length of the found keyword is returned in the size
+ * parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* Set returnlength preemptively in case no keyword is found */
+ *size = 0;
+
+ /* Skip initial whitespace */
+ while (isspace(*ptr))
+ ptr++;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr++;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * filter_get_pattern - Read an object identifier pattern from the buffer
+ *
+ * Parses an object identifier pattern from the passed in buffer and sets
+ * objname to a string with object identifier pattern. Returns pointer to the
+ * first character after the pattern.
+ */
+static char *
+filter_get_pattern(FilterStateData *fstate,
+ char *str,
+ char **objname)
+{
+ /* Skip whitespace */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ {
+ log_invalid_filter_format(fstate, "missing object name pattern");
+ return NULL;
+ }
+
+ /*
+ * If the object name pattern has been quoted, we must take care to parse
+ * out the entire quoted pattern, which may contain whitespace and can span
+ * many lines.
+ */
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ Assert(fstate->linebuff.data);
+
+ if (!pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m",
+ fstate->filename);
+ fstate->is_error = true;
+ }
+ else
+ log_invalid_filter_format(fstate, "unexpected end of file");
+
+ return NULL;
+ }
+
+ str = fstate->linebuff.data;
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ fstate->lineno++;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = pg_strdup(quoted_name->data);
+ destroyPQExpBuffer(quoted_name);
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* Simple variant, read to EOL or to first whitespace */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ return str;
+}
+
+/*
+ * read_filter_item - Read command/type/pattern triplet from filter file
+ *
+ * This will parse one filter item from the filter file, and while it is a
+ * row based format a pattern may span more than one line due to how object
+ * names can be constructed. The expected format of the filter file is:
+ *
+ * <command> <object_type> <pattern>
+ *
+ * Where command is "include" or "exclude", and object_type is one of:
+ * "table", "schema", "foreign_data", "data", "database", "function",
+ * "trigger" or "index". The pattern is either simple without any
+ * whitespace, or properly quoted in case there may be whitespace in the
+ * object name. The pattern handling follows the same rules as other object
+ * include and exclude functions; it can use wildcards. Returns true, when
+ * one filter item was successfully read and parsed. When object name
+ * contains \n chars, then more than one line from input file can be
+ * processed. Returns false when the filter file reaches EOF. In case of
+ * errors, the function wont return but will exit with an appropriate error
+ * message.
+ */
+bool
+filter_read_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ Assert(!fstate->is_error);
+
+ if (pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ char *str = fstate->linebuff.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno++;
+
+ (void) pg_strip_crlf(str);
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Skip empty lines or lines where the first non-whitespace character
+ * is a hash indicating a comment.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ /*
+ * First we expect sequence of two keywords, {include|exclude}
+ * followed by the object type to operate on.
+ */
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate,
+ "no filter command found (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ if (is_keyword_str("include", keyword, size))
+ *is_include = true;
+ else if (is_keyword_str("exclude", keyword, size))
+ *is_include = false;
+ else
+ {
+ log_invalid_filter_format(fstate,
+ "invalid filter command (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate, "missing filter object type");
+ return false;
+ }
+
+ if (is_keyword_str("data", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else if (is_keyword_str("database", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_DATABASE;
+ else if (is_keyword_str("foreign_data",keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (is_keyword_str("function", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FUNCTION;
+ else if (is_keyword_str("index", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_INDEX;
+ else if (is_keyword_str("schema", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (is_keyword_str("table", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (is_keyword_str("trigger", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TRIGGER;
+ else
+ {
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str, "unsupported filter object type: \"%.*s\"", size, keyword);
+ log_invalid_filter_format(fstate, str->data);
+ return false;
+ }
+
+ str = filter_get_pattern(fstate, str, objname);
+ if (!str)
+ return false;
+
+ /*
+ * Look for any content after the object identifier. Comments and
+ * whitespace are allowed, other content may indicate that the
+ * user needed to quote the object name so exit with an invalid
+ * format error.
+ */
+ while (isspace(*str))
+ str++;
+
+ if (*str != '\0' && *str != '#')
+ {
+ log_invalid_filter_format(fstate,
+ "unexpected extra data after pattern");
+ return false;
+ }
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m", fstate->filename);
+ fstate->is_error = true;
+ }
+
+ return false;
+}
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..f2503286aa
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,55 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+#include "c.h"
+
+#include "lib/stringinfo.h"
+
+/*
+ * State data for reading filter items from stream
+ */
+typedef struct
+{
+ FILE *fp;
+ const char *filename;
+ int lineno;
+ StringInfoData linebuff;
+ bool is_error;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern bool filter_init(FilterStateData *fstate, const char *filename);
+extern void filter_free_sources(FilterStateData *fstate);
+
+extern void log_invalid_filter_format(FilterStateData *fstate, char *message);
+extern void log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname, FilterObjectType fot);
+extern bool filter_read_item(FilterStateData *fstate, bool *is_include, char **objname, FilterObjectType *objtype);
+
+#endif
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e4fdb6b75b..040f5b07ed 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -58,6 +58,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -317,6 +318,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void getFiltersFromFile(const char *filename, DumpOptions *dopt);
int
@@ -389,6 +391,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -622,6 +625,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ getFiltersFromFile(optarg, &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1027,6 +1034,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18130,3 +18139,89 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+/*
+ * getFiltersFromFile - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getFiltersFromFile(const char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments and empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ {
+ log_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ break;
+ }
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dump", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free_sources(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 26d3d53809..d87d29093f 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -25,6 +25,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -56,6 +57,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -133,6 +135,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -335,6 +338,11 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ getDatabaseExcludeFiltersFromFile(optarg,
+ &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -628,6 +636,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1735,7 +1744,6 @@ executeCommand(PGconn *conn, const char *query)
PQclear(res);
}
-
/*
* dumpTimestamp
*/
@@ -1748,3 +1756,53 @@ dumpTimestamp(const char *msg)
if (strftime(buf, sizeof(buf), PGDUMP_STRFTIME_FMT, localtime(&now)) != 0)
fprintf(OPF, "-- %s %s\n\n", msg, buf);
}
+
+/*
+ * getDatabaseFiltersFromFile - retrieve database identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *pattern)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATABASE)
+ {
+ if (!is_include)
+ simple_string_list_append(pattern, objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "include database filter is not suppported");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dumpall", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free_sources(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..ab8525cafd 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,11 +47,13 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
static void usage(const char *progname);
+static void getFiltersFromFile(const char *filename, RestoreOptions *dopt);
int
main(int argc, char **argv)
@@ -123,6 +125,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +289,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ getFiltersFromFile(optarg, opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +470,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +502,109 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * getFiltersFromFile - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getFiltersFromFile(const char *filename, RestoreOptions *opts)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments or empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_FUNCTION)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_INDEX)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ simple_string_list_append(&opts->schemaNames, objname);
+ else
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TRIGGER)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, optarg);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_restore", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free_sources(&fstate);
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..246880f6ec
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,398 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 49;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;;
+my $inputfile;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres',
+ 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three_one(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres',
+ "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three_one VALUES('*** TABLE THREE_ONE ***')");
+
+#
+# Test interaction of correctly specified filter file
+#
+my ($cmd, $stdout, $stderr, $result);
+
+# Empty filterfile
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "\n # a comment and nothing more\n\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_(one|two|three|three_one)/m,
+ "tables dumped");
+
+# Test various combinations of whitespace, comments and correct filters
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "\t\n";
+print $inputfile " \t# another comment\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter patterns as well as comments and whitespace");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public\.table_three/m, "table three not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_three_one/m,
+ "table three_one not dumped");
+ok( $dump !~ qr/^COPY public\.table_one/m,
+ "content of table one is not included");
+ok($dump =~ qr/^COPY public\.table_two/m, "content of table two is included");
+
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with exclusion of a single table");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping tables with a wildcard pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_thre*\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with wildcard in pattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_two/m, "table two not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping table with multiline quoted tablename
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with multiline names requiring quoting");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding multiline quoted tablename from dump
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding an entire schema
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "exclude the public schema");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test including and excluding an entire schema by multiple filterfiles
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema public\n";
+close $inputfile;
+
+open my $alt_inputfile, '>', "$tempdir/inputfile2.txt"
+ or die "unable to open filterfile for writing";
+print $alt_inputfile "exclude schema public\n";
+close $alt_inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "--filter=$tempdir/inputfile2.txt", 'postgres'
+ ],
+ "exclude the public schema with multiple filters");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test dumping a table with a single leading newline on a row
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump nonexisting foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummyserver\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump foreign_data with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE SERVER dummyserver/m, "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude foreign_data dummy*\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/exclude filter is not allowed/,
+ "erroneously exclude foreign server");
+
+#########################################
+# Test broken input format
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: invalid object type specified, should be table, schema, foreign_data or data"
+);
+
+# Test missing object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/missing object name/,
+ "invalid syntax: missing object identifier pattern");
+
+# Test adding extra content after the object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table one";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unexpected extra data/,
+ "invalid syntax: extra content after object identifier pattern");
+
+#########################################
+# Combined with --strict-names
+
+# First ensure that a matching filter works
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ "strict names with matching mattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "no table dumped");
+
+# Now append a pattern to the filter file which doesn't resolve
+open $inputfile, '>>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_nonexisting_name";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "inclusion of non-existing objects with --strict names");
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index e4feda10fd..ccd36d5634 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -446,6 +446,7 @@ sub mkvcbuild
$pgdumpall->AddIncludeDir('src/backend');
$pgdumpall->AddFile('src/bin/pg_dump/pg_dumpall.c');
$pgdumpall->AddFile('src/bin/pg_dump/dumputils.c');
+ $pgdumpall->AddFile('src/bin/pg_dump/filter.c');
$pgdumpall->AddLibrary('ws2_32.lib');
my $pgrestore = AddSimpleFrontend('pg_dump', 1);
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-08-22 09:52 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2022-08-22 09:52 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
I am sending fresh rebase + enhancing tests for pg_dumpall and pg_restore
Regards
Pavel
Attachments:
[text/x-patch] pg_dump_filteropt.patch (48.3K, ../../CAFj8pRAMne5wd--K9aEBcvE8M_KZZaeUkbEXgz0h1hpHQXd-DA@mail.gmail.com/3-pg_dump_filteropt.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index c08276bc0a..b64bae6987 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 5d54074e01..7671795060 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpretted according to the same rules
+ as <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for excluding databases,
+ and can also be specified more than once for multiple filter files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 47bd7dbda0..ffeb564c52 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules as <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 2f524b09bf..d8c1babc26 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -25,6 +25,7 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filter.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -42,8 +43,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/filter.c b/src/bin/pg_dump/filter.c
new file mode 100644
index 0000000000..5a76fd57f1
--- /dev/null
+++ b/src/bin/pg_dump/filter.c
@@ -0,0 +1,431 @@
+/*-------------------------------------------------------------------------
+ *
+ * Implementation of simple filter file parser
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include "filter.h"
+
+#include "common/logging.h"
+#include "common/fe_memutils.h"
+#include "common/string.h"
+#include "lib/stringinfo.h"
+#include "pqexpbuffer.h"
+
+#define is_keyword_str(cstr, str, bytes) \
+ ((strlen(cstr) == bytes) && (pg_strncasecmp(cstr, str, bytes) == 0))
+
+/*
+ * Following routines are called from pg_dump, pg_dumpall and pg_restore.
+ * Unfortunatelly, implementation of exit_nicely in pg_dump and pg_restore
+ * is different from implementation of this rutine in pg_dumpall. So instead
+ * direct calling exit_nicely we have to return some error flag (in this
+ * case NULL), and exit_nicelly will be executed from caller's routine.
+ */
+
+/*
+ * Simple routines - just don't repeat same code
+ *
+ * Returns true, when filter's file is opened
+ */
+bool
+filter_init(FilterStateData *fstate, const char *filename)
+{
+ fstate->filename = filename;
+ fstate->lineno = 0;
+ initStringInfo(&fstate->linebuff);
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate->fp = fopen(filename, "r");
+ if (!fstate->fp)
+ {
+ pg_log_error("could not open filter file \"%s\": %m", filename);
+ return false;
+ }
+ }
+ else
+ fstate->fp = stdin;
+
+ fstate->is_error = false;
+
+ return true;
+}
+
+/*
+ * Release allocated sources for filter
+ */
+void
+filter_free_sources(FilterStateData *fstate)
+{
+ free(fstate->linebuff.data);
+ fstate->linebuff.data = NULL;
+
+ if (fstate->fp && fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ pg_log_error("could not close filter file \"%s\": %m", fstate->filename);
+
+ fstate->fp = NULL;
+ }
+}
+
+/*
+ * log_format_error - Emit error message
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ fstate->is_error = true;
+}
+
+static const char *
+filter_object_type_name(FilterObjectType fot)
+{
+ switch (fot)
+ {
+ case FILTER_OBJECT_TYPE_NONE:
+ return "comment or empty line";
+ case FILTER_OBJECT_TYPE_DATA:
+ return "data";
+ case FILTER_OBJECT_TYPE_DATABASE:
+ return "database";
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ return "foreign data";
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ return "function";
+ case FILTER_OBJECT_TYPE_INDEX:
+ return "index";
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ return "schema";
+ case FILTER_OBJECT_TYPE_TABLE:
+ return "table";
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ return "trigger";
+ }
+
+ return "unknown object type";
+}
+
+/*
+ * Helper routine to reduce duplicated code
+ */
+void
+log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname,
+ FilterObjectType fot)
+{
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str,
+ "The application \"%s\" doesn't support filter for object type \"%s\".",
+ appname,
+ filter_object_type_name(fot));
+
+ log_invalid_filter_format(fstate, str->data);
+}
+
+/*
+ * filter_get_keyword - read the next filter keyword from buffer
+ *
+ * Search for keywords (limited to ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL, when the buffer is empty or first
+ * char is not alpha. The length of the found keyword is returned in the size
+ * parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* Set returnlength preemptively in case no keyword is found */
+ *size = 0;
+
+ /* Skip initial whitespace */
+ while (isspace(*ptr))
+ ptr++;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr++;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * filter_get_pattern - Read an object identifier pattern from the buffer
+ *
+ * Parses an object identifier pattern from the passed in buffer and sets
+ * objname to a string with object identifier pattern. Returns pointer to the
+ * first character after the pattern.
+ */
+static char *
+filter_get_pattern(FilterStateData *fstate,
+ char *str,
+ char **objname)
+{
+ /* Skip whitespace */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ {
+ log_invalid_filter_format(fstate, "missing object name pattern");
+ return NULL;
+ }
+
+ /*
+ * If the object name pattern has been quoted, we must take care to parse
+ * out the entire quoted pattern, which may contain whitespace and can span
+ * many lines.
+ */
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ Assert(fstate->linebuff.data);
+
+ if (!pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m",
+ fstate->filename);
+ fstate->is_error = true;
+ }
+ else
+ log_invalid_filter_format(fstate, "unexpected end of file");
+
+ return NULL;
+ }
+
+ str = fstate->linebuff.data;
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ fstate->lineno++;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = pg_strdup(quoted_name->data);
+ destroyPQExpBuffer(quoted_name);
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* Simple variant, read to EOL or to first whitespace */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ return str;
+}
+
+/*
+ * read_filter_item - Read command/type/pattern triplet from filter file
+ *
+ * This will parse one filter item from the filter file, and while it is a
+ * row based format a pattern may span more than one line due to how object
+ * names can be constructed. The expected format of the filter file is:
+ *
+ * <command> <object_type> <pattern>
+ *
+ * Where command is "include" or "exclude", and object_type is one of:
+ * "table", "schema", "foreign_data", "data", "database", "function",
+ * "trigger" or "index". The pattern is either simple without any
+ * whitespace, or properly quoted in case there may be whitespace in the
+ * object name. The pattern handling follows the same rules as other object
+ * include and exclude functions; it can use wildcards. Returns true, when
+ * one filter item was successfully read and parsed. When object name
+ * contains \n chars, then more than one line from input file can be
+ * processed. Returns false when the filter file reaches EOF. In case of
+ * errors, the function wont return but will exit with an appropriate error
+ * message.
+ */
+bool
+filter_read_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ Assert(!fstate->is_error);
+
+ if (pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ char *str = fstate->linebuff.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno++;
+
+ (void) pg_strip_crlf(str);
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Skip empty lines or lines where the first non-whitespace character
+ * is a hash indicating a comment.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ /*
+ * First we expect sequence of two keywords, {include|exclude}
+ * followed by the object type to operate on.
+ */
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate,
+ "no filter command found (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ if (is_keyword_str("include", keyword, size))
+ *is_include = true;
+ else if (is_keyword_str("exclude", keyword, size))
+ *is_include = false;
+ else
+ {
+ log_invalid_filter_format(fstate,
+ "invalid filter command (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate, "missing filter object type");
+ return false;
+ }
+
+ if (is_keyword_str("data", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else if (is_keyword_str("database", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_DATABASE;
+ else if (is_keyword_str("foreign_data",keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (is_keyword_str("function", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FUNCTION;
+ else if (is_keyword_str("index", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_INDEX;
+ else if (is_keyword_str("schema", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (is_keyword_str("table", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (is_keyword_str("trigger", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TRIGGER;
+ else
+ {
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str, "unsupported filter object type: \"%.*s\"", size, keyword);
+ log_invalid_filter_format(fstate, str->data);
+ return false;
+ }
+
+ str = filter_get_pattern(fstate, str, objname);
+ if (!str)
+ return false;
+
+ /*
+ * Look for any content after the object identifier. Comments and
+ * whitespace are allowed, other content may indicate that the
+ * user needed to quote the object name so exit with an invalid
+ * format error.
+ */
+ while (isspace(*str))
+ str++;
+
+ if (*str != '\0' && *str != '#')
+ {
+ log_invalid_filter_format(fstate,
+ "unexpected extra data after pattern");
+ return false;
+ }
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m", fstate->filename);
+ fstate->is_error = true;
+ }
+
+ return false;
+}
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..f2503286aa
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,55 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+#include "c.h"
+
+#include "lib/stringinfo.h"
+
+/*
+ * State data for reading filter items from stream
+ */
+typedef struct
+{
+ FILE *fp;
+ const char *filename;
+ int lineno;
+ StringInfoData linebuff;
+ bool is_error;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern bool filter_init(FilterStateData *fstate, const char *filename);
+extern void filter_free_sources(FilterStateData *fstate);
+
+extern void log_invalid_filter_format(FilterStateData *fstate, char *message);
+extern void log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname, FilterObjectType fot);
+extern bool filter_read_item(FilterStateData *fstate, bool *is_include, char **objname, FilterObjectType *objtype);
+
+#endif
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 2c68915732..533e078889 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -58,6 +58,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -317,6 +318,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void getFiltersFromFile(const char *filename, DumpOptions *dopt);
int
@@ -389,6 +391,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -622,6 +625,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ getFiltersFromFile(optarg, &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1027,6 +1034,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18164,3 +18173,89 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+/*
+ * getFiltersFromFile - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getFiltersFromFile(const char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments and empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ {
+ log_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ break;
+ }
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dump", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free_sources(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 26d3d53809..d87d29093f 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -25,6 +25,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -56,6 +57,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -133,6 +135,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -335,6 +338,11 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ getDatabaseExcludeFiltersFromFile(optarg,
+ &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -628,6 +636,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1735,7 +1744,6 @@ executeCommand(PGconn *conn, const char *query)
PQclear(res);
}
-
/*
* dumpTimestamp
*/
@@ -1748,3 +1756,53 @@ dumpTimestamp(const char *msg)
if (strftime(buf, sizeof(buf), PGDUMP_STRFTIME_FMT, localtime(&now)) != 0)
fprintf(OPF, "-- %s %s\n\n", msg, buf);
}
+
+/*
+ * getDatabaseFiltersFromFile - retrieve database identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *pattern)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATABASE)
+ {
+ if (!is_include)
+ simple_string_list_append(pattern, objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "include database filter is not suppported");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dumpall", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free_sources(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..ab8525cafd 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,11 +47,13 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
static void usage(const char *progname);
+static void getFiltersFromFile(const char *filename, RestoreOptions *dopt);
int
main(int argc, char **argv)
@@ -123,6 +125,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +289,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ getFiltersFromFile(optarg, opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +470,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +502,109 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * getFiltersFromFile - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getFiltersFromFile(const char *filename, RestoreOptions *opts)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments or empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_FUNCTION)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_INDEX)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ simple_string_list_append(&opts->schemaNames, objname);
+ else
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TRIGGER)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, optarg);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_restore", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free_sources(&fstate);
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..dd533d79eb
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,491 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 62;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;;
+my $inputfile;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres',
+ 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three_one(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres',
+ "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three_one VALUES('*** TABLE THREE_ONE ***')");
+
+#
+# Test interaction of correctly specified filter file
+#
+my ($cmd, $stdout, $stderr, $result);
+
+# Empty filterfile
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "\n # a comment and nothing more\n\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_(one|two|three|three_one)/m,
+ "tables dumped");
+
+# Test various combinations of whitespace, comments and correct filters
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "\t\n";
+print $inputfile " \t# another comment\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter patterns as well as comments and whitespace");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public\.table_three/m, "table three not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_three_one/m,
+ "table three_one not dumped");
+ok( $dump !~ qr/^COPY public\.table_one/m,
+ "content of table one is not included");
+ok($dump =~ qr/^COPY public\.table_two/m, "content of table two is included");
+
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with exclusion of a single table");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping tables with a wildcard pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_thre*\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with wildcard in pattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_two/m, "table two not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping table with multiline quoted tablename
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with multiline names requiring quoting");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding multiline quoted tablename from dump
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding an entire schema
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "exclude the public schema");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test including and excluding an entire schema by multiple filterfiles
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema public\n";
+close $inputfile;
+
+open my $alt_inputfile, '>', "$tempdir/inputfile2.txt"
+ or die "unable to open filterfile for writing";
+print $alt_inputfile "exclude schema public\n";
+close $alt_inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "--filter=$tempdir/inputfile2.txt", 'postgres'
+ ],
+ "exclude the public schema with multiple filters");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test dumping a table with a single leading newline on a row
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump nonexisting foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummyserver\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump foreign_data with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE SERVER dummyserver/m, "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude foreign_data dummy*\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/exclude filter is not allowed/,
+ "erroneously exclude foreign server");
+
+#########################################
+# Test broken input format
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: invalid object type specified, should be table, schema, foreign_data or data"
+);
+
+# Test missing object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/missing object name/,
+ "invalid syntax: missing object identifier pattern");
+
+# Test adding extra content after the object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table one";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unexpected extra data/,
+ "invalid syntax: extra content after object identifier pattern");
+
+#########################################
+# Combined with --strict-names
+
+# First ensure that a matching filter works
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ "strict names with matching mattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "no table dumped");
+
+# Now append a pattern to the filter file which doesn't resolve
+open $inputfile, '>>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_nonexisting_name";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "inclusion of non-existing objects with --strict names");
+
+#########################################
+# pg_dumpall tests
+
+###########################
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude database postgres\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ "dump tables with exclusion of a database");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^\\connect postgres/m, "database postgres is not dumped");
+ok($dump =~ qr/^\\connect template1/m, "database template1 is dumped");
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: exclusion of non-existing object type"
+);
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table foo";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/The application "pg_dumpall" doesn't support filter for object type "table"/,
+ "invalid syntax: exclusion of unsupported object type"
+);
+
+#########################################
+# pg_restore tests
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', "$tempdir/filter_test.dump",
+ "-Fc", 'postgres'
+ ],
+ "dump all tables");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_two";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "wanted table restored");
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "unwanted tables are not restored");
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index ee963d85f3..a24de02e31 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -454,6 +454,7 @@ sub mkvcbuild
$pgdumpall->AddIncludeDir('src/backend');
$pgdumpall->AddFile('src/bin/pg_dump/pg_dumpall.c');
$pgdumpall->AddFile('src/bin/pg_dump/dumputils.c');
+ $pgdumpall->AddFile('src/bin/pg_dump/filter.c');
$pgdumpall->AddLibrary('ws2_32.lib');
my $pgrestore = AddSimpleFrontend('pg_dump', 1);
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-07 19:45 Daniel Gustafsson <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Daniel Gustafsson @ 2022-09-07 19:45 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
As noted upthread at some point, I'm not overly excited about the parser in
filter.c, for maintainability and readability reasons. So, I've reimplemented
the parser in Flex/Bison in the attached patch, which IMHO provides a clear(er)
picture of the grammar and is more per project standards. This version of the
patch is your latest version with just the parser replaced (at a reduction in
size as a side benefit).
All features supported in your latest patch version are present, and it passes
all the tests added by this patch. It's been an undisclosed amount of years
since I wrote a Bison parser (well, yacc really) from scratch so I don't rule
out having made silly mistakes. I would very much appreciate review from those
more well versed in this area.
One thing this patchversion currently lacks is refined error messaging, but if
we feel that this approach is a viable path then that can be tweaked. The
function which starts the parser can also be refactored to be shared across
pg_dump, pg_dumpall and pg_restore but I've kept it simple for now.
Thoughts? It would be nice to get this patch across the finishline during this
commitfest.
--
Daniel Gustafsson https://vmware.com/
Attachments:
[application/octet-stream] 0001-Add-include-exclude-filtering-via-file-in-pg_dump.patch (29.8K, ../../[email protected]/2-0001-Add-include-exclude-filtering-via-file-in-pg_dump.patch)
download | inline diff:
From b683523b1477936c1381994e3a69bb28c512a3c1 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Wed, 7 Sep 2022 15:22:05 +0200
Subject: [PATCH] Add include/exclude filtering via file in pg_dump
Author: Pavel Stehule <[email protected]>
Discussion: https://postgr.es/m/CAFj8pRB10wvW0CC9Xq=1XDs=zCQxer3cbLcNZa+qiX4cUH-G_A@mail.gmail.com
---
doc/src/sgml/ref/pg_dump.sgml | 88 ++++++++++++++++++++
doc/src/sgml/ref/pg_dumpall.sgml | 22 +++++
doc/src/sgml/ref/pg_restore.sgml | 25 ++++++
src/bin/pg_dump/.gitignore | 4 +
src/bin/pg_dump/Makefile | 17 +++-
src/bin/pg_dump/filter.h | 44 ++++++++++
src/bin/pg_dump/filterparse.y | 65 +++++++++++++++
src/bin/pg_dump/filterscan.l | 133 +++++++++++++++++++++++++++++++
src/bin/pg_dump/pg_dump.c | 89 +++++++++++++++++++++
src/bin/pg_dump/pg_dumpall.c | 65 ++++++++++++++-
src/bin/pg_dump/pg_restore.c | 91 +++++++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 4 +-
12 files changed, 643 insertions(+), 4 deletions(-)
create mode 100644 src/bin/pg_dump/filter.h
create mode 100644 src/bin/pg_dump/filterparse.y
create mode 100644 src/bin/pg_dump/filterscan.l
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index c08276bc0a..b64bae6987 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 5d54074e01..46a667c17a 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpreted according to the same rules
+ as <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for excluding databases,
+ and can also be specified more than once for multiple filter files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 47bd7dbda0..ffeb564c52 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules as <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/.gitignore b/src/bin/pg_dump/.gitignore
index e6d7812779..11f2d68bea 100644
--- a/src/bin/pg_dump/.gitignore
+++ b/src/bin/pg_dump/.gitignore
@@ -2,4 +2,8 @@
/pg_dumpall
/pg_restore
+# Local generated source files
+/filterparse.c
+/filterscan.c
+
/tmp_check/
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 9dc5a784dd..e3befdc9b1 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -26,6 +26,8 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filterparse.o \
+ filterscan.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -37,14 +39,23 @@ OBJS = \
all: pg_dump pg_restore pg_dumpall
+# See notes in src/backend/parser/Makefile about the following two rules
+filterparse.h: filterparse.c
+ touch $@
+
+filterparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+filterparse.o filterscan.o: filterparse.h
+
pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dump.o common.o pg_dump_sort.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filterparse.o filterscan.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filterparse.o filterscan.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
@@ -63,6 +74,8 @@ installcheck:
uninstall:
rm -f $(addprefix '$(DESTDIR)$(bindir)'/, pg_dump$(X) pg_restore$(X) pg_dumpall$(X))
+distprep: filterparse.c filterscan.c
+
clean distclean maintainer-clean:
rm -f pg_dump$(X) pg_restore$(X) pg_dumpall$(X) $(OBJS) pg_dump.o common.o pg_dump_sort.o pg_restore.o pg_dumpall.o
rm -rf tmp_check
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..5dff4161f0
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+#include "c.h"
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern FILE *filter_yyin;
+
+extern int filter_yylex(void);
+extern void filter_yyerror(void *priv, const char *msg);
+extern void filter_scanner_init(void);
+extern void filter_scanner_finish(void);
+extern int filter_yyparse(void *priv);
+
+extern void include_item(void *priv, FilterObjectType objtype, const char *objname);
+extern void exclude_item(void *priv, FilterObjectType objtype, const char *objname);
+
+#endif
diff --git a/src/bin/pg_dump/filterparse.y b/src/bin/pg_dump/filterparse.y
new file mode 100644
index 0000000000..16623ff824
--- /dev/null
+++ b/src/bin/pg_dump/filterparse.y
@@ -0,0 +1,65 @@
+%{
+/*-------------------------------------------------------------------------
+ *
+ * filterparse.y
+ * bison grammar for the pg_dump object filter files
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filterparse.y
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "filter.h"
+
+%}
+
+%name-prefix="filter_yy"
+%parse-param {void *priv}
+
+%union
+{
+ char *str;
+ int integer;
+}
+
+%type <integer> include_object exclude_object
+%token <str> pattern
+%token C_INCLUDE C_EXCLUDE
+%token OBJ_DATA OBJ_DATABASE OBJ_FOREIGN_DATA OBJ_FUNCTION OBJ_INDEX
+%token OBJ_SCHEMA OBJ_TABLE OBJ_TRIGGER
+%start Patterns
+
+%%
+
+Patterns:
+ /* EMPTY */
+ | Patterns Pattern
+ | Pattern
+ ;
+
+Pattern:
+ C_INCLUDE include_object pattern { include_item(priv, $2, $3); }
+ | C_EXCLUDE exclude_object pattern { exclude_item(priv, $2, $3); }
+ ;
+
+include_object:
+ OBJ_FOREIGN_DATA { $$ = FILTER_OBJECT_TYPE_FOREIGN_DATA; }
+ | OBJ_FUNCTION { $$ = FILTER_OBJECT_TYPE_FUNCTION; }
+ | OBJ_INDEX { $$ = FILTER_OBJECT_TYPE_INDEX; }
+ | OBJ_SCHEMA { $$ = FILTER_OBJECT_TYPE_SCHEMA; }
+ | OBJ_TABLE { $$ = FILTER_OBJECT_TYPE_TABLE; }
+ | OBJ_TRIGGER { $$ = FILTER_OBJECT_TYPE_TRIGGER; }
+ ;
+
+exclude_object:
+ OBJ_DATA { $$ = FILTER_OBJECT_TYPE_DATA; }
+ | OBJ_DATABASE { $$ = FILTER_OBJECT_TYPE_DATABASE; }
+ | OBJ_SCHEMA { $$ = FILTER_OBJECT_TYPE_SCHEMA; }
+ | OBJ_TABLE { $$ = FILTER_OBJECT_TYPE_TABLE; }
+ ;
+
+%%
diff --git a/src/bin/pg_dump/filterscan.l b/src/bin/pg_dump/filterscan.l
new file mode 100644
index 0000000000..472645ac35
--- /dev/null
+++ b/src/bin/pg_dump/filterscan.l
@@ -0,0 +1,133 @@
+%top{
+/*-------------------------------------------------------------------------
+ *
+ * filterscan.l
+ * a lexical scanner for pg_dump object filter files
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filterscan.l
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+#include "filterparse.h"
+#include "filter.h"
+}
+
+%{
+
+static int yyline = 1;
+
+static void add_pattern_char(char c);
+
+#define PATTERNBUF_INIT 128
+
+static char *patternbuf = NULL;
+static size_t patternbufsize = PATTERNBUF_INIT;
+static size_t patternbufpos = 0;
+
+%}
+
+%option never-interactive
+%option noinput
+%option nounput
+%option noyywrap
+%option warn
+%option prefix="filter_yy"
+
+%x command
+%x type
+%x qident
+
+space [ \t\r\f\v]
+newline [\n]
+non_newline [^\n\r]
+
+comment ("#"{non_newline}*)
+pattern ([^"\n# \t\r\f\v]+)
+
+%%
+
+ /* Commands */
+include { BEGIN(command); return C_INCLUDE; }
+exclude { BEGIN(command); return C_EXCLUDE; }
+
+[\n] yyline++;
+{comment} { /* ignore */ }
+{space} { /* ignore */ }
+
+ /* Object types */
+<command>data { BEGIN(type); return OBJ_DATA; }
+<command>database { BEGIN(type); return OBJ_DATABASE; }
+<command>foreign_data { BEGIN(type); return OBJ_FOREIGN_DATA; }
+<command>function { BEGIN(type); return OBJ_FUNCTION; }
+<command>index { BEGIN(type); return OBJ_INDEX; }
+<command>schema { BEGIN(type); return OBJ_SCHEMA; }
+<command>table { BEGIN(type); return OBJ_TABLE; }
+<command>trigger { BEGIN(type); return OBJ_TRIGGER; }
+<command>{pattern} { filter_yyerror(NULL, "unsupported filter object type"); }
+<command><<EOF>> { filter_yyerror(NULL, "missing type name"); }
+
+ /* Patterns */
+<type><<EOF>> { filter_yyerror(NULL, "missing object name"); }
+<type>{pattern} {
+ filter_yylval.str = pg_strdup(yytext);
+ BEGIN(INITIAL);
+ return(pattern);
+ }
+<type>\" {
+ patternbufpos = 0;
+ memset(patternbuf, 0, patternbufsize);
+ BEGIN(qident);
+ }
+<qident>\\\" { add_pattern_char(yytext[1]); }
+<qident>\" {
+ filter_yylval.str = pg_strdup(patternbuf);
+ BEGIN(INITIAL);
+ return(pattern);
+ }
+<qident>{newline} { add_pattern_char('\n'); }
+<qident>. { add_pattern_char(yytext[0]); }
+<qident><<EOF>> {
+ fprintf(stderr, "syntax error at line %i; unterminated quoted pattern: %s\n",
+ yyline, patternbuf);
+ exit(1);
+ }
+ /* Anything else is a syntax error */
+. { filter_yyerror(NULL, "unexpected character"); }
+%%
+
+static void
+add_pattern_char(char c)
+{
+ /* Leave room for trailing zero */
+ if (patternbufpos >= patternbufsize - 1)
+ {
+ /* XXX: check for overflow */
+
+ /* Double the size of litbuf if it gets full */
+ patternbufsize += patternbufsize;
+ patternbuf = pg_realloc(patternbuf, patternbufsize);
+ }
+ patternbuf[patternbufpos++] = c;
+}
+
+void
+filter_scanner_init(void)
+{
+ patternbuf = pg_malloc(patternbufsize);
+}
+
+void
+filter_scanner_finish(void)
+{
+ pfree(patternbuf);
+}
+
+void
+filter_yyerror(void *priv, const char *msg)
+{
+ fprintf(stderr, "syntax error at line %i: %s: \"%s\"\n", yyline, msg, yytext);
+ exit(1);
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 67b6d9079e..a20f4c0b94 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -58,6 +58,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -317,6 +318,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void getFiltersFromFile(const char *filename, DumpOptions *dopt);
int
@@ -389,6 +391,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -622,6 +625,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ getFiltersFromFile(optarg, &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1027,6 +1034,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18197,3 +18206,83 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+/*
+ * getFiltersFromFile - retrieve object identifier patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getFiltersFromFile(const char *filename, DumpOptions *dopt)
+{
+ bool need_close = true;
+
+ if (strlen(filename) == 1 && filename[0] == '-')
+ {
+ filter_yyin = stdin;
+ need_close = false;
+ }
+ else
+ {
+ filter_yyin = fopen(filename, "r");
+ if (!filter_yyin)
+ pg_fatal("unable to open filterfile \"%s\"", filename);
+ }
+
+ filter_scanner_init();
+ filter_yyparse((void *) dopt);
+ filter_scanner_finish();
+
+ if (need_close)
+ fclose(filter_yyin);
+}
+
+void
+exclude_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ (void) priv;
+
+ switch (objtype)
+ {
+ case FILTER_OBJECT_TYPE_DATA:
+ simple_string_list_append(&tabledata_exclude_patterns, objname);
+ break;
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ simple_string_list_append(&schema_exclude_patterns, objname);
+ break;
+ case FILTER_OBJECT_TYPE_TABLE:
+ simple_string_list_append(&table_exclude_patterns, objname);
+ break;
+ default:
+ pg_log_error("Unsupported exclude object");
+ exit_nicely(1);
+ break;
+ }
+}
+
+void
+include_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ DumpOptions *dopt = (DumpOptions *) priv;
+
+ switch (objtype)
+ {
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ simple_string_list_append(&foreign_servers_include_patterns, objname);
+ break;
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ simple_string_list_append(&schema_include_patterns, objname);
+ dopt->include_everything = false;
+ break;
+ case FILTER_OBJECT_TYPE_TABLE:
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ break;
+ default:
+ pg_log_error("Unsupported include object");
+ exit_nicely(1);
+ break;
+ }
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 69ae027bd3..d3763603a4 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -26,6 +26,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -79,6 +80,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -156,6 +158,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -358,6 +361,11 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ getDatabaseExcludeFiltersFromFile(optarg,
+ &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -651,6 +659,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1888,7 +1897,6 @@ executeCommand(PGconn *conn, const char *query)
PQclear(res);
}
-
/*
* dumpTimestamp
*/
@@ -1912,3 +1920,58 @@ hash_string_pointer(char *s)
return hash_bytes(ss, strlen(s));
}
+
+/*
+ * getDatabaseFiltersFromFile - retrieve database identifier patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *pattern)
+{
+ bool need_close = true;
+
+ if (strlen(filename) == 1 && filename[0] == '-')
+ {
+ filter_yyin = stdin;
+ need_close = false;
+ }
+ else
+ {
+ filter_yyin = fopen(filename, "r");
+ if (!filter_yyin)
+ pg_fatal("unable to open filterfile \"%s\"", filename);
+ }
+
+ filter_scanner_init();
+ filter_yyparse((void *) pattern);
+ filter_scanner_finish();
+
+ if (need_close)
+ fclose(filter_yyin);
+}
+
+void
+exclude_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ SimpleStringList *pattern = (SimpleStringList *) priv;
+
+ if (objtype != FILTER_OBJECT_TYPE_DATABASE)
+ {
+ pg_log_error("Unsupported exclude object");
+ exit_nicely(1);
+ }
+
+ simple_string_list_append(pattern, objname);
+}
+
+void
+include_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ (void) priv;
+
+ pg_log_error("Unsupported include object");
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..d4e6b753e0 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,11 +47,13 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
static void usage(const char *progname);
+static void getFiltersFromFile(const char *filename, RestoreOptions *dopt);
int
main(int argc, char **argv)
@@ -123,6 +125,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +289,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ getFiltersFromFile(optarg, opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +470,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +502,86 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * getFiltersFromFile - retrieve object identifier patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getFiltersFromFile(const char *filename, RestoreOptions *opts)
+{
+ bool need_close = true;
+
+ if (strlen(filename) == 1 && filename[0] == '-')
+ {
+ filter_yyin = stdin;
+ need_close = false;
+ }
+ else
+ {
+ filter_yyin = fopen(filename, "r");
+ if (!filter_yyin)
+ pg_fatal("unable to open filterfile \"%s\"", filename);
+ }
+
+ filter_scanner_init();
+ filter_yyparse((void *) opts);
+ filter_scanner_finish();
+
+ if (need_close)
+ fclose(filter_yyin);
+}
+
+void
+exclude_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ RestoreOptions *opts = (RestoreOptions *) priv;
+
+ if (objtype != FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ pg_log_error("Unsupported exclude object");
+ exit_nicely(1);
+ }
+
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+}
+
+void
+include_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ RestoreOptions *opts = (RestoreOptions *) priv;
+
+ switch (objtype)
+ {
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_INDEX:
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ simple_string_list_append(&opts->schemaNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_TABLE:
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, optarg);
+ break;
+ default:
+ pg_log_error("Unsupported include object");
+ exit_nicely(1);
+ break;
+ }
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 098bc3f1b0..6418db9261 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -78,7 +78,9 @@ my $frontend_extraincludes = {
my $frontend_extrasource = {
'psql' => ['src/bin/psql/psqlscanslash.l'],
'pgbench' =>
- [ 'src/bin/pgbench/exprscan.l', 'src/bin/pgbench/exprparse.y' ]
+ [ 'src/bin/pgbench/exprscan.l', 'src/bin/pgbench/exprparse.y' ],
+ 'pg_dump' =>
+ ['src/bin/pg_dump/filterscan.l', 'src/bin/pg_dump/filterparse.y']
};
my @frontend_excludes = (
'pgevent', 'pg_basebackup', 'pg_rewind', 'pg_dump',
--
2.32.1 (Apple Git-133)
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-08 10:00 Erik Rijkers <[email protected]>
parent: Daniel Gustafsson <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Erik Rijkers @ 2022-09-08 10:00 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; Pavel Stehule <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Op 07-09-2022 om 21:45 schreef Daniel Gustafsson:
>
> One thing this patchversion currently lacks is refined error messaging, but if
> we feel that this approach is a viable path then that can be tweaked. The
> function which starts the parser can also be refactored to be shared across
> pg_dump, pg_dumpall and pg_restore but I've kept it simple for now.
>
> Thoughts? It would be nice to get this patch across the finishline during this
> commitfest.
> [0001-Add-include-exclude-filtering-via-file-in-pg_dump.patch]
This seems to dump & restore well (as Pavels patch does).
I did notice one peculiarity (in your patch) where for each table a few
spaces are omitted by pg_dump.
-------------
#! /bin/bash
psql -qXc "drop database if exists testdb2"
psql -qXc "create database testdb2"
echo "
create schema if not exists test;
create table table0 (id integer);
create table table1 (id integer);
insert into table0 select n from generate_series(1,2) as f(n);
insert into table1 select n from generate_series(1,2) as f(n);
" | psql -qXad testdb2
echo "include table table0" > inputfile1.txt
echo "include table table0
include table table1" > inputfile2.txt
# 1 table, emits 2 spaces
echo -ne ">"
pg_dump -F p -f plainfile1 --filter=inputfile1.txt -d testdb2
echo "<"
# 2 tables, emits 4 space
echo -ne ">"
pg_dump -F p -f plainfile2 --filter=inputfile2.txt -d testdb2
echo "<"
# dump without filter emits no spaces
echo -ne ">"
pg_dump -F c -f plainfile3 -t table0 -table1 -d testdb2
echo "<"
-------------
It's probably a small thing -- but I didn't find it.
thanks,
Erik Rijkers
>
> --
> Daniel Gustafsson https://vmware.com/
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-08 11:38 Daniel Gustafsson <[email protected]>
parent: Erik Rijkers <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Daniel Gustafsson @ 2022-09-08 11:38 UTC (permalink / raw)
To: Erik Rijkers <[email protected]>; +Cc: Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 8 Sep 2022, at 12:00, Erik Rijkers <[email protected]> wrote:
>
> Op 07-09-2022 om 21:45 schreef Daniel Gustafsson:
>> One thing this patchversion currently lacks is refined error messaging, but if
>> we feel that this approach is a viable path then that can be tweaked. The
>> function which starts the parser can also be refactored to be shared across
>> pg_dump, pg_dumpall and pg_restore but I've kept it simple for now.
>> Thoughts? It would be nice to get this patch across the finishline during this
>> commitfest.
>
> > [0001-Add-include-exclude-filtering-via-file-in-pg_dump.patch]
>
> This seems to dump & restore well (as Pavels patch does).
Thanks for looking!
> I did notice one peculiarity (in your patch) where for each table a few spaces are omitted by pg_dump.
Right, I had that on my TODO to fix before submitting but clearly forgot. It
boils down to consuming the space between commands and object types and object
patterns. The attached v2 fixes that.
--
Daniel Gustafsson https://vmware.com/
Attachments:
[application/octet-stream] v2-0001-Add-include-exclude-filtering-via-file-in-pg_dump.patch (29.9K, ../../[email protected]/2-v2-0001-Add-include-exclude-filtering-via-file-in-pg_dump.patch)
download | inline diff:
From 90e0ea85d2dd66969962be92c1c78cc587398896 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Wed, 7 Sep 2022 15:22:05 +0200
Subject: [PATCH v2] Add include/exclude filtering via file in pg_dump
Author: Pavel Stehule <[email protected]>
Discussion: https://postgr.es/m/CAFj8pRB10wvW0CC9Xq=1XDs=zCQxer3cbLcNZa+qiX4cUH-G_A@mail.gmail.com
---
doc/src/sgml/ref/pg_dump.sgml | 88 ++++++++++++++++++++
doc/src/sgml/ref/pg_dumpall.sgml | 22 +++++
doc/src/sgml/ref/pg_restore.sgml | 25 ++++++
src/bin/pg_dump/.gitignore | 4 +
src/bin/pg_dump/Makefile | 17 +++-
src/bin/pg_dump/filter.h | 44 ++++++++++
src/bin/pg_dump/filterparse.y | 65 +++++++++++++++
src/bin/pg_dump/filterscan.l | 134 +++++++++++++++++++++++++++++++
src/bin/pg_dump/pg_dump.c | 89 ++++++++++++++++++++
src/bin/pg_dump/pg_dumpall.c | 65 ++++++++++++++-
src/bin/pg_dump/pg_restore.c | 91 +++++++++++++++++++++
src/tools/msvc/Mkvcbuild.pm | 4 +-
12 files changed, 644 insertions(+), 4 deletions(-)
create mode 100644 src/bin/pg_dump/filter.h
create mode 100644 src/bin/pg_dump/filterparse.y
create mode 100644 src/bin/pg_dump/filterscan.l
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index c08276bc0a..b64bae6987 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 5d54074e01..46a667c17a 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpreted according to the same rules
+ as <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for excluding databases,
+ and can also be specified more than once for multiple filter files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 47bd7dbda0..ffeb564c52 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules as <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/.gitignore b/src/bin/pg_dump/.gitignore
index e6d7812779..11f2d68bea 100644
--- a/src/bin/pg_dump/.gitignore
+++ b/src/bin/pg_dump/.gitignore
@@ -2,4 +2,8 @@
/pg_dumpall
/pg_restore
+# Local generated source files
+/filterparse.c
+/filterscan.c
+
/tmp_check/
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 9dc5a784dd..e3befdc9b1 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -26,6 +26,8 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filterparse.o \
+ filterscan.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -37,14 +39,23 @@ OBJS = \
all: pg_dump pg_restore pg_dumpall
+# See notes in src/backend/parser/Makefile about the following two rules
+filterparse.h: filterparse.c
+ touch $@
+
+filterparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+filterparse.o filterscan.o: filterparse.h
+
pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dump.o common.o pg_dump_sort.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filterparse.o filterscan.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filterparse.o filterscan.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
@@ -63,6 +74,8 @@ installcheck:
uninstall:
rm -f $(addprefix '$(DESTDIR)$(bindir)'/, pg_dump$(X) pg_restore$(X) pg_dumpall$(X))
+distprep: filterparse.c filterscan.c
+
clean distclean maintainer-clean:
rm -f pg_dump$(X) pg_restore$(X) pg_dumpall$(X) $(OBJS) pg_dump.o common.o pg_dump_sort.o pg_restore.o pg_dumpall.o
rm -rf tmp_check
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..5dff4161f0
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+#include "c.h"
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern FILE *filter_yyin;
+
+extern int filter_yylex(void);
+extern void filter_yyerror(void *priv, const char *msg);
+extern void filter_scanner_init(void);
+extern void filter_scanner_finish(void);
+extern int filter_yyparse(void *priv);
+
+extern void include_item(void *priv, FilterObjectType objtype, const char *objname);
+extern void exclude_item(void *priv, FilterObjectType objtype, const char *objname);
+
+#endif
diff --git a/src/bin/pg_dump/filterparse.y b/src/bin/pg_dump/filterparse.y
new file mode 100644
index 0000000000..16623ff824
--- /dev/null
+++ b/src/bin/pg_dump/filterparse.y
@@ -0,0 +1,65 @@
+%{
+/*-------------------------------------------------------------------------
+ *
+ * filterparse.y
+ * bison grammar for the pg_dump object filter files
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filterparse.y
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "filter.h"
+
+%}
+
+%name-prefix="filter_yy"
+%parse-param {void *priv}
+
+%union
+{
+ char *str;
+ int integer;
+}
+
+%type <integer> include_object exclude_object
+%token <str> pattern
+%token C_INCLUDE C_EXCLUDE
+%token OBJ_DATA OBJ_DATABASE OBJ_FOREIGN_DATA OBJ_FUNCTION OBJ_INDEX
+%token OBJ_SCHEMA OBJ_TABLE OBJ_TRIGGER
+%start Patterns
+
+%%
+
+Patterns:
+ /* EMPTY */
+ | Patterns Pattern
+ | Pattern
+ ;
+
+Pattern:
+ C_INCLUDE include_object pattern { include_item(priv, $2, $3); }
+ | C_EXCLUDE exclude_object pattern { exclude_item(priv, $2, $3); }
+ ;
+
+include_object:
+ OBJ_FOREIGN_DATA { $$ = FILTER_OBJECT_TYPE_FOREIGN_DATA; }
+ | OBJ_FUNCTION { $$ = FILTER_OBJECT_TYPE_FUNCTION; }
+ | OBJ_INDEX { $$ = FILTER_OBJECT_TYPE_INDEX; }
+ | OBJ_SCHEMA { $$ = FILTER_OBJECT_TYPE_SCHEMA; }
+ | OBJ_TABLE { $$ = FILTER_OBJECT_TYPE_TABLE; }
+ | OBJ_TRIGGER { $$ = FILTER_OBJECT_TYPE_TRIGGER; }
+ ;
+
+exclude_object:
+ OBJ_DATA { $$ = FILTER_OBJECT_TYPE_DATA; }
+ | OBJ_DATABASE { $$ = FILTER_OBJECT_TYPE_DATABASE; }
+ | OBJ_SCHEMA { $$ = FILTER_OBJECT_TYPE_SCHEMA; }
+ | OBJ_TABLE { $$ = FILTER_OBJECT_TYPE_TABLE; }
+ ;
+
+%%
diff --git a/src/bin/pg_dump/filterscan.l b/src/bin/pg_dump/filterscan.l
new file mode 100644
index 0000000000..d9ef8b21fc
--- /dev/null
+++ b/src/bin/pg_dump/filterscan.l
@@ -0,0 +1,134 @@
+%top{
+/*-------------------------------------------------------------------------
+ *
+ * filterscan.l
+ * a lexical scanner for pg_dump object filter files
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filterscan.l
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+#include "filterparse.h"
+#include "filter.h"
+}
+
+%{
+
+static int yyline = 1;
+
+static void add_pattern_char(char c);
+
+#define PATTERNBUF_INIT 128
+
+static char *patternbuf = NULL;
+static size_t patternbufsize = PATTERNBUF_INIT;
+static size_t patternbufpos = 0;
+
+%}
+
+%option never-interactive
+%option noinput
+%option nounput
+%option noyywrap
+%option warn
+%option prefix="filter_yy"
+
+%x command
+%x type
+%x qident
+
+space [ \t\r\f\v]
+newline [\n]
+non_newline [^\n\r]
+
+comment ("#"{non_newline}*)
+pattern ([^"\n# \t\r\f\v]+)
+
+%%
+
+ /* Commands */
+include { BEGIN(command); return C_INCLUDE; }
+exclude { BEGIN(command); return C_EXCLUDE; }
+
+[\n] yyline++;
+{comment} { /* ignore */ }
+{space} { /* ignore */ }
+
+ /* Object types */
+<command>({space}*)data { BEGIN(type); return OBJ_DATA; }
+<command>({space}*)database { BEGIN(type); return OBJ_DATABASE; }
+<command>({space}*)foreign_data { BEGIN(type); return OBJ_FOREIGN_DATA; }
+<command>({space}*)function { BEGIN(type); return OBJ_FUNCTION; }
+<command>({space}*)index { BEGIN(type); return OBJ_INDEX; }
+<command>({space}*)schema { BEGIN(type); return OBJ_SCHEMA; }
+<command>({space}*)table { BEGIN(type); return OBJ_TABLE; }
+<command>({space}*)trigger { BEGIN(type); return OBJ_TRIGGER; }
+<command>{pattern} { filter_yyerror(NULL, "unsupported filter object type"); }
+<command><<EOF>> { filter_yyerror(NULL, "missing type name"); }
+
+ /* Patterns */
+<type><<EOF>> { filter_yyerror(NULL, "missing object name"); }
+<type>{space} { /* ignore */ }
+<type>{pattern} {
+ filter_yylval.str = pg_strdup(yytext);
+ BEGIN(INITIAL);
+ return(pattern);
+ }
+<type>\" {
+ patternbufpos = 0;
+ memset(patternbuf, 0, patternbufsize);
+ BEGIN(qident);
+ }
+<qident>\\\" { add_pattern_char(yytext[1]); }
+<qident>\" {
+ filter_yylval.str = pg_strdup(patternbuf);
+ BEGIN(INITIAL);
+ return(pattern);
+ }
+<qident>{newline} { add_pattern_char('\n'); }
+<qident>. { add_pattern_char(yytext[0]); }
+<qident><<EOF>> {
+ fprintf(stderr, "syntax error at line %i; unterminated quoted pattern: %s\n",
+ yyline, patternbuf);
+ exit(1);
+ }
+ /* Anything else is a syntax error */
+. { filter_yyerror(NULL, "unexpected character"); }
+%%
+
+static void
+add_pattern_char(char c)
+{
+ /* Leave room for trailing zero */
+ if (patternbufpos >= patternbufsize - 1)
+ {
+ /* XXX: check for overflow */
+
+ /* Double the size of litbuf if it gets full */
+ patternbufsize += patternbufsize;
+ patternbuf = pg_realloc(patternbuf, patternbufsize);
+ }
+ patternbuf[patternbufpos++] = c;
+}
+
+void
+filter_scanner_init(void)
+{
+ patternbuf = pg_malloc(patternbufsize);
+}
+
+void
+filter_scanner_finish(void)
+{
+ pfree(patternbuf);
+}
+
+void
+filter_yyerror(void *priv, const char *msg)
+{
+ fprintf(stderr, "syntax error at line %i: %s: \"%s\"\n", yyline, msg, yytext);
+ exit(1);
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 67b6d9079e..a20f4c0b94 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -58,6 +58,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -317,6 +318,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AHX);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void getFiltersFromFile(const char *filename, DumpOptions *dopt);
int
@@ -389,6 +391,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -622,6 +625,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ getFiltersFromFile(optarg, &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1027,6 +1034,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18197,3 +18206,83 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+/*
+ * getFiltersFromFile - retrieve object identifier patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getFiltersFromFile(const char *filename, DumpOptions *dopt)
+{
+ bool need_close = true;
+
+ if (strlen(filename) == 1 && filename[0] == '-')
+ {
+ filter_yyin = stdin;
+ need_close = false;
+ }
+ else
+ {
+ filter_yyin = fopen(filename, "r");
+ if (!filter_yyin)
+ pg_fatal("unable to open filterfile \"%s\"", filename);
+ }
+
+ filter_scanner_init();
+ filter_yyparse((void *) dopt);
+ filter_scanner_finish();
+
+ if (need_close)
+ fclose(filter_yyin);
+}
+
+void
+exclude_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ (void) priv;
+
+ switch (objtype)
+ {
+ case FILTER_OBJECT_TYPE_DATA:
+ simple_string_list_append(&tabledata_exclude_patterns, objname);
+ break;
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ simple_string_list_append(&schema_exclude_patterns, objname);
+ break;
+ case FILTER_OBJECT_TYPE_TABLE:
+ simple_string_list_append(&table_exclude_patterns, objname);
+ break;
+ default:
+ pg_log_error("Unsupported exclude object");
+ exit_nicely(1);
+ break;
+ }
+}
+
+void
+include_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ DumpOptions *dopt = (DumpOptions *) priv;
+
+ switch (objtype)
+ {
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ simple_string_list_append(&foreign_servers_include_patterns, objname);
+ break;
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ simple_string_list_append(&schema_include_patterns, objname);
+ dopt->include_everything = false;
+ break;
+ case FILTER_OBJECT_TYPE_TABLE:
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ break;
+ default:
+ pg_log_error("Unsupported include object");
+ exit_nicely(1);
+ break;
+ }
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 69ae027bd3..d3763603a4 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -26,6 +26,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -79,6 +80,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -156,6 +158,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -358,6 +361,11 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ getDatabaseExcludeFiltersFromFile(optarg,
+ &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -651,6 +659,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1888,7 +1897,6 @@ executeCommand(PGconn *conn, const char *query)
PQclear(res);
}
-
/*
* dumpTimestamp
*/
@@ -1912,3 +1920,58 @@ hash_string_pointer(char *s)
return hash_bytes(ss, strlen(s));
}
+
+/*
+ * getDatabaseFiltersFromFile - retrieve database identifier patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *pattern)
+{
+ bool need_close = true;
+
+ if (strlen(filename) == 1 && filename[0] == '-')
+ {
+ filter_yyin = stdin;
+ need_close = false;
+ }
+ else
+ {
+ filter_yyin = fopen(filename, "r");
+ if (!filter_yyin)
+ pg_fatal("unable to open filterfile \"%s\"", filename);
+ }
+
+ filter_scanner_init();
+ filter_yyparse((void *) pattern);
+ filter_scanner_finish();
+
+ if (need_close)
+ fclose(filter_yyin);
+}
+
+void
+exclude_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ SimpleStringList *pattern = (SimpleStringList *) priv;
+
+ if (objtype != FILTER_OBJECT_TYPE_DATABASE)
+ {
+ pg_log_error("Unsupported exclude object");
+ exit_nicely(1);
+ }
+
+ simple_string_list_append(pattern, objname);
+}
+
+void
+include_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ (void) priv;
+
+ pg_log_error("Unsupported include object");
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..d4e6b753e0 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,11 +47,13 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
static void usage(const char *progname);
+static void getFiltersFromFile(const char *filename, RestoreOptions *dopt);
int
main(int argc, char **argv)
@@ -123,6 +125,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +289,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ getFiltersFromFile(optarg, opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +470,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +502,86 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * getFiltersFromFile - retrieve object identifier patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getFiltersFromFile(const char *filename, RestoreOptions *opts)
+{
+ bool need_close = true;
+
+ if (strlen(filename) == 1 && filename[0] == '-')
+ {
+ filter_yyin = stdin;
+ need_close = false;
+ }
+ else
+ {
+ filter_yyin = fopen(filename, "r");
+ if (!filter_yyin)
+ pg_fatal("unable to open filterfile \"%s\"", filename);
+ }
+
+ filter_scanner_init();
+ filter_yyparse((void *) opts);
+ filter_scanner_finish();
+
+ if (need_close)
+ fclose(filter_yyin);
+}
+
+void
+exclude_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ RestoreOptions *opts = (RestoreOptions *) priv;
+
+ if (objtype != FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ pg_log_error("Unsupported exclude object");
+ exit_nicely(1);
+ }
+
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+}
+
+void
+include_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ RestoreOptions *opts = (RestoreOptions *) priv;
+
+ switch (objtype)
+ {
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_INDEX:
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ simple_string_list_append(&opts->schemaNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_TABLE:
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, optarg);
+ break;
+ default:
+ pg_log_error("Unsupported include object");
+ exit_nicely(1);
+ break;
+ }
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 098bc3f1b0..6418db9261 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -78,7 +78,9 @@ my $frontend_extraincludes = {
my $frontend_extrasource = {
'psql' => ['src/bin/psql/psqlscanslash.l'],
'pgbench' =>
- [ 'src/bin/pgbench/exprscan.l', 'src/bin/pgbench/exprparse.y' ]
+ [ 'src/bin/pgbench/exprscan.l', 'src/bin/pgbench/exprparse.y' ],
+ 'pg_dump' =>
+ ['src/bin/pg_dump/filterscan.l', 'src/bin/pg_dump/filterparse.y']
};
my @frontend_excludes = (
'pgevent', 'pg_basebackup', 'pg_rewind', 'pg_dump',
--
2.32.1 (Apple Git-133)
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-08 11:44 Julien Rouhaud <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Julien Rouhaud @ 2022-09-08 11:44 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Erik Rijkers <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi,
On Thu, Sep 08, 2022 at 01:38:42PM +0200, Daniel Gustafsson wrote:
> > On 8 Sep 2022, at 12:00, Erik Rijkers <[email protected]> wrote:
> >
> > Op 07-09-2022 om 21:45 schreef Daniel Gustafsson:
> >> One thing this patchversion currently lacks is refined error messaging, but if
> >> we feel that this approach is a viable path then that can be tweaked. The
> >> function which starts the parser can also be refactored to be shared across
> >> pg_dump, pg_dumpall and pg_restore but I've kept it simple for now.
> >> Thoughts? It would be nice to get this patch across the finishline during this
> >> commitfest.
> >
> > > [0001-Add-include-exclude-filtering-via-file-in-pg_dump.patch]
> >
> > This seems to dump & restore well (as Pavels patch does).
>
> Thanks for looking!
>
> > I did notice one peculiarity (in your patch) where for each table a few spaces are omitted by pg_dump.
>
> Right, I had that on my TODO to fix before submitting but clearly forgot. It
> boils down to consuming the space between commands and object types and object
> patterns. The attached v2 fixes that.
I only had a quick look at the parser, and one thing that strikes me is:
+Patterns:
+ /* EMPTY */
+ | Patterns Pattern
+ | Pattern
+ ;
+
+Pattern:
+ C_INCLUDE include_object pattern { include_item(priv, $2, $3); }
It seems confusing to mix Pattern(s) (the rules) and pattern (the token).
Maybe instead using Include(s) or Item(s) on the bison side, and/or
name_pattern on the lexer side?
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-08 12:32 Daniel Gustafsson <[email protected]>
parent: Julien Rouhaud <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Daniel Gustafsson @ 2022-09-08 12:32 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Erik Rijkers <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 8 Sep 2022, at 13:44, Julien Rouhaud <[email protected]> wrote:
> On Thu, Sep 08, 2022 at 01:38:42PM +0200, Daniel Gustafsson wrote:
>>> On 8 Sep 2022, at 12:00, Erik Rijkers <[email protected]> wrote:
>>> I did notice one peculiarity (in your patch) where for each table a few spaces are omitted by pg_dump.
>>
>> Right, I had that on my TODO to fix before submitting but clearly forgot. It
>> boils down to consuming the space between commands and object types and object
>> patterns. The attached v2 fixes that.
>
> I only had a quick look at the parser,
Thanks for looking!
> .. and one thing that strikes me is:
>
> +Patterns:
> + /* EMPTY */
> + | Patterns Pattern
> + | Pattern
> + ;
> +
> +Pattern:
> + C_INCLUDE include_object pattern { include_item(priv, $2, $3); }
>
> It seems confusing to mix Pattern(s) (the rules) and pattern (the token).
> Maybe instead using Include(s) or Item(s) on the bison side, and/or
> name_pattern on the lexer side?
That makes a lot of sense, I renamed the rules in the parser but kept them in
the lexer since that seemed like the clearest scheme.
Also in the attached is a small refactoring to share parser init between
pg_dump and pg_restore (pg_dumpall shares little with these so not there for
now), buffer resize overflow calculation and some error message tweaking.
--
Daniel Gustafsson https://vmware.com/
Attachments:
[application/octet-stream] v3-0001-Add-include-exclude-filtering-via-file-in-pg_dump.patch (29.8K, ../../[email protected]/2-v3-0001-Add-include-exclude-filtering-via-file-in-pg_dump.patch)
download | inline diff:
From 6102f42501afc47b917afa583d88c2c22ead22d8 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Wed, 7 Sep 2022 15:22:05 +0200
Subject: [PATCH v3] Add include/exclude filtering via file in pg_dump
Author: Pavel Stehule <[email protected]>
Discussion: https://postgr.es/m/CAFj8pRB10wvW0CC9Xq=1XDs=zCQxer3cbLcNZa+qiX4cUH-G_A@mail.gmail.com
---
doc/src/sgml/ref/pg_dump.sgml | 88 +++++++++++++++++++
doc/src/sgml/ref/pg_dumpall.sgml | 22 +++++
doc/src/sgml/ref/pg_restore.sgml | 25 ++++++
src/bin/pg_dump/.gitignore | 4 +
src/bin/pg_dump/Makefile | 17 +++-
src/bin/pg_dump/filter.h | 44 ++++++++++
src/bin/pg_dump/filterparse.y | 65 ++++++++++++++
src/bin/pg_dump/filterscan.l | 139 ++++++++++++++++++++++++++++++
src/bin/pg_dump/pg_backup_utils.c | 33 +++++++
src/bin/pg_dump/pg_backup_utils.h | 1 +
src/bin/pg_dump/pg_dump.c | 56 ++++++++++++
src/bin/pg_dump/pg_dumpall.c | 65 +++++++++++++-
src/bin/pg_dump/pg_restore.c | 58 +++++++++++++
src/tools/msvc/Mkvcbuild.pm | 4 +-
14 files changed, 617 insertions(+), 4 deletions(-)
create mode 100644 src/bin/pg_dump/filter.h
create mode 100644 src/bin/pg_dump/filterparse.y
create mode 100644 src/bin/pg_dump/filterscan.l
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index c08276bc0a..b64bae6987 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 5d54074e01..46a667c17a 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpreted according to the same rules
+ as <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for excluding databases,
+ and can also be specified more than once for multiple filter files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 47bd7dbda0..ffeb564c52 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules as <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/.gitignore b/src/bin/pg_dump/.gitignore
index e6d7812779..11f2d68bea 100644
--- a/src/bin/pg_dump/.gitignore
+++ b/src/bin/pg_dump/.gitignore
@@ -2,4 +2,8 @@
/pg_dumpall
/pg_restore
+# Local generated source files
+/filterparse.c
+/filterscan.c
+
/tmp_check/
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 9dc5a784dd..e3befdc9b1 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -26,6 +26,8 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filterparse.o \
+ filterscan.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -37,14 +39,23 @@ OBJS = \
all: pg_dump pg_restore pg_dumpall
+# See notes in src/backend/parser/Makefile about the following two rules
+filterparse.h: filterparse.c
+ touch $@
+
+filterparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+filterparse.o filterscan.o: filterparse.h
+
pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dump.o common.o pg_dump_sort.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filterparse.o filterscan.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filterparse.o filterscan.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
@@ -63,6 +74,8 @@ installcheck:
uninstall:
rm -f $(addprefix '$(DESTDIR)$(bindir)'/, pg_dump$(X) pg_restore$(X) pg_dumpall$(X))
+distprep: filterparse.c filterscan.c
+
clean distclean maintainer-clean:
rm -f pg_dump$(X) pg_restore$(X) pg_dumpall$(X) $(OBJS) pg_dump.o common.o pg_dump_sort.o pg_restore.o pg_dumpall.o
rm -rf tmp_check
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..5dff4161f0
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+#include "c.h"
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern FILE *filter_yyin;
+
+extern int filter_yylex(void);
+extern void filter_yyerror(void *priv, const char *msg);
+extern void filter_scanner_init(void);
+extern void filter_scanner_finish(void);
+extern int filter_yyparse(void *priv);
+
+extern void include_item(void *priv, FilterObjectType objtype, const char *objname);
+extern void exclude_item(void *priv, FilterObjectType objtype, const char *objname);
+
+#endif
diff --git a/src/bin/pg_dump/filterparse.y b/src/bin/pg_dump/filterparse.y
new file mode 100644
index 0000000000..424719c012
--- /dev/null
+++ b/src/bin/pg_dump/filterparse.y
@@ -0,0 +1,65 @@
+%{
+/*-------------------------------------------------------------------------
+ *
+ * filterparse.y
+ * bison grammar for the pg_dump object filter files
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filterparse.y
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "filter.h"
+
+%}
+
+%name-prefix="filter_yy"
+%parse-param {void *priv}
+
+%union
+{
+ char *str;
+ int integer;
+}
+
+%type <integer> include_object exclude_object
+%token <str> pattern
+%token C_INCLUDE C_EXCLUDE
+%token OBJ_DATA OBJ_DATABASE OBJ_FOREIGN_DATA OBJ_FUNCTION OBJ_INDEX
+%token OBJ_SCHEMA OBJ_TABLE OBJ_TRIGGER
+%start Filters
+
+%%
+
+Filters:
+ /* EMPTY */
+ | Filters Filter
+ | Filter
+ ;
+
+Filter:
+ C_INCLUDE include_object pattern { include_item(priv, $2, $3); }
+ | C_EXCLUDE exclude_object pattern { exclude_item(priv, $2, $3); }
+ ;
+
+include_object:
+ OBJ_FOREIGN_DATA { $$ = FILTER_OBJECT_TYPE_FOREIGN_DATA; }
+ | OBJ_FUNCTION { $$ = FILTER_OBJECT_TYPE_FUNCTION; }
+ | OBJ_INDEX { $$ = FILTER_OBJECT_TYPE_INDEX; }
+ | OBJ_SCHEMA { $$ = FILTER_OBJECT_TYPE_SCHEMA; }
+ | OBJ_TABLE { $$ = FILTER_OBJECT_TYPE_TABLE; }
+ | OBJ_TRIGGER { $$ = FILTER_OBJECT_TYPE_TRIGGER; }
+ ;
+
+exclude_object:
+ OBJ_DATA { $$ = FILTER_OBJECT_TYPE_DATA; }
+ | OBJ_DATABASE { $$ = FILTER_OBJECT_TYPE_DATABASE; }
+ | OBJ_SCHEMA { $$ = FILTER_OBJECT_TYPE_SCHEMA; }
+ | OBJ_TABLE { $$ = FILTER_OBJECT_TYPE_TABLE; }
+ ;
+
+%%
diff --git a/src/bin/pg_dump/filterscan.l b/src/bin/pg_dump/filterscan.l
new file mode 100644
index 0000000000..0d7341086b
--- /dev/null
+++ b/src/bin/pg_dump/filterscan.l
@@ -0,0 +1,139 @@
+%top{
+/*-------------------------------------------------------------------------
+ *
+ * filterscan.l
+ * a lexical scanner for pg_dump object filter files
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filterscan.l
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+#include "filterparse.h"
+#include "filter.h"
+}
+
+%{
+
+static int yyline = 1;
+
+static void add_pattern_char(char c);
+
+#define PATTERNBUF_INIT 128
+
+static char *patternbuf = NULL;
+static size_t patternbufsize = PATTERNBUF_INIT;
+static size_t patternbufpos = 0;
+
+%}
+
+%option never-interactive
+%option noinput
+%option nounput
+%option noyywrap
+%option warn
+%option prefix="filter_yy"
+
+%x command
+%x type
+%x qident
+
+space [ \t\r\f\v]
+newline [\n]
+non_newline [^\n\r]
+
+comment ("#"{non_newline}*)
+pattern ([^"\n# \t\r\f\v]+)
+
+%%
+
+ /* Commands */
+include { BEGIN(command); return C_INCLUDE; }
+exclude { BEGIN(command); return C_EXCLUDE; }
+
+[\n] yyline++;
+{comment} { /* ignore */ }
+{space} { /* ignore */ }
+
+ /* Object types */
+<command>({space}*)data { BEGIN(type); return OBJ_DATA; }
+<command>({space}*)database { BEGIN(type); return OBJ_DATABASE; }
+<command>({space}*)foreign_data { BEGIN(type); return OBJ_FOREIGN_DATA; }
+<command>({space}*)function { BEGIN(type); return OBJ_FUNCTION; }
+<command>({space}*)index { BEGIN(type); return OBJ_INDEX; }
+<command>({space}*)schema { BEGIN(type); return OBJ_SCHEMA; }
+<command>({space}*)table { BEGIN(type); return OBJ_TABLE; }
+<command>({space}*)trigger { BEGIN(type); return OBJ_TRIGGER; }
+<command>{pattern} { filter_yyerror(NULL, "unsupported filter object type"); }
+<command><<EOF>> { filter_yyerror(NULL, "missing type name"); }
+
+ /* Patterns */
+<type><<EOF>> { filter_yyerror(NULL, "missing object name"); }
+<type>{space} { /* ignore */ }
+<type>{pattern} {
+ filter_yylval.str = pg_strdup(yytext);
+ BEGIN(INITIAL);
+ return(pattern);
+ }
+<type>\" {
+ patternbufpos = 0;
+ memset(patternbuf, 0, patternbufsize);
+ BEGIN(qident);
+ }
+<qident>\\\" { add_pattern_char(yytext[1]); }
+<qident>\" {
+ filter_yylval.str = pg_strdup(patternbuf);
+ BEGIN(INITIAL);
+ return(pattern);
+ }
+<qident>{newline} { add_pattern_char('\n'); }
+<qident>. { add_pattern_char(yytext[0]); }
+<qident><<EOF>> {
+ fprintf(stderr, "syntax error at line %i; unterminated quoted pattern: %s\n",
+ yyline, patternbuf);
+ exit(1);
+ }
+ /* Anything else is a syntax error */
+. { filter_yyerror(NULL, "unexpected character"); }
+%%
+
+static void
+add_pattern_char(char c)
+{
+ /* Leave room for trailing zero */
+ if (patternbufpos >= patternbufsize - 1)
+ {
+ /* Ensure that we don't overflow */
+ if (patternbufsize > (SIZE_MAX / 2))
+ {
+ fprintf(stderr, "out of memory\n");
+ exit(1);
+ }
+
+ /* Double the size of patternbuf if it gets full */
+ patternbufsize += patternbufsize;
+ patternbuf = pg_realloc(patternbuf, patternbufsize);
+ }
+ patternbuf[patternbufpos++] = c;
+}
+
+void
+filter_scanner_init(void)
+{
+ patternbuf = pg_malloc(patternbufsize);
+}
+
+void
+filter_scanner_finish(void)
+{
+ pfree(patternbuf);
+}
+
+void
+filter_yyerror(void *priv, const char *msg)
+{
+ fprintf(stderr, "syntax error at line %i: %s: \"%s\"\n", yyline, msg, yytext);
+ exit(1);
+}
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index e40890cb26..d6db156bae 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -13,6 +13,7 @@
*/
#include "postgres_fe.h"
+#include "filter.h"
#include "parallel.h"
#include "pg_backup_utils.h"
@@ -102,3 +103,35 @@ exit_nicely(int code)
exit(code);
}
+
+/*
+ * Retrieve object identifier patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+void
+parse_filters_from_file(const char *filename, void *priv)
+{
+ bool need_close = true;
+
+ if (strlen(filename) == 1 && filename[0] == '-')
+ {
+ filter_yyin = stdin;
+ need_close = false;
+ }
+ else
+ {
+ filter_yyin = fopen(filename, "r");
+ if (!filter_yyin)
+ pg_fatal("unable to open filterfile \"%s\": %m", filename);
+ }
+
+ filter_scanner_init();
+ filter_yyparse(priv);
+ filter_scanner_finish();
+
+ if (need_close)
+ fclose(filter_yyin);
+}
diff --git a/src/bin/pg_dump/pg_backup_utils.h b/src/bin/pg_dump/pg_backup_utils.h
index 8173bb93cf..74b942e712 100644
--- a/src/bin/pg_dump/pg_backup_utils.h
+++ b/src/bin/pg_dump/pg_backup_utils.h
@@ -30,6 +30,7 @@ extern const char *progname;
extern void set_dump_section(const char *arg, int *dumpSections);
extern void on_exit_nicely(on_exit_nicely_callback function, void *arg);
extern void exit_nicely(int code) pg_attribute_noreturn();
+extern void parse_filters_from_file(const char *filename, void *priv);
/* In pg_dump, we modify pg_fatal to call exit_nicely instead of exit */
#undef pg_fatal
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 67b6d9079e..d88adfe59a 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -58,6 +58,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -389,6 +390,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -622,6 +624,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ parse_filters_from_file(optarg, (void *) &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1027,6 +1033,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18197,3 +18205,51 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+void
+exclude_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ (void) priv;
+
+ switch (objtype)
+ {
+ case FILTER_OBJECT_TYPE_DATA:
+ simple_string_list_append(&tabledata_exclude_patterns, objname);
+ break;
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ simple_string_list_append(&schema_exclude_patterns, objname);
+ break;
+ case FILTER_OBJECT_TYPE_TABLE:
+ simple_string_list_append(&table_exclude_patterns, objname);
+ break;
+ default:
+ pg_log_error("Unsupported exclude object");
+ exit_nicely(1);
+ break;
+ }
+}
+
+void
+include_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ DumpOptions *dopt = (DumpOptions *) priv;
+
+ switch (objtype)
+ {
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ simple_string_list_append(&foreign_servers_include_patterns, objname);
+ break;
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ simple_string_list_append(&schema_include_patterns, objname);
+ dopt->include_everything = false;
+ break;
+ case FILTER_OBJECT_TYPE_TABLE:
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ break;
+ default:
+ pg_log_error("Unsupported include object");
+ exit_nicely(1);
+ break;
+ }
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 69ae027bd3..4e9d898387 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -26,6 +26,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -79,6 +80,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -156,6 +158,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -358,6 +361,11 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ getDatabaseExcludeFiltersFromFile(optarg,
+ &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -651,6 +659,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1888,7 +1897,6 @@ executeCommand(PGconn *conn, const char *query)
PQclear(res);
}
-
/*
* dumpTimestamp
*/
@@ -1912,3 +1920,58 @@ hash_string_pointer(char *s)
return hash_bytes(ss, strlen(s));
}
+
+/*
+ * getDatabaseFiltersFromFile - retrieve database identifier patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *pattern)
+{
+ bool need_close = true;
+
+ if (strlen(filename) == 1 && filename[0] == '-')
+ {
+ filter_yyin = stdin;
+ need_close = false;
+ }
+ else
+ {
+ filter_yyin = fopen(filename, "r");
+ if (!filter_yyin)
+ pg_fatal("unable to open filterfile \"%s\": %m", filename);
+ }
+
+ filter_scanner_init();
+ filter_yyparse((void *) pattern);
+ filter_scanner_finish();
+
+ if (need_close)
+ fclose(filter_yyin);
+}
+
+void
+exclude_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ SimpleStringList *pattern = (SimpleStringList *) priv;
+
+ if (objtype != FILTER_OBJECT_TYPE_DATABASE)
+ {
+ pg_log_error("Unsupported exclude object");
+ exit_nicely(1);
+ }
+
+ simple_string_list_append(pattern, objname);
+}
+
+void
+include_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ (void) priv;
+
+ pg_log_error("Unsupported include object");
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..70897d9056 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,6 +47,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
@@ -123,6 +124,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +288,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ parse_filters_from_file(optarg, (void *) opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +469,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +501,54 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+void
+exclude_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ RestoreOptions *opts = (RestoreOptions *) priv;
+
+ if (objtype != FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ pg_log_error("Unsupported exclude object");
+ exit_nicely(1);
+ }
+
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+}
+
+void
+include_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ RestoreOptions *opts = (RestoreOptions *) priv;
+
+ switch (objtype)
+ {
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_INDEX:
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ simple_string_list_append(&opts->schemaNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_TABLE:
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, optarg);
+ break;
+ default:
+ pg_log_error("Unsupported include object");
+ exit_nicely(1);
+ break;
+ }
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 098bc3f1b0..6418db9261 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -78,7 +78,9 @@ my $frontend_extraincludes = {
my $frontend_extrasource = {
'psql' => ['src/bin/psql/psqlscanslash.l'],
'pgbench' =>
- [ 'src/bin/pgbench/exprscan.l', 'src/bin/pgbench/exprparse.y' ]
+ [ 'src/bin/pgbench/exprscan.l', 'src/bin/pgbench/exprparse.y' ],
+ 'pg_dump' =>
+ ['src/bin/pg_dump/filterscan.l', 'src/bin/pg_dump/filterparse.y']
};
my @frontend_excludes = (
'pgevent', 'pg_basebackup', 'pg_rewind', 'pg_dump',
--
2.32.1 (Apple Git-133)
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-08 15:03 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2022-09-08 15:03 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
st 7. 9. 2022 v 21:46 odesílatel Daniel Gustafsson <[email protected]> napsal:
> As noted upthread at some point, I'm not overly excited about the parser in
> filter.c, for maintainability and readability reasons. So, I've
> reimplemented
> the parser in Flex/Bison in the attached patch, which IMHO provides a
> clear(er)
> picture of the grammar and is more per project standards. This version of
> the
> patch is your latest version with just the parser replaced (at a reduction
> in
> size as a side benefit).
>
> All features supported in your latest patch version are present, and it
> passes
> all the tests added by this patch. It's been an undisclosed amount of
> years
> since I wrote a Bison parser (well, yacc really) from scratch so I don't
> rule
> out having made silly mistakes. I would very much appreciate review from
> those
> more well versed in this area.
>
> One thing this patchversion currently lacks is refined error messaging,
> but if
> we feel that this approach is a viable path then that can be tweaked. The
> function which starts the parser can also be refactored to be shared across
> pg_dump, pg_dumpall and pg_restore but I've kept it simple for now.
>
> Thoughts? It would be nice to get this patch across the finishline during
> this
> commitfest.
>
>
I have no objections to this, and thank you so you try to move this patch
forward.
Regards
Pavel
--
> Daniel Gustafsson https://vmware.com/
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-09 07:53 John Naylor <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: John Naylor @ 2022-09-09 07:53 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Thu, Sep 8, 2022 at 7:32 PM Daniel Gustafsson <[email protected]> wrote:
> [v3]
Note that the grammar has shift-reduce conflicts. If you run a fairly
recent Bison, you can show them like this:
bison -Wno-deprecated -Wcounterexamples -d -o filterparse.c filterparse.y
filterparse.y: warning: 2 shift/reduce conflicts [-Wconflicts-sr]
filterparse.y: warning: shift/reduce conflict on token C_INCLUDE
[-Wcounterexamples]
Example: • C_INCLUDE include_object pattern
Shift derivation
Filters
↳ 3: Filter
↳ 4: • C_INCLUDE include_object pattern
Reduce derivation
Filters
↳ 2: Filters Filter
↳ 1: ε • ↳ 4: C_INCLUDE include_object pattern
filterparse.y: warning: shift/reduce conflict on token C_EXCLUDE
[-Wcounterexamples]
Example: • C_EXCLUDE exclude_object pattern
Shift derivation
Filters
↳ 3: Filter
↳ 5: • C_EXCLUDE exclude_object pattern
Reduce derivation
Filters
↳ 2: Filters Filter
↳ 1: ε • ↳ 5: C_EXCLUDE exclude_object pattern
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-09 09:00 Andrew Dunstan <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Andrew Dunstan @ 2022-09-09 09:00 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On Sep 9, 2022, at 5:53 PM, John Naylor <[email protected]> wrote:
>
> On Thu, Sep 8, 2022 at 7:32 PM Daniel Gustafsson <[email protected]> wrote:
>> [v3]
>
> Note that the grammar has shift-reduce conflicts. If you run a fairly
> recent Bison, you can show them like this:
>
> bison -Wno-deprecated -Wcounterexamples -d -o filterparse.c filterparse.y
>
> filterparse.y: warning: 2 shift/reduce conflicts [-Wconflicts-sr]
> filterparse.y: warning: shift/reduce conflict on token C_INCLUDE
> [-Wcounterexamples]
> Example: • C_INCLUDE include_object pattern
> Shift derivation
> Filters
> ↳ 3: Filter
> ↳ 4: • C_INCLUDE include_object pattern
> Reduce derivation
> Filters
> ↳ 2: Filters Filter
> ↳ 1: ε • ↳ 4: C_INCLUDE include_object pattern
> filterparse.y: warning: shift/reduce conflict on token C_EXCLUDE
> [-Wcounterexamples]
> Example: • C_EXCLUDE exclude_object pattern
> Shift derivation
> Filters
> ↳ 3: Filter
> ↳ 5: • C_EXCLUDE exclude_object pattern
> Reduce derivation
> Filters
> ↳ 2: Filters Filter
> ↳ 1: ε • ↳ 5: C_EXCLUDE exclude_object pattern
>
Looks like the last rule for Filters should not be there. I do wonder whether we should be using bison/flex here, seems like using a sledgehammer to crack a nut.
Cheers
Andrew
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-12 07:58 Daniel Gustafsson <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 4 replies; 181+ messages in thread
From: Daniel Gustafsson @ 2022-09-12 07:58 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: John Naylor <[email protected]>; Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 9 Sep 2022, at 11:00, Andrew Dunstan <[email protected]> wrote:
>
>> On Sep 9, 2022, at 5:53 PM, John Naylor <[email protected]> wrote:
>>
>> Note that the grammar has shift-reduce conflicts.
> Looks like the last rule for Filters should not be there.
Correct, fixed in the attached.
> I do wonder whether we should be using bison/flex here, seems like using a
> sledgehammer to crack a nut.
I don't the capabilities of the tool is all that interesting compared to the
long term maintainability and readability of the source code. Personally I
think a simple Bison/Flex parser is easier to read and reason about than the
corresponding written in C.
--
Daniel Gustafsson https://vmware.com/
Attachments:
[application/octet-stream] v4-0001-Add-include-exclude-filtering-via-file-in-pg_dump.patch (29.8K, ../../[email protected]/2-v4-0001-Add-include-exclude-filtering-via-file-in-pg_dump.patch)
download | inline diff:
From 5d36ef7c3814e60268e3d0dba36856f207bd1704 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Wed, 7 Sep 2022 15:22:05 +0200
Subject: [PATCH v4] Add include/exclude filtering via file in pg_dump
Author: Pavel Stehule <[email protected]>
Discussion: https://postgr.es/m/CAFj8pRB10wvW0CC9Xq=1XDs=zCQxer3cbLcNZa+qiX4cUH-G_A@mail.gmail.com
---
doc/src/sgml/ref/pg_dump.sgml | 88 +++++++++++++++++++
doc/src/sgml/ref/pg_dumpall.sgml | 22 +++++
doc/src/sgml/ref/pg_restore.sgml | 25 ++++++
src/bin/pg_dump/.gitignore | 4 +
src/bin/pg_dump/Makefile | 17 +++-
src/bin/pg_dump/filter.h | 44 ++++++++++
src/bin/pg_dump/filterparse.y | 64 ++++++++++++++
src/bin/pg_dump/filterscan.l | 139 ++++++++++++++++++++++++++++++
src/bin/pg_dump/pg_backup_utils.c | 33 +++++++
src/bin/pg_dump/pg_backup_utils.h | 1 +
src/bin/pg_dump/pg_dump.c | 56 ++++++++++++
src/bin/pg_dump/pg_dumpall.c | 65 +++++++++++++-
src/bin/pg_dump/pg_restore.c | 58 +++++++++++++
src/tools/msvc/Mkvcbuild.pm | 4 +-
14 files changed, 616 insertions(+), 4 deletions(-)
create mode 100644 src/bin/pg_dump/filter.h
create mode 100644 src/bin/pg_dump/filterparse.y
create mode 100644 src/bin/pg_dump/filterscan.l
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index c08276bc0a..b64bae6987 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 5d54074e01..46a667c17a 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpreted according to the same rules
+ as <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for excluding databases,
+ and can also be specified more than once for multiple filter files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 47bd7dbda0..ffeb564c52 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules as <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/.gitignore b/src/bin/pg_dump/.gitignore
index e6d7812779..11f2d68bea 100644
--- a/src/bin/pg_dump/.gitignore
+++ b/src/bin/pg_dump/.gitignore
@@ -2,4 +2,8 @@
/pg_dumpall
/pg_restore
+# Local generated source files
+/filterparse.c
+/filterscan.c
+
/tmp_check/
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 9dc5a784dd..e3befdc9b1 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -26,6 +26,8 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filterparse.o \
+ filterscan.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -37,14 +39,23 @@ OBJS = \
all: pg_dump pg_restore pg_dumpall
+# See notes in src/backend/parser/Makefile about the following two rules
+filterparse.h: filterparse.c
+ touch $@
+
+filterparse.c: BISONFLAGS += -d
+
+# Force these dependencies to be known even without dependency info built:
+filterparse.o filterscan.o: filterparse.h
+
pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dump.o common.o pg_dump_sort.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filterparse.o filterscan.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filterparse.o filterscan.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
@@ -63,6 +74,8 @@ installcheck:
uninstall:
rm -f $(addprefix '$(DESTDIR)$(bindir)'/, pg_dump$(X) pg_restore$(X) pg_dumpall$(X))
+distprep: filterparse.c filterscan.c
+
clean distclean maintainer-clean:
rm -f pg_dump$(X) pg_restore$(X) pg_dumpall$(X) $(OBJS) pg_dump.o common.o pg_dump_sort.o pg_restore.o pg_dumpall.o
rm -rf tmp_check
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..5dff4161f0
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+#include "c.h"
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern FILE *filter_yyin;
+
+extern int filter_yylex(void);
+extern void filter_yyerror(void *priv, const char *msg);
+extern void filter_scanner_init(void);
+extern void filter_scanner_finish(void);
+extern int filter_yyparse(void *priv);
+
+extern void include_item(void *priv, FilterObjectType objtype, const char *objname);
+extern void exclude_item(void *priv, FilterObjectType objtype, const char *objname);
+
+#endif
diff --git a/src/bin/pg_dump/filterparse.y b/src/bin/pg_dump/filterparse.y
new file mode 100644
index 0000000000..c7f06f788d
--- /dev/null
+++ b/src/bin/pg_dump/filterparse.y
@@ -0,0 +1,64 @@
+%{
+/*-------------------------------------------------------------------------
+ *
+ * filterparse.y
+ * bison grammar for the pg_dump object filter files
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filterparse.y
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "filter.h"
+
+%}
+
+%name-prefix="filter_yy"
+%parse-param {void *priv}
+
+%union
+{
+ char *str;
+ int integer;
+}
+
+%type <integer> include_object exclude_object
+%token <str> pattern
+%token C_INCLUDE C_EXCLUDE
+%token OBJ_DATA OBJ_DATABASE OBJ_FOREIGN_DATA OBJ_FUNCTION OBJ_INDEX
+%token OBJ_SCHEMA OBJ_TABLE OBJ_TRIGGER
+%start Filters
+
+%%
+
+Filters:
+ /* EMPTY */
+ | Filters Filter
+ ;
+
+Filter:
+ C_INCLUDE include_object pattern { include_item(priv, $2, $3); }
+ | C_EXCLUDE exclude_object pattern { exclude_item(priv, $2, $3); }
+ ;
+
+include_object:
+ OBJ_FOREIGN_DATA { $$ = FILTER_OBJECT_TYPE_FOREIGN_DATA; }
+ | OBJ_FUNCTION { $$ = FILTER_OBJECT_TYPE_FUNCTION; }
+ | OBJ_INDEX { $$ = FILTER_OBJECT_TYPE_INDEX; }
+ | OBJ_SCHEMA { $$ = FILTER_OBJECT_TYPE_SCHEMA; }
+ | OBJ_TABLE { $$ = FILTER_OBJECT_TYPE_TABLE; }
+ | OBJ_TRIGGER { $$ = FILTER_OBJECT_TYPE_TRIGGER; }
+ ;
+
+exclude_object:
+ OBJ_DATA { $$ = FILTER_OBJECT_TYPE_DATA; }
+ | OBJ_DATABASE { $$ = FILTER_OBJECT_TYPE_DATABASE; }
+ | OBJ_SCHEMA { $$ = FILTER_OBJECT_TYPE_SCHEMA; }
+ | OBJ_TABLE { $$ = FILTER_OBJECT_TYPE_TABLE; }
+ ;
+
+%%
diff --git a/src/bin/pg_dump/filterscan.l b/src/bin/pg_dump/filterscan.l
new file mode 100644
index 0000000000..0d7341086b
--- /dev/null
+++ b/src/bin/pg_dump/filterscan.l
@@ -0,0 +1,139 @@
+%top{
+/*-------------------------------------------------------------------------
+ *
+ * filterscan.l
+ * a lexical scanner for pg_dump object filter files
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filterscan.l
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+#include "filterparse.h"
+#include "filter.h"
+}
+
+%{
+
+static int yyline = 1;
+
+static void add_pattern_char(char c);
+
+#define PATTERNBUF_INIT 128
+
+static char *patternbuf = NULL;
+static size_t patternbufsize = PATTERNBUF_INIT;
+static size_t patternbufpos = 0;
+
+%}
+
+%option never-interactive
+%option noinput
+%option nounput
+%option noyywrap
+%option warn
+%option prefix="filter_yy"
+
+%x command
+%x type
+%x qident
+
+space [ \t\r\f\v]
+newline [\n]
+non_newline [^\n\r]
+
+comment ("#"{non_newline}*)
+pattern ([^"\n# \t\r\f\v]+)
+
+%%
+
+ /* Commands */
+include { BEGIN(command); return C_INCLUDE; }
+exclude { BEGIN(command); return C_EXCLUDE; }
+
+[\n] yyline++;
+{comment} { /* ignore */ }
+{space} { /* ignore */ }
+
+ /* Object types */
+<command>({space}*)data { BEGIN(type); return OBJ_DATA; }
+<command>({space}*)database { BEGIN(type); return OBJ_DATABASE; }
+<command>({space}*)foreign_data { BEGIN(type); return OBJ_FOREIGN_DATA; }
+<command>({space}*)function { BEGIN(type); return OBJ_FUNCTION; }
+<command>({space}*)index { BEGIN(type); return OBJ_INDEX; }
+<command>({space}*)schema { BEGIN(type); return OBJ_SCHEMA; }
+<command>({space}*)table { BEGIN(type); return OBJ_TABLE; }
+<command>({space}*)trigger { BEGIN(type); return OBJ_TRIGGER; }
+<command>{pattern} { filter_yyerror(NULL, "unsupported filter object type"); }
+<command><<EOF>> { filter_yyerror(NULL, "missing type name"); }
+
+ /* Patterns */
+<type><<EOF>> { filter_yyerror(NULL, "missing object name"); }
+<type>{space} { /* ignore */ }
+<type>{pattern} {
+ filter_yylval.str = pg_strdup(yytext);
+ BEGIN(INITIAL);
+ return(pattern);
+ }
+<type>\" {
+ patternbufpos = 0;
+ memset(patternbuf, 0, patternbufsize);
+ BEGIN(qident);
+ }
+<qident>\\\" { add_pattern_char(yytext[1]); }
+<qident>\" {
+ filter_yylval.str = pg_strdup(patternbuf);
+ BEGIN(INITIAL);
+ return(pattern);
+ }
+<qident>{newline} { add_pattern_char('\n'); }
+<qident>. { add_pattern_char(yytext[0]); }
+<qident><<EOF>> {
+ fprintf(stderr, "syntax error at line %i; unterminated quoted pattern: %s\n",
+ yyline, patternbuf);
+ exit(1);
+ }
+ /* Anything else is a syntax error */
+. { filter_yyerror(NULL, "unexpected character"); }
+%%
+
+static void
+add_pattern_char(char c)
+{
+ /* Leave room for trailing zero */
+ if (patternbufpos >= patternbufsize - 1)
+ {
+ /* Ensure that we don't overflow */
+ if (patternbufsize > (SIZE_MAX / 2))
+ {
+ fprintf(stderr, "out of memory\n");
+ exit(1);
+ }
+
+ /* Double the size of patternbuf if it gets full */
+ patternbufsize += patternbufsize;
+ patternbuf = pg_realloc(patternbuf, patternbufsize);
+ }
+ patternbuf[patternbufpos++] = c;
+}
+
+void
+filter_scanner_init(void)
+{
+ patternbuf = pg_malloc(patternbufsize);
+}
+
+void
+filter_scanner_finish(void)
+{
+ pfree(patternbuf);
+}
+
+void
+filter_yyerror(void *priv, const char *msg)
+{
+ fprintf(stderr, "syntax error at line %i: %s: \"%s\"\n", yyline, msg, yytext);
+ exit(1);
+}
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index e40890cb26..d6db156bae 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -13,6 +13,7 @@
*/
#include "postgres_fe.h"
+#include "filter.h"
#include "parallel.h"
#include "pg_backup_utils.h"
@@ -102,3 +103,35 @@ exit_nicely(int code)
exit(code);
}
+
+/*
+ * Retrieve object identifier patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+void
+parse_filters_from_file(const char *filename, void *priv)
+{
+ bool need_close = true;
+
+ if (strlen(filename) == 1 && filename[0] == '-')
+ {
+ filter_yyin = stdin;
+ need_close = false;
+ }
+ else
+ {
+ filter_yyin = fopen(filename, "r");
+ if (!filter_yyin)
+ pg_fatal("unable to open filterfile \"%s\": %m", filename);
+ }
+
+ filter_scanner_init();
+ filter_yyparse(priv);
+ filter_scanner_finish();
+
+ if (need_close)
+ fclose(filter_yyin);
+}
diff --git a/src/bin/pg_dump/pg_backup_utils.h b/src/bin/pg_dump/pg_backup_utils.h
index 8173bb93cf..74b942e712 100644
--- a/src/bin/pg_dump/pg_backup_utils.h
+++ b/src/bin/pg_dump/pg_backup_utils.h
@@ -30,6 +30,7 @@ extern const char *progname;
extern void set_dump_section(const char *arg, int *dumpSections);
extern void on_exit_nicely(on_exit_nicely_callback function, void *arg);
extern void exit_nicely(int code) pg_attribute_noreturn();
+extern void parse_filters_from_file(const char *filename, void *priv);
/* In pg_dump, we modify pg_fatal to call exit_nicely instead of exit */
#undef pg_fatal
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 67b6d9079e..d88adfe59a 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -58,6 +58,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -389,6 +390,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -622,6 +624,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ parse_filters_from_file(optarg, (void *) &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1027,6 +1033,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18197,3 +18205,51 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+void
+exclude_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ (void) priv;
+
+ switch (objtype)
+ {
+ case FILTER_OBJECT_TYPE_DATA:
+ simple_string_list_append(&tabledata_exclude_patterns, objname);
+ break;
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ simple_string_list_append(&schema_exclude_patterns, objname);
+ break;
+ case FILTER_OBJECT_TYPE_TABLE:
+ simple_string_list_append(&table_exclude_patterns, objname);
+ break;
+ default:
+ pg_log_error("Unsupported exclude object");
+ exit_nicely(1);
+ break;
+ }
+}
+
+void
+include_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ DumpOptions *dopt = (DumpOptions *) priv;
+
+ switch (objtype)
+ {
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ simple_string_list_append(&foreign_servers_include_patterns, objname);
+ break;
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ simple_string_list_append(&schema_include_patterns, objname);
+ dopt->include_everything = false;
+ break;
+ case FILTER_OBJECT_TYPE_TABLE:
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ break;
+ default:
+ pg_log_error("Unsupported include object");
+ exit_nicely(1);
+ break;
+ }
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 69ae027bd3..4e9d898387 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -26,6 +26,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -79,6 +80,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -156,6 +158,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -358,6 +361,11 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ getDatabaseExcludeFiltersFromFile(optarg,
+ &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -651,6 +659,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1888,7 +1897,6 @@ executeCommand(PGconn *conn, const char *query)
PQclear(res);
}
-
/*
* dumpTimestamp
*/
@@ -1912,3 +1920,58 @@ hash_string_pointer(char *s)
return hash_bytes(ss, strlen(s));
}
+
+/*
+ * getDatabaseFiltersFromFile - retrieve database identifier patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+getDatabaseExcludeFiltersFromFile(const char *filename, SimpleStringList *pattern)
+{
+ bool need_close = true;
+
+ if (strlen(filename) == 1 && filename[0] == '-')
+ {
+ filter_yyin = stdin;
+ need_close = false;
+ }
+ else
+ {
+ filter_yyin = fopen(filename, "r");
+ if (!filter_yyin)
+ pg_fatal("unable to open filterfile \"%s\": %m", filename);
+ }
+
+ filter_scanner_init();
+ filter_yyparse((void *) pattern);
+ filter_scanner_finish();
+
+ if (need_close)
+ fclose(filter_yyin);
+}
+
+void
+exclude_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ SimpleStringList *pattern = (SimpleStringList *) priv;
+
+ if (objtype != FILTER_OBJECT_TYPE_DATABASE)
+ {
+ pg_log_error("Unsupported exclude object");
+ exit_nicely(1);
+ }
+
+ simple_string_list_append(pattern, objname);
+}
+
+void
+include_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ (void) priv;
+
+ pg_log_error("Unsupported include object");
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..70897d9056 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,6 +47,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
@@ -123,6 +124,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +288,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ parse_filters_from_file(optarg, (void *) opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +469,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +501,54 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+void
+exclude_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ RestoreOptions *opts = (RestoreOptions *) priv;
+
+ if (objtype != FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ pg_log_error("Unsupported exclude object");
+ exit_nicely(1);
+ }
+
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+}
+
+void
+include_item(void *priv, FilterObjectType objtype, const char *objname)
+{
+ RestoreOptions *opts = (RestoreOptions *) priv;
+
+ switch (objtype)
+ {
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_INDEX:
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ simple_string_list_append(&opts->schemaNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_TABLE:
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ break;
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, optarg);
+ break;
+ default:
+ pg_log_error("Unsupported include object");
+ exit_nicely(1);
+ break;
+ }
+}
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 098bc3f1b0..6418db9261 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -78,7 +78,9 @@ my $frontend_extraincludes = {
my $frontend_extrasource = {
'psql' => ['src/bin/psql/psqlscanslash.l'],
'pgbench' =>
- [ 'src/bin/pgbench/exprscan.l', 'src/bin/pgbench/exprparse.y' ]
+ [ 'src/bin/pgbench/exprscan.l', 'src/bin/pgbench/exprparse.y' ],
+ 'pg_dump' =>
+ ['src/bin/pg_dump/filterscan.l', 'src/bin/pg_dump/filterparse.y']
};
my @frontend_excludes = (
'pgevent', 'pg_basebackup', 'pg_rewind', 'pg_dump',
--
2.32.1 (Apple Git-133)
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-12 13:09 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
3 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2022-09-12 13:09 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
po 12. 9. 2022 v 9:59 odesílatel Daniel Gustafsson <[email protected]> napsal:
> > On 9 Sep 2022, at 11:00, Andrew Dunstan <[email protected]> wrote:
> >
> >> On Sep 9, 2022, at 5:53 PM, John Naylor <[email protected]>
> wrote:
> >>
> >> Note that the grammar has shift-reduce conflicts.
>
> > Looks like the last rule for Filters should not be there.
>
> Correct, fixed in the attached.
>
> > I do wonder whether we should be using bison/flex here, seems like using
> a
> > sledgehammer to crack a nut.
>
>
> I don't the capabilities of the tool is all that interesting compared to
> the
> long term maintainability and readability of the source code. Personally I
> think a simple Bison/Flex parser is easier to read and reason about than
> the
> corresponding written in C.
>
When this work is done, then there is no reason to throw it. The parser in
bison/flex does the same work and it is true, so code is more readable.
Although for this case, a handy written parser was trivial too.
Regards
Pavel
>
> --
> Daniel Gustafsson https://vmware.com/
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-12 14:00 Erik Rijkers <[email protected]>
parent: Daniel Gustafsson <[email protected]>
3 siblings, 1 reply; 181+ messages in thread
From: Erik Rijkers @ 2022-09-12 14:00 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; +Cc: John Naylor <[email protected]>; Julien Rouhaud <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Op 12-09-2022 om 09:58 schreef Daniel Gustafsson:
>> On 9 Sep 2022, at 11:00, Andrew Dunstan <[email protected]> wrote:
>>
>>> On Sep 9, 2022, at 5:53 PM, John Naylor <[email protected]> wrote:
>>> [v4-0001-Add-include-exclude-filtering-via-file-in-pg_dump.patch]
I noticed that pg_restore --filter cannot, or at last not always, be
used with the same filter-file that was used to produce a dump with
pg_dump --filter.
Is that as designed? It seems a bit counterintuitive. It'd be nice if
that could be fixed. Admittedly, the 'same' problem in pg_restore -t,
also less than ideal.
(A messy bashdemo below)
thanks,
Erik Rijkers
#! /bin/bash
db2='testdb2' db3='testdb3'
db2='testdb_source' db3='testdb_target'
sql_dropdb="drop database if exists $db2; drop database if exists $db3;"
sql_createdb="create database $db2; create database $db3;"
schema1=s1 table1=table1 t1=$schema1.$table1
schema2=s2 table2=table2 t2=$schema2.$table2
sql_schema_init="create schema if not exists $schema1; create schema if
not exists $schema2;"
sql_test="select '$t1', n from $t1 order by n; select '$t2', n from $t2
order by n;"
function sqltest()
{
for database_name in $db2 $db3 ;do
port_used=$( echo "show port" |psql -qtAX -d $database_name )
echo -n "-- $database_name ($port_used): "
echo "$sql_test" | psql -qtAX -a -d $database_name | md5sum
done
echo
}
echo "setting up orig db $db2, target db $db3"
echo "$sql_dropdb" | psql -qtAX
echo "$sql_createdb" | psql -qtAX
psql -X -d $db2 << SQL
$sql_schema_init
create table $t1 as select n from generate_series(1, (10^1)::int) as f(n);
create table $t2 as select n from generate_series(2, (10^2)::int) as f(n);
SQL
echo "
include table $t1
include table $t2
# include schema $s1
# include schema $s2
" > inputfile1.txt
# in filter; out plain
echo "-- pg_dump -F p -f plainfile1 --filter=inputfile1.txt -d $db2"
pg_dump -F p -f plainfile1 --filter=inputfile1.txt -d $db2
echo "$sql_schema_init" | psql -qX -d $db3
echo "-- pg_restore -d $db3 dumpfile1"
pg_restore -d $db3 dumpfile1
rc=$?
echo "-- pg_restore returned [$rc] -- pg_restore without --filter"
sqltest
# enable this to see it fail
if [[ 1 -eq 1 ]]
then
# clean out
echo "drop schema $schema1 cascade; drop schema $schema2 cascade; " |
psql -qtAXad $db3
--filter=inputfile1.txt"
echo "$sql_schema_init" | psql -qX -d $db3
echo "-- pg_restore -d $db3 --filter=inputfile1.txt dumpfile1"
pg_restore -d $db3 --filter=inputfile1.txt dumpfile1
rc=$?
echo "-- pg_restore returned [$rc] -- pg_restore without --filter"
sqltest
fi
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-12 14:18 Erik Rijkers <[email protected]>
parent: Erik Rijkers <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Erik Rijkers @ 2022-09-12 14:18 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; +Cc: John Naylor <[email protected]>; Julien Rouhaud <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Op 12-09-2022 om 16:00 schreef Erik Rijkers:
> Op 12-09-2022 om 09:58 schreef Daniel Gustafsson:
>>> On 9 Sep 2022, at 11:00, Andrew Dunstan <[email protected]> wrote:
>>>
>>>> On Sep 9, 2022, at 5:53 PM, John Naylor
>>>> <[email protected]> wrote:
>
>>>> [v4-0001-Add-include-exclude-filtering-via-file-in-pg_dump.patch]
>
> I noticed that pg_restore --filter cannot, or at last not always, be
> used with the same filter-file that was used to produce a dump with
> pg_dump --filter.
>
> Is that as designed? It seems a bit counterintuitive. It'd be nice if
> that could be fixed. Admittedly, the 'same' problem in pg_restore -t,
> also less than ideal.
>
> (A messy bashdemo below)
I hope the issue is still clear, even though in the bash I sent, I
messed up the dumpfile name (i.e., in the bash that I sent the pg_dump
creates another dump name than what is given to pg_restore. They should
use the same dumpname, obviously)
>
> thanks,
>
> Erik Rijkers
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-13 08:46 John Naylor <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: John Naylor @ 2022-09-13 08:46 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Mon, Sep 12, 2022 at 8:10 PM Pavel Stehule <[email protected]> wrote:
>
> po 12. 9. 2022 v 9:59 odesílatel Daniel Gustafsson <[email protected]> napsal:
>> I don't the capabilities of the tool is all that interesting compared to the
>> long term maintainability and readability of the source code.
With make distprep and maintainer-clean, separate makefile and MSVC
build logic a short time before converting to Meson, I'm not sure that
even the short term maintainability here is a good trade off for what
we're getting.
> The parser in bison/flex does the same work and it is true, so code is more readable. Although for this case, a handy written parser was trivial too.
If the hand-written version is trivial, then we should prefer it.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-13 09:36 Pavel Stehule <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2022-09-13 09:36 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
út 13. 9. 2022 v 10:46 odesílatel John Naylor <[email protected]>
napsal:
> On Mon, Sep 12, 2022 at 8:10 PM Pavel Stehule <[email protected]>
> wrote:
> >
> > po 12. 9. 2022 v 9:59 odesílatel Daniel Gustafsson <[email protected]>
> napsal:
> >> I don't the capabilities of the tool is all that interesting compared
> to the
> >> long term maintainability and readability of the source code.
>
> With make distprep and maintainer-clean, separate makefile and MSVC
> build logic a short time before converting to Meson, I'm not sure that
> even the short term maintainability here is a good trade off for what
> we're getting.
>
> > The parser in bison/flex does the same work and it is true, so code is
> more readable. Although for this case, a handy written parser was trivial
> too.
>
> If the hand-written version is trivial, then we should prefer it.
>
Please, can you check and compare both versions? My view is subjective.
Regards
Pavel
> --
> John Naylor
> EDB: http://www.enterprisedb.com
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-09-22 15:10 Andres Freund <[email protected]>
parent: Daniel Gustafsson <[email protected]>
3 siblings, 0 replies; 181+ messages in thread
From: Andres Freund @ 2022-09-22 15:10 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi,
On 2022-09-12 09:58:37 +0200, Daniel Gustafsson wrote:
> > On 9 Sep 2022, at 11:00, Andrew Dunstan <[email protected]> wrote:
> >
> >> On Sep 9, 2022, at 5:53 PM, John Naylor <[email protected]> wrote:
> >>
> >> Note that the grammar has shift-reduce conflicts.
>
> > Looks like the last rule for Filters should not be there.
>
> Correct, fixed in the attached.
Due to the merge of the meson build, this patch now needs to adjust the
relevant meson.build. This is the cause of the failures at:
https://cirrus-ci.com/build/5788292678418432
See e.g. src/bin/pgbench/meson.build
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-02 06:56 Andres Freund <[email protected]>
parent: Daniel Gustafsson <[email protected]>
3 siblings, 1 reply; 181+ messages in thread
From: Andres Freund @ 2022-10-02 06:56 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi,
On 2022-09-12 09:58:37 +0200, Daniel Gustafsson wrote:
> Correct, fixed in the attached.
Updated patch adding meson compatibility attached.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-02 07:19 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Andres Freund @ 2022-10-02 07:19 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi,
On 2022-10-01 23:56:59 -0700, Andres Freund wrote:
> On 2022-09-12 09:58:37 +0200, Daniel Gustafsson wrote:
> > Correct, fixed in the attached.
>
> Updated patch adding meson compatibility attached.
Err, forgot to amend one hunk :(
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-02 16:04 Andres Freund <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Andres Freund @ 2022-10-02 16:04 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi,
On 2022-10-02 00:19:59 -0700, Andres Freund wrote:
> On 2022-10-01 23:56:59 -0700, Andres Freund wrote:
> > On 2022-09-12 09:58:37 +0200, Daniel Gustafsson wrote:
> > > Correct, fixed in the attached.
> >
> > Updated patch adding meson compatibility attached.
>
> Err, forgot to amend one hunk :(
That fixed it on all platforms but windows, due to copy-pasto. I really should
have stopped earlier yesterday...
> +/*-------------------------------------------------------------------------
> + *
> + * filter.h
> + * Common header file for the parser of filter file
> + *
> + * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
> + * Portions Copyright (c) 1994, Regents of the University of California
> + *
> + * src/bin/pg_dump/filter.h
> + *
> + *-------------------------------------------------------------------------
> + */
> +#ifndef FILTER_H
> +#define FILTER_H
> +#include "c.h"
c.h (and postgres.h, postgres_fe.h) shouldn't be included in headers.
This is a common enough mistake that I'm wondering if we could automate
warning about it somehow.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-02 20:52 Daniel Gustafsson <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 3 replies; 181+ messages in thread
From: Daniel Gustafsson @ 2022-10-02 20:52 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
> On 2 Oct 2022, at 18:04, Andres Freund <[email protected]> wrote:
> On 2022-10-02 00:19:59 -0700, Andres Freund wrote:
>> On 2022-10-01 23:56:59 -0700, Andres Freund wrote:
>>> On 2022-09-12 09:58:37 +0200, Daniel Gustafsson wrote:
>>>> Correct, fixed in the attached.
>>>
>>> Updated patch adding meson compatibility attached.
>>
>> Err, forgot to amend one hunk :(
>
> That fixed it on all platforms but windows, due to copy-pasto. I really should
> have stopped earlier yesterday...
Thanks for updating the patch!
The parser in the original submission was -1'd by me, and the current version
proposed as an alternative. This was subsequently -1'd as well but no updated
patch with a rewritten parser has been posted. So this is now stalled again.
Having been around in 12 commitfests without a committer feeling confident
about pushing this I plan to mark it returned with feedback, and if a new
parser materializes itc can be readded instead of being dragged along.
> c.h (and postgres.h, postgres_fe.h) shouldn't be included in headers.
>
> This is a common enough mistake that I'm wondering if we could automate
> warning about it somehow.
Maybe we can add a simple git grep invocation in the CompilerWarnings CI job to
catch this in the CFBot? If something like the below sketch matches then we
can throw an error. (only for illustration, all three files needs to checked).
git grep "\"c\.h" -- *.h :^src/include/postgres*.h;
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-02 21:02 Tom Lane <[email protected]>
parent: Daniel Gustafsson <[email protected]>
2 siblings, 0 replies; 181+ messages in thread
From: Tom Lane @ 2022-10-02 21:02 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Andres Freund <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Daniel Gustafsson <[email protected]> writes:
> On 2 Oct 2022, at 18:04, Andres Freund <[email protected]> wrote:
>> c.h (and postgres.h, postgres_fe.h) shouldn't be included in headers.
>> This is a common enough mistake that I'm wondering if we could automate
>> warning about it somehow.
> Maybe we can add a simple git grep invocation in the CompilerWarnings CI job to
> catch this in the CFBot?
I'd be inclined to teach headerscheck or cpluspluscheck about it.
regards, tom lane
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-02 21:11 Andres Freund <[email protected]>
parent: Daniel Gustafsson <[email protected]>
2 siblings, 0 replies; 181+ messages in thread
From: Andres Freund @ 2022-10-02 21:11 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Pavel Stehule <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On 2022-10-02 22:52:33 +0200, Daniel Gustafsson wrote:
> The parser in the original submission was -1'd by me, and the current version
> proposed as an alternative. This was subsequently -1'd as well but no updated
> patch with a rewritten parser has been posted. So this is now stalled again.
>
> Having been around in 12 commitfests without a committer feeling confident
> about pushing this I plan to mark it returned with feedback, and if a new
> parser materializes itc can be readded instead of being dragged along.
Makes sense to me.
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-03 04:00 Pavel Stehule <[email protected]>
parent: Daniel Gustafsson <[email protected]>
2 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2022-10-03 04:00 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Andres Freund <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Julien Rouhaud <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
ne 2. 10. 2022 v 22:52 odesílatel Daniel Gustafsson <[email protected]>
napsal:
> > On 2 Oct 2022, at 18:04, Andres Freund <[email protected]> wrote:
> > On 2022-10-02 00:19:59 -0700, Andres Freund wrote:
> >> On 2022-10-01 23:56:59 -0700, Andres Freund wrote:
> >>> On 2022-09-12 09:58:37 +0200, Daniel Gustafsson wrote:
> >>>> Correct, fixed in the attached.
> >>>
> >>> Updated patch adding meson compatibility attached.
> >>
> >> Err, forgot to amend one hunk :(
> >
> > That fixed it on all platforms but windows, due to copy-pasto. I really
> should
> > have stopped earlier yesterday...
>
> Thanks for updating the patch!
>
> The parser in the original submission was -1'd by me, and the current
> version
> proposed as an alternative. This was subsequently -1'd as well but no
> updated
> patch with a rewritten parser has been posted. So this is now stalled
> again.
>
You started rewriting it, but you didn't finish it.
Unfortunately, there is not a clean opinion on using bison's parser for
this purpose. I understand that the complexity of this language is too low,
so the benefit of using bison's gramatic is low too. Personally, I have not
any problem using bison for this purpose. For this case, I think we compare
two similarly long ways, but unfortunately, customers that have a problem
with long command lines still have this problem.
Can we go forward? Daniel is strongly against handwritten parser. Is there
somebody strongly against bison's based parser? There is not any other way.
I am able to complete Daniel's patch, if there will not be objections.
Regards
Pavel
> Having been around in 12 commitfests without a committer feeling confident
> about pushing this I plan to mark it returned with feedback, and if a new
> parser materializes itc can be readded instead of being dragged along.
>
> > c.h (and postgres.h, postgres_fe.h) shouldn't be included in headers.
> >
> > This is a common enough mistake that I'm wondering if we could automate
> > warning about it somehow.
>
> Maybe we can add a simple git grep invocation in the CompilerWarnings CI
> job to
> catch this in the CFBot? If something like the below sketch matches then
> we
> can throw an error. (only for illustration, all three files needs to
> checked).
>
> git grep "\"c\.h" -- *.h :^src/include/postgres*.h;
>
> --
> Daniel Gustafsson https://vmware.com/
>
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-03 04:34 Julien Rouhaud <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Julien Rouhaud @ 2022-10-03 04:34 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Andres Freund <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi,
On Mon, Oct 03, 2022 at 06:00:12AM +0200, Pavel Stehule wrote:
> ne 2. 10. 2022 v 22:52 odesílatel Daniel Gustafsson <[email protected]>
> napsal:
>
> > > On 2 Oct 2022, at 18:04, Andres Freund <[email protected]> wrote:
> > > On 2022-10-02 00:19:59 -0700, Andres Freund wrote:
> > >> On 2022-10-01 23:56:59 -0700, Andres Freund wrote:
> > >>> On 2022-09-12 09:58:37 +0200, Daniel Gustafsson wrote:
> > >>>> Correct, fixed in the attached.
> > >>>
> > >>> Updated patch adding meson compatibility attached.
> > >>
> > >> Err, forgot to amend one hunk :(
> > >
> > > That fixed it on all platforms but windows, due to copy-pasto. I really
> > should
> > > have stopped earlier yesterday...
> >
> > Thanks for updating the patch!
> >
> > The parser in the original submission was -1'd by me, and the current
> > version
> > proposed as an alternative. This was subsequently -1'd as well but no
> > updated
> > patch with a rewritten parser has been posted. So this is now stalled
> > again.
> >
>
> You started rewriting it, but you didn't finish it.
>
> Unfortunately, there is not a clean opinion on using bison's parser for
> this purpose. I understand that the complexity of this language is too low,
> so the benefit of using bison's gramatic is low too. Personally, I have not
> any problem using bison for this purpose. For this case, I think we compare
> two similarly long ways, but unfortunately, customers that have a problem
> with long command lines still have this problem.
>
> Can we go forward? Daniel is strongly against handwritten parser. Is there
> somebody strongly against bison's based parser? There is not any other way.
I don't have a strong opinion either, but it seems that 2 people argued against
a bison parser (vs only 1 arguing for) and the fact that the current habit is
to rely on hand written parsers for simple cases (e.g. jsonapi.c /
pg_parse_json()), it seems that we should go back to Pavel's original parser.
I only had a quick look but it indeed seems trivial, it just maybe need a bit
of refactoring to avoid some code duplication (getFiltersFromFile is
duplicated, and getDatabaseExcludeFiltersFromFile could be removed if
getFiltersFromFile knew about the 2 patterns).
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-07 05:26 Pavel Stehule <[email protected]>
parent: Julien Rouhaud <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2022-10-07 05:26 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Andres Freund <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
I am sending version with handy written parser and meson support
po 3. 10. 2022 v 6:34 odesílatel Julien Rouhaud <[email protected]> napsal:
> Hi,
>
> > You started rewriting it, but you didn't finish it.
> >
> > Unfortunately, there is not a clean opinion on using bison's parser for
> > this purpose. I understand that the complexity of this language is too
> low,
> > so the benefit of using bison's gramatic is low too. Personally, I have
> not
> > any problem using bison for this purpose. For this case, I think we
> compare
> > two similarly long ways, but unfortunately, customers that have a problem
> > with long command lines still have this problem.
> >
> > Can we go forward? Daniel is strongly against handwritten parser. Is
> there
> > somebody strongly against bison's based parser? There is not any other
> way.
>
> I don't have a strong opinion either, but it seems that 2 people argued
> against
> a bison parser (vs only 1 arguing for) and the fact that the current habit
> is
> to rely on hand written parsers for simple cases (e.g. jsonapi.c /
> pg_parse_json()), it seems that we should go back to Pavel's original
> parser.
>
> I only had a quick look but it indeed seems trivial, it just maybe need a
> bit
> of refactoring to avoid some code duplication (getFiltersFromFile is
> duplicated, and getDatabaseExcludeFiltersFromFile could be removed if
> getFiltersFromFile knew about the 2 patterns).
>
I checked this code again, and I don't think some refactoring is easy.
getFiltersFromFile is not duplicated. It is just probably badly named.
These routines are used from pg_dump, pg_dumpall and pg_restore. There are
significant differences in supported objects and in types used for returned
lists (dumpOptions, SimpleStringList, and RestoreOptions). If I have one
routine, then I need to implement some mechanism for specification of
supported objects, and a special type that can be used as a proxy between
caller and parser to hold lists of parsed values. To be names less
confusing I renamed them to read_dump_filters, read_dumpall_filters and
read_restore_filters
Regards
Pavel
Attachments:
[text/x-patch] pg_dump-filter-20221007.patch (48.8K, ../../CAFj8pRDdK9vBcpqt_4UvtqWtWCZYwYoP3yyVW+mmwhzyNGh3-A@mail.gmail.com/3-pg_dump-filter-20221007.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 8b9d9f4cad..955bfcfdad 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index e62d05e5ab..9cad26bbe6 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpretted according to the same rules
+ as <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for excluding databases,
+ and can also be specified more than once for multiple filter files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 47bd7dbda0..ffeb564c52 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules as <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 9dc5a784dd..700e1400c8 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -26,6 +26,7 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filter.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -43,8 +44,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/filter.c b/src/bin/pg_dump/filter.c
new file mode 100644
index 0000000000..5a76fd57f1
--- /dev/null
+++ b/src/bin/pg_dump/filter.c
@@ -0,0 +1,431 @@
+/*-------------------------------------------------------------------------
+ *
+ * Implementation of simple filter file parser
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include "filter.h"
+
+#include "common/logging.h"
+#include "common/fe_memutils.h"
+#include "common/string.h"
+#include "lib/stringinfo.h"
+#include "pqexpbuffer.h"
+
+#define is_keyword_str(cstr, str, bytes) \
+ ((strlen(cstr) == bytes) && (pg_strncasecmp(cstr, str, bytes) == 0))
+
+/*
+ * Following routines are called from pg_dump, pg_dumpall and pg_restore.
+ * Unfortunatelly, implementation of exit_nicely in pg_dump and pg_restore
+ * is different from implementation of this rutine in pg_dumpall. So instead
+ * direct calling exit_nicely we have to return some error flag (in this
+ * case NULL), and exit_nicelly will be executed from caller's routine.
+ */
+
+/*
+ * Simple routines - just don't repeat same code
+ *
+ * Returns true, when filter's file is opened
+ */
+bool
+filter_init(FilterStateData *fstate, const char *filename)
+{
+ fstate->filename = filename;
+ fstate->lineno = 0;
+ initStringInfo(&fstate->linebuff);
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate->fp = fopen(filename, "r");
+ if (!fstate->fp)
+ {
+ pg_log_error("could not open filter file \"%s\": %m", filename);
+ return false;
+ }
+ }
+ else
+ fstate->fp = stdin;
+
+ fstate->is_error = false;
+
+ return true;
+}
+
+/*
+ * Release allocated sources for filter
+ */
+void
+filter_free_sources(FilterStateData *fstate)
+{
+ free(fstate->linebuff.data);
+ fstate->linebuff.data = NULL;
+
+ if (fstate->fp && fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ pg_log_error("could not close filter file \"%s\": %m", fstate->filename);
+
+ fstate->fp = NULL;
+ }
+}
+
+/*
+ * log_format_error - Emit error message
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ fstate->is_error = true;
+}
+
+static const char *
+filter_object_type_name(FilterObjectType fot)
+{
+ switch (fot)
+ {
+ case FILTER_OBJECT_TYPE_NONE:
+ return "comment or empty line";
+ case FILTER_OBJECT_TYPE_DATA:
+ return "data";
+ case FILTER_OBJECT_TYPE_DATABASE:
+ return "database";
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ return "foreign data";
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ return "function";
+ case FILTER_OBJECT_TYPE_INDEX:
+ return "index";
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ return "schema";
+ case FILTER_OBJECT_TYPE_TABLE:
+ return "table";
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ return "trigger";
+ }
+
+ return "unknown object type";
+}
+
+/*
+ * Helper routine to reduce duplicated code
+ */
+void
+log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname,
+ FilterObjectType fot)
+{
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str,
+ "The application \"%s\" doesn't support filter for object type \"%s\".",
+ appname,
+ filter_object_type_name(fot));
+
+ log_invalid_filter_format(fstate, str->data);
+}
+
+/*
+ * filter_get_keyword - read the next filter keyword from buffer
+ *
+ * Search for keywords (limited to ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL, when the buffer is empty or first
+ * char is not alpha. The length of the found keyword is returned in the size
+ * parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* Set returnlength preemptively in case no keyword is found */
+ *size = 0;
+
+ /* Skip initial whitespace */
+ while (isspace(*ptr))
+ ptr++;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr++;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * filter_get_pattern - Read an object identifier pattern from the buffer
+ *
+ * Parses an object identifier pattern from the passed in buffer and sets
+ * objname to a string with object identifier pattern. Returns pointer to the
+ * first character after the pattern.
+ */
+static char *
+filter_get_pattern(FilterStateData *fstate,
+ char *str,
+ char **objname)
+{
+ /* Skip whitespace */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ {
+ log_invalid_filter_format(fstate, "missing object name pattern");
+ return NULL;
+ }
+
+ /*
+ * If the object name pattern has been quoted, we must take care to parse
+ * out the entire quoted pattern, which may contain whitespace and can span
+ * many lines.
+ */
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ Assert(fstate->linebuff.data);
+
+ if (!pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m",
+ fstate->filename);
+ fstate->is_error = true;
+ }
+ else
+ log_invalid_filter_format(fstate, "unexpected end of file");
+
+ return NULL;
+ }
+
+ str = fstate->linebuff.data;
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ fstate->lineno++;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = pg_strdup(quoted_name->data);
+ destroyPQExpBuffer(quoted_name);
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* Simple variant, read to EOL or to first whitespace */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ return str;
+}
+
+/*
+ * read_filter_item - Read command/type/pattern triplet from filter file
+ *
+ * This will parse one filter item from the filter file, and while it is a
+ * row based format a pattern may span more than one line due to how object
+ * names can be constructed. The expected format of the filter file is:
+ *
+ * <command> <object_type> <pattern>
+ *
+ * Where command is "include" or "exclude", and object_type is one of:
+ * "table", "schema", "foreign_data", "data", "database", "function",
+ * "trigger" or "index". The pattern is either simple without any
+ * whitespace, or properly quoted in case there may be whitespace in the
+ * object name. The pattern handling follows the same rules as other object
+ * include and exclude functions; it can use wildcards. Returns true, when
+ * one filter item was successfully read and parsed. When object name
+ * contains \n chars, then more than one line from input file can be
+ * processed. Returns false when the filter file reaches EOF. In case of
+ * errors, the function wont return but will exit with an appropriate error
+ * message.
+ */
+bool
+filter_read_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ Assert(!fstate->is_error);
+
+ if (pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ char *str = fstate->linebuff.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno++;
+
+ (void) pg_strip_crlf(str);
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Skip empty lines or lines where the first non-whitespace character
+ * is a hash indicating a comment.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ /*
+ * First we expect sequence of two keywords, {include|exclude}
+ * followed by the object type to operate on.
+ */
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate,
+ "no filter command found (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ if (is_keyword_str("include", keyword, size))
+ *is_include = true;
+ else if (is_keyword_str("exclude", keyword, size))
+ *is_include = false;
+ else
+ {
+ log_invalid_filter_format(fstate,
+ "invalid filter command (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate, "missing filter object type");
+ return false;
+ }
+
+ if (is_keyword_str("data", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else if (is_keyword_str("database", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_DATABASE;
+ else if (is_keyword_str("foreign_data",keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (is_keyword_str("function", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FUNCTION;
+ else if (is_keyword_str("index", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_INDEX;
+ else if (is_keyword_str("schema", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (is_keyword_str("table", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (is_keyword_str("trigger", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TRIGGER;
+ else
+ {
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str, "unsupported filter object type: \"%.*s\"", size, keyword);
+ log_invalid_filter_format(fstate, str->data);
+ return false;
+ }
+
+ str = filter_get_pattern(fstate, str, objname);
+ if (!str)
+ return false;
+
+ /*
+ * Look for any content after the object identifier. Comments and
+ * whitespace are allowed, other content may indicate that the
+ * user needed to quote the object name so exit with an invalid
+ * format error.
+ */
+ while (isspace(*str))
+ str++;
+
+ if (*str != '\0' && *str != '#')
+ {
+ log_invalid_filter_format(fstate,
+ "unexpected extra data after pattern");
+ return false;
+ }
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m", fstate->filename);
+ fstate->is_error = true;
+ }
+
+ return false;
+}
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..f2503286aa
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,55 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+#include "c.h"
+
+#include "lib/stringinfo.h"
+
+/*
+ * State data for reading filter items from stream
+ */
+typedef struct
+{
+ FILE *fp;
+ const char *filename;
+ int lineno;
+ StringInfoData linebuff;
+ bool is_error;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern bool filter_init(FilterStateData *fstate, const char *filename);
+extern void filter_free_sources(FilterStateData *fstate);
+
+extern void log_invalid_filter_format(FilterStateData *fstate, char *message);
+extern void log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname, FilterObjectType fot);
+extern bool filter_read_item(FilterStateData *fstate, bool *is_include, char **objname, FilterObjectType *objtype);
+
+#endif
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index e66f632b54..c099d66450 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -1,6 +1,7 @@
pg_dump_common_sources = files(
'compress_io.c',
'dumputils.c',
+ 'filter.c',
'parallel.c',
'pg_backup_archiver.c',
'pg_backup_custom.c',
@@ -86,6 +87,7 @@ tests += {
't/001_basic.pl',
't/002_pg_dump.pl',
't/003_pg_dump_with_server.pl',
+ 't/004_pg_dump_filterfile.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index bd9b066e4e..a5d958c3dc 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -59,6 +59,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -318,6 +319,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AH);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void read_dump_filters(const char *filename, DumpOptions *dopt);
int
@@ -390,6 +392,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -623,6 +626,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ read_dump_filters(optarg, &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1028,6 +1035,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18198,3 +18207,89 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+/*
+ * read_dump_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_dump_filters(const char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments and empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ {
+ log_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ break;
+ }
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dump", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free_sources(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 083012ca39..f393bb5eaf 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -26,6 +26,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -81,6 +82,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void read_dumpall_filters(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -158,6 +160,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -360,6 +363,10 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ read_dumpall_filters(optarg, &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -653,6 +660,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1890,7 +1898,6 @@ executeCommand(PGconn *conn, const char *query)
PQclear(res);
}
-
/*
* dumpTimestamp
*/
@@ -1914,3 +1921,55 @@ hash_string_pointer(char *s)
return hash_bytes(ss, strlen(s));
}
+
+/*
+ * read_dumpall_filters - retrieve database identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ *
+ * In this moment only excluded databases can be filtered.
+ */
+static void
+read_dumpall_filters(const char *filename, SimpleStringList *pattern)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATABASE)
+ {
+ if (!is_include)
+ simple_string_list_append(pattern, objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "include database filter is not suppported");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dumpall", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free_sources(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..701ca07a02 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,11 +47,13 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
static void usage(const char *progname);
+static void read_restore_filters(const char *filename, RestoreOptions *dopt);
int
main(int argc, char **argv)
@@ -123,6 +125,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +289,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ read_restore_filters(optarg, opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +470,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +502,109 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * read_restore_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_restore_filters(const char *filename, RestoreOptions *opts)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments or empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_FUNCTION)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_INDEX)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ simple_string_list_append(&opts->schemaNames, objname);
+ else
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TRIGGER)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, optarg);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_restore", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free_sources(&fstate);
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..dd533d79eb
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,491 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 62;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;;
+my $inputfile;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres',
+ 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three_one(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres',
+ "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three_one VALUES('*** TABLE THREE_ONE ***')");
+
+#
+# Test interaction of correctly specified filter file
+#
+my ($cmd, $stdout, $stderr, $result);
+
+# Empty filterfile
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "\n # a comment and nothing more\n\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_(one|two|three|three_one)/m,
+ "tables dumped");
+
+# Test various combinations of whitespace, comments and correct filters
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "\t\n";
+print $inputfile " \t# another comment\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter patterns as well as comments and whitespace");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public\.table_three/m, "table three not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_three_one/m,
+ "table three_one not dumped");
+ok( $dump !~ qr/^COPY public\.table_one/m,
+ "content of table one is not included");
+ok($dump =~ qr/^COPY public\.table_two/m, "content of table two is included");
+
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with exclusion of a single table");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping tables with a wildcard pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_thre*\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with wildcard in pattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_two/m, "table two not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping table with multiline quoted tablename
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with multiline names requiring quoting");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding multiline quoted tablename from dump
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding an entire schema
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "exclude the public schema");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test including and excluding an entire schema by multiple filterfiles
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema public\n";
+close $inputfile;
+
+open my $alt_inputfile, '>', "$tempdir/inputfile2.txt"
+ or die "unable to open filterfile for writing";
+print $alt_inputfile "exclude schema public\n";
+close $alt_inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "--filter=$tempdir/inputfile2.txt", 'postgres'
+ ],
+ "exclude the public schema with multiple filters");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test dumping a table with a single leading newline on a row
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump nonexisting foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummyserver\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump foreign_data with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE SERVER dummyserver/m, "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude foreign_data dummy*\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/exclude filter is not allowed/,
+ "erroneously exclude foreign server");
+
+#########################################
+# Test broken input format
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: invalid object type specified, should be table, schema, foreign_data or data"
+);
+
+# Test missing object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/missing object name/,
+ "invalid syntax: missing object identifier pattern");
+
+# Test adding extra content after the object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table one";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unexpected extra data/,
+ "invalid syntax: extra content after object identifier pattern");
+
+#########################################
+# Combined with --strict-names
+
+# First ensure that a matching filter works
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ "strict names with matching mattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "no table dumped");
+
+# Now append a pattern to the filter file which doesn't resolve
+open $inputfile, '>>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_nonexisting_name";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "inclusion of non-existing objects with --strict names");
+
+#########################################
+# pg_dumpall tests
+
+###########################
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude database postgres\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ "dump tables with exclusion of a database");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^\\connect postgres/m, "database postgres is not dumped");
+ok($dump =~ qr/^\\connect template1/m, "database template1 is dumped");
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: exclusion of non-existing object type"
+);
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table foo";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/The application "pg_dumpall" doesn't support filter for object type "table"/,
+ "invalid syntax: exclusion of unsupported object type"
+);
+
+#########################################
+# pg_restore tests
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', "$tempdir/filter_test.dump",
+ "-Fc", 'postgres'
+ ],
+ "dump all tables");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_two";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "wanted table restored");
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "unwanted tables are not restored");
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 83a3e40425..d3c7292203 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -451,6 +451,7 @@ sub mkvcbuild
$pgdumpall->AddIncludeDir('src/backend');
$pgdumpall->AddFile('src/bin/pg_dump/pg_dumpall.c');
$pgdumpall->AddFile('src/bin/pg_dump/dumputils.c');
+ $pgdumpall->AddFile('src/bin/pg_dump/filter.c');
$pgdumpall->AddLibrary('ws2_32.lib');
my $pgrestore = AddSimpleFrontend('pg_dump', 1);
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-07 14:03 Julien Rouhaud <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Julien Rouhaud @ 2022-10-07 14:03 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Andres Freund <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Fri, Oct 07, 2022 at 07:26:08AM +0200, Pavel Stehule wrote:
>
> I checked this code again, and I don't think some refactoring is easy.
> getFiltersFromFile is not duplicated. It is just probably badly named.
>
> These routines are used from pg_dump, pg_dumpall and pg_restore. There are
> significant differences in supported objects and in types used for returned
> lists (dumpOptions, SimpleStringList, and RestoreOptions). If I have one
> routine, then I need to implement some mechanism for specification of
> supported objects, and a special type that can be used as a proxy between
> caller and parser to hold lists of parsed values. To be names less
> confusing I renamed them to read_dump_filters, read_dumpall_filters and
> read_restore_filters
Ah right, I missed the different argument types. Now that the functions have
improved names it looks way clearer, and it seems just fine!
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-07 16:48 Pavel Stehule <[email protected]>
parent: Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2022-10-07 16:48 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Andres Freund <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
pá 7. 10. 2022 v 16:03 odesílatel Julien Rouhaud <[email protected]>
napsal:
> On Fri, Oct 07, 2022 at 07:26:08AM +0200, Pavel Stehule wrote:
> >
> > I checked this code again, and I don't think some refactoring is easy.
> > getFiltersFromFile is not duplicated. It is just probably badly named.
> >
> > These routines are used from pg_dump, pg_dumpall and pg_restore. There
> are
> > significant differences in supported objects and in types used for
> returned
> > lists (dumpOptions, SimpleStringList, and RestoreOptions). If I have one
> > routine, then I need to implement some mechanism for specification of
> > supported objects, and a special type that can be used as a proxy between
> > caller and parser to hold lists of parsed values. To be names less
> > confusing I renamed them to read_dump_filters, read_dumpall_filters and
> > read_restore_filters
>
> Ah right, I missed the different argument types. Now that the functions
> have
> improved names it looks way clearer, and it seems just fine!
>
Thank you for check
Pavel
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-13 18:46 Andres Freund <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Andres Freund @ 2022-10-13 18:46 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi,
On 2022-10-07 07:26:08 +0200, Pavel Stehule wrote:
> I am sending version with handy written parser and meson support
Given this is a new approach it seems inaccurate to have the CF entry marked
ready-for-committer. I've updated it to needs-review.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-18 09:33 Julien Rouhaud <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Julien Rouhaud @ 2022-10-18 09:33 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Pavel Stehule <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Thu, Oct 13, 2022 at 11:46:34AM -0700, Andres Freund wrote:
> Hi,
>
> On 2022-10-07 07:26:08 +0200, Pavel Stehule wrote:
> > I am sending version with handy written parser and meson support
>
> Given this is a new approach it seems inaccurate to have the CF entry marked
> ready-for-committer. I've updated it to needs-review.
I just had a quick look at the rest of the patch.
For the parser, it seems that filter_get_pattern is reimplementing an
identifier parsing function but isn't entirely correct. It can correctly parse
quoted non-qualified identifiers and non-quoted qualified identifiers, but not
quoted and qualified ones. For instance:
$ echo 'include table nsp.tbl' | pg_dump --filter - >/dev/null
$echo $?
0
$ echo 'include table "TBL"' | pg_dump --filter - >/dev/null
$echo $?
0
$ echo 'include table "NSP"."TBL"' | pg_dump --filter - >/dev/null
pg_dump: error: invalid format of filter on line 1: unexpected extra data after pattern
This should also be covered in the regression tests.
I'm wondering if psql's parse_identifier() could be exported and reused here
rather than creating yet another version.
Nitpicking: the comments needs some improvements:
+ /*
+ * Simple routines - just don't repeat same code
+ *
+ * Returns true, when filter's file is opened
+ */
+ bool
+ filter_init(FilterStateData *fstate, const char *filename)
also, is there any reason why this function doesn't call exit_nicely in case of
error rather than letting each caller do it without any other cleanup?
+ /*
+ * Release allocated sources for filter
+ */
+ void
+ filter_free_sources(FilterStateData *fstate)
I'm assuming "ressources" not "sources"?
+ /*
+ * log_format_error - Emit error message
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+ void
+ log_invalid_filter_format(FilterStateData *fstate, char *message)
mismatch between comment and function name (same for filter_read_item)
+ static const char *
+ filter_object_type_name(FilterObjectType fot)
No description.
/*
* Helper routine to reduce duplicated code
*/
void
log_unsupported_filter_object_type(FilterStateData *fstate,
const char *appname,
FilterObjectType fot)
Need more helpful comment.
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-10-26 04:26 Pavel Stehule <[email protected]>
parent: Julien Rouhaud <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2022-10-26 04:26 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
út 18. 10. 2022 v 11:33 odesílatel Julien Rouhaud <[email protected]>
napsal:
> On Thu, Oct 13, 2022 at 11:46:34AM -0700, Andres Freund wrote:
> > Hi,
> >
> > On 2022-10-07 07:26:08 +0200, Pavel Stehule wrote:
> > > I am sending version with handy written parser and meson support
> >
> > Given this is a new approach it seems inaccurate to have the CF entry
> marked
> > ready-for-committer. I've updated it to needs-review.
>
> I just had a quick look at the rest of the patch.
>
> For the parser, it seems that filter_get_pattern is reimplementing an
> identifier parsing function but isn't entirely correct. It can correctly
> parse
> quoted non-qualified identifiers and non-quoted qualified identifiers, but
> not
> quoted and qualified ones. For instance:
>
> $ echo 'include table nsp.tbl' | pg_dump --filter - >/dev/null
> $echo $?
> 0
>
> $ echo 'include table "TBL"' | pg_dump --filter - >/dev/null
> $echo $?
> 0
>
> $ echo 'include table "NSP"."TBL"' | pg_dump --filter - >/dev/null
> pg_dump: error: invalid format of filter on line 1: unexpected extra data
> after pattern
>
fixed
>
> This should also be covered in the regression tests.
>
done
>
> I'm wondering if psql's parse_identifier() could be exported and reused
> here
> rather than creating yet another version.
>
I looked there, and I don't think this parser is usable for this purpose.
It is very sensitive on white spaces, and doesn't support multi-lines. It
is designed for support readline tab complete, it is designed for
simplicity not for correctness.
>
> Nitpicking: the comments needs some improvements:
>
> + /*
> + * Simple routines - just don't repeat same code
> + *
> + * Returns true, when filter's file is opened
> + */
> + bool
> + filter_init(FilterStateData *fstate, const char *filename)
>
done
>
> also, is there any reason why this function doesn't call exit_nicely in
> case of
> error rather than letting each caller do it without any other cleanup?
>
It is commented few lines up
/*
* Following routines are called from pg_dump, pg_dumpall and pg_restore.
* Unfortunatelly, implementation of exit_nicely in pg_dump and pg_restore
* is different from implementation of this rutine in pg_dumpall. So instead
* direct calling exit_nicely we have to return some error flag (in this
* case NULL), and exit_nicelly will be executed from caller's routine.
*/
>
> + /*
> + * Release allocated sources for filter
> + */
> + void
> + filter_free_sources(FilterStateData *fstate)
>
> I'm assuming "ressources" not "sources"?
>
changed
>
> + /*
> + * log_format_error - Emit error message
> + *
> + * This is mostly a convenience routine to avoid duplicating file
> closing code
> + * in multiple callsites.
> + */
> + void
> + log_invalid_filter_format(FilterStateData *fstate, char *message)
>
> mismatch between comment and function name (same for filter_read_item)
>
fixes
>
> + static const char *
> + filter_object_type_name(FilterObjectType fot)
>
> No description.
>
>
fixed
> /*
> * Helper routine to reduce duplicated code
> */
> void
> log_unsupported_filter_object_type(FilterStateData *fstate,
>
> const char *appname,
>
> FilterObjectType fot)
>
> Need more helpful comment.
>
fixed
Thank you for comments
attached updated patch
Regards
Pavel
Attachments:
[text/x-patch] pg_dump-filter-20221026.patch (50.6K, ../../CAFj8pRCGX_9YtMDXwaVXA6qKBJVziyHj662jmRuei1CPvy=Sqw@mail.gmail.com/3-pg_dump-filter-20221026.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 8b9d9f4cad..955bfcfdad 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index e62d05e5ab..9cad26bbe6 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpretted according to the same rules
+ as <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for excluding databases,
+ and can also be specified more than once for multiple filter files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 47bd7dbda0..ffeb564c52 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules as <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 9dc5a784dd..700e1400c8 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -26,6 +26,7 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filter.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -43,8 +44,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/filter.c b/src/bin/pg_dump/filter.c
new file mode 100644
index 0000000000..67e1874224
--- /dev/null
+++ b/src/bin/pg_dump/filter.c
@@ -0,0 +1,467 @@
+/*-------------------------------------------------------------------------
+ *
+ * Implementation of simple filter file parser
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include "filter.h"
+
+#include "common/logging.h"
+#include "common/fe_memutils.h"
+#include "common/string.h"
+#include "lib/stringinfo.h"
+#include "pqexpbuffer.h"
+
+#define is_keyword_str(cstr, str, bytes) \
+ ((strlen(cstr) == bytes) && (pg_strncasecmp(cstr, str, bytes) == 0))
+
+/*
+ * Following routines are called from pg_dump, pg_dumpall and pg_restore.
+ * Unfortunatelly, implementation of exit_nicely in pg_dump and pg_restore
+ * is different from implementation of this rutine in pg_dumpall. So instead
+ * direct calling exit_nicely we have to return some error flag (in this
+ * case NULL), and exit_nicelly will be executed from caller's routine.
+ */
+
+/*
+ * Opens filter's file and initialize fstate structure.
+ * Returns true on success.
+ */
+bool
+filter_init(FilterStateData *fstate, const char *filename)
+{
+ fstate->filename = filename;
+ fstate->lineno = 0;
+ initStringInfo(&fstate->linebuff);
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate->fp = fopen(filename, "r");
+ if (!fstate->fp)
+ {
+ pg_log_error("could not open filter file \"%s\": %m", filename);
+ return false;
+ }
+ }
+ else
+ fstate->fp = stdin;
+
+ fstate->is_error = false;
+
+ return true;
+}
+
+/*
+ * Release allocated resources for filter
+ */
+void
+filter_free(FilterStateData *fstate)
+{
+ free(fstate->linebuff.data);
+ fstate->linebuff.data = NULL;
+
+ if (fstate->fp && fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ pg_log_error("could not close filter file \"%s\": %m", fstate->filename);
+
+ fstate->fp = NULL;
+ }
+}
+
+/*
+ * Translate FilterObjectType enum to string. It is designed for formatting
+ * of error message in log_unsupported_filter_object_type routine.
+ */
+static const char *
+filter_object_type_name(FilterObjectType fot)
+{
+ switch (fot)
+ {
+ case FILTER_OBJECT_TYPE_NONE:
+ return "comment or empty line";
+ case FILTER_OBJECT_TYPE_DATA:
+ return "data";
+ case FILTER_OBJECT_TYPE_DATABASE:
+ return "database";
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ return "foreign data";
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ return "function";
+ case FILTER_OBJECT_TYPE_INDEX:
+ return "index";
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ return "schema";
+ case FILTER_OBJECT_TYPE_TABLE:
+ return "table";
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ return "trigger";
+ }
+
+ return "unknown object type";
+}
+
+/*
+ * Emit error message "invalid format of filter file ..."
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format of filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+ }
+ else
+ pg_log_error("invalid format of filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ fstate->is_error = true;
+}
+
+/*
+ * Emit error message "The application doesn't support filter for object type ..."
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname,
+ FilterObjectType fot)
+{
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str,
+ "The application \"%s\" doesn't support filter for object type \"%s\".",
+ appname,
+ filter_object_type_name(fot));
+
+ log_invalid_filter_format(fstate, str->data);
+}
+
+/*
+ * filter_get_keyword - read the next filter keyword from buffer
+ *
+ * Search for keywords (limited to ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL, when the buffer is empty or first
+ * char is not alpha. The length of the found keyword is returned in the size
+ * parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* Set returnlength preemptively in case no keyword is found */
+ *size = 0;
+
+ /* Skip initial whitespace */
+ while (isspace(*ptr))
+ ptr++;
+
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr++;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * filter_get_pattern - Read an object identifier pattern from the buffer
+ *
+ * Parses an object identifier pattern from the passed in buffer and sets
+ * objname to a string with object identifier pattern. Returns pointer to the
+ * first character after the pattern. Returns NULL on error.
+ */
+static char *
+filter_get_pattern(FilterStateData *fstate,
+ char *str,
+ char **objname)
+{
+ /* Skip whitespace */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ {
+ log_invalid_filter_format(fstate, "missing object name pattern");
+ return NULL;
+ }
+
+ /*
+ * If the object name pattern has been quoted, we must take care to parse
+ * out the entire quoted pattern, which may contain whitespace and can span
+ * many lines.
+ */
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ Assert(fstate->linebuff.data);
+
+ if (!pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m",
+ fstate->filename);
+ fstate->is_error = true;
+ }
+ else
+ log_invalid_filter_format(fstate, "unexpected end of file");
+
+ return NULL;
+ }
+
+ str = fstate->linebuff.data;
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ fstate->lineno++;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = pg_strdup(quoted_name->data);
+ destroyPQExpBuffer(quoted_name);
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* Simple variant, read to EOL or to first whitespace */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ return str;
+}
+
+/*
+ * filter_read_item - Read command/type/pattern triplet from filter file
+ *
+ * This will parse one filter item from the filter file, and while it is a
+ * row based format a pattern may span more than one line due to how object
+ * names can be constructed. The expected format of the filter file is:
+ *
+ * <command> <object_type> <pattern>
+ *
+ * Where command is "include" or "exclude", and object_type is one of:
+ * "table", "schema", "foreign_data", "data", "database", "function",
+ * "trigger" or "index". The pattern is either simple without any
+ * whitespace, or properly quoted in case there may be whitespace in the
+ * object name. The pattern handling follows the same rules as other object
+ * include and exclude functions; it can use wildcards. Returns true, when
+ * one filter item was successfully read and parsed. When object name
+ * contains \n chars, then more than one line from input file can be
+ * processed. Returns false when the filter file reaches EOF. In case of
+ * errors, the function wont return but will exit with an appropriate error
+ * message.
+ */
+bool
+filter_read_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ Assert(!fstate->is_error);
+
+ if (pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ char *str = fstate->linebuff.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno++;
+
+ (void) pg_strip_crlf(str);
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Skip empty lines or lines where the first non-whitespace character
+ * is a hash indicating a comment.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ /*
+ * First we expect sequence of two keywords, {include|exclude}
+ * followed by the object type to operate on.
+ */
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate,
+ "no filter command found (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ if (is_keyword_str("include", keyword, size))
+ *is_include = true;
+ else if (is_keyword_str("exclude", keyword, size))
+ *is_include = false;
+ else
+ {
+ log_invalid_filter_format(fstate,
+ "invalid filter command (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate, "missing filter object type");
+ return false;
+ }
+
+ if (is_keyword_str("data", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_DATA;
+ else if (is_keyword_str("database", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_DATABASE;
+ else if (is_keyword_str("foreign_data",keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (is_keyword_str("function", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FUNCTION;
+ else if (is_keyword_str("index", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_INDEX;
+ else if (is_keyword_str("schema", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (is_keyword_str("table", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (is_keyword_str("trigger", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TRIGGER;
+ else
+ {
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str, "unsupported filter object type: \"%.*s\"", size, keyword);
+ log_invalid_filter_format(fstate, str->data);
+ return false;
+ }
+
+ str = filter_get_pattern(fstate, str, objname);
+ if (!str)
+ return false;
+
+ /* skip whitespaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * qualified identifier can be valid pattern. So repeat
+ * reading if char after pattern is dot.
+ */
+ if (*str == '.')
+ {
+ PQExpBuffer qual_name = createPQExpBuffer();
+
+ appendPQExpBufferStr(qual_name, *objname);
+ free(*objname);
+
+ do
+ {
+ str = filter_get_pattern(fstate, ++str, objname);
+ if (!str)
+ return false;
+
+ appendPQExpBufferChar(qual_name, '.');
+ appendPQExpBufferStr(qual_name, *objname);
+ free(*objname);
+
+ while (isspace(*str))
+ str++;
+ }
+ while (*str == '.');
+
+ *objname = qual_name->data;
+ }
+
+ /*
+ * Look for any content after the object identifier. Comments and
+ * whitespace are allowed, other content may indicate that the
+ * user needed to quote the object name so exit with an invalid
+ * format error.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ log_invalid_filter_format(fstate,
+ "unexpected extra data after pattern");
+ return false;
+ }
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m", fstate->filename);
+ fstate->is_error = true;
+ }
+
+ return false;
+}
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..0e17a71665
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,55 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+#include "c.h"
+
+#include "lib/stringinfo.h"
+
+/*
+ * State data for reading filter items from stream
+ */
+typedef struct
+{
+ FILE *fp;
+ const char *filename;
+ int lineno;
+ StringInfoData linebuff;
+ bool is_error;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern bool filter_init(FilterStateData *fstate, const char *filename);
+extern void filter_free(FilterStateData *fstate);
+
+extern void log_invalid_filter_format(FilterStateData *fstate, char *message);
+extern void log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname, FilterObjectType fot);
+extern bool filter_read_item(FilterStateData *fstate, bool *is_include, char **objname, FilterObjectType *objtype);
+
+#endif
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index e66f632b54..c099d66450 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -1,6 +1,7 @@
pg_dump_common_sources = files(
'compress_io.c',
'dumputils.c',
+ 'filter.c',
'parallel.c',
'pg_backup_archiver.c',
'pg_backup_custom.c',
@@ -86,6 +87,7 @@ tests += {
't/001_basic.pl',
't/002_pg_dump.pl',
't/003_pg_dump_with_server.pl',
+ 't/004_pg_dump_filterfile.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index bd9b066e4e..276be070ab 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -59,6 +59,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -318,6 +319,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AH);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void read_dump_filters(const char *filename, DumpOptions *dopt);
int
@@ -390,6 +392,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -623,6 +626,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ read_dump_filters(optarg, &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1028,6 +1035,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18198,3 +18207,89 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+/*
+ * read_dump_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_dump_filters(const char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments and empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATA)
+ {
+ if (is_include)
+ {
+ log_invalid_filter_format(&fstate,
+ "include filter is not allowed for this type of object");
+ break;
+ }
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dump", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 083012ca39..e060bcc2fe 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -26,6 +26,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -81,6 +82,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void read_dumpall_filters(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -158,6 +160,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -360,6 +363,10 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ read_dumpall_filters(optarg, &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -653,6 +660,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1890,7 +1898,6 @@ executeCommand(PGconn *conn, const char *query)
PQclear(res);
}
-
/*
* dumpTimestamp
*/
@@ -1914,3 +1921,55 @@ hash_string_pointer(char *s)
return hash_bytes(ss, strlen(s));
}
+
+/*
+ * read_dumpall_filters - retrieve database identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ *
+ * In this moment only excluded databases can be filtered.
+ */
+static void
+read_dumpall_filters(const char *filename, SimpleStringList *pattern)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATABASE)
+ {
+ if (!is_include)
+ simple_string_list_append(pattern, objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "include database filter is not suppported");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dumpall", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..96ffc2455f 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,11 +47,13 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
static void usage(const char *progname);
+static void read_restore_filters(const char *filename, RestoreOptions *dopt);
int
main(int argc, char **argv)
@@ -123,6 +125,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +289,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ read_restore_filters(optarg, opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +470,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +502,109 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * read_restore_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_restore_filters(const char *filename, RestoreOptions *opts)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments or empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_FUNCTION)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_INDEX)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ simple_string_list_append(&opts->schemaNames, objname);
+ else
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TRIGGER)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, optarg);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "exclude filter is not allowed for this type of object");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_restore", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..e89ebaf1b9
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,514 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 69;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;;
+my $inputfile;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres',
+ 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three_one(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres',
+ "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three_one VALUES('*** TABLE THREE_ONE ***')");
+
+#
+# Test interaction of correctly specified filter file
+#
+my ($cmd, $stdout, $stderr, $result);
+
+# Empty filterfile
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "\n # a comment and nothing more\n\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "table one dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "table two dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "table three dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m, "table three one dumped");
+
+# Test various combinations of whitespace, comments and correct filters
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "\t\n";
+print $inputfile " \t# another comment\n";
+print $inputfile "exclude data table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter patterns as well as comments and whitespace");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public\.table_three/m, "table three not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_three_one/m,
+ "table three_one not dumped");
+ok( $dump !~ qr/^COPY public\.table_one/m,
+ "content of table one is not included");
+ok($dump =~ qr/^COPY public\.table_two/m, "content of table two is included");
+
+# Test dumping tables specified by qualified names
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table public.table_one\n";
+print $inputfile "include table \"public\".\"table_two\"\n";
+print $inputfile "include table \"public\". table_three\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with exclusion of a single table");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping tables with a wildcard pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_thre*\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with wildcard in pattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_two/m, "table two not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping table with multiline quoted tablename
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with multiline names requiring quoting");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding multiline quoted tablename from dump
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding an entire schema
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "exclude the public schema");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test including and excluding an entire schema by multiple filterfiles
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema public\n";
+close $inputfile;
+
+open my $alt_inputfile, '>', "$tempdir/inputfile2.txt"
+ or die "unable to open filterfile for writing";
+print $alt_inputfile "exclude schema public\n";
+close $alt_inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "--filter=$tempdir/inputfile2.txt", 'postgres'
+ ],
+ "exclude the public schema with multiple filters");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test dumping a table with a single leading newline on a row
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump nonexisting foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummyserver\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump foreign_data with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE SERVER dummyserver/m, "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude foreign_data dummy*\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/exclude filter is not allowed/,
+ "erroneously exclude foreign server");
+
+#########################################
+# Test broken input format
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: invalid object type specified, should be table, schema, foreign_data or data"
+);
+
+# Test missing object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/missing object name/,
+ "invalid syntax: missing object identifier pattern");
+
+# Test adding extra content after the object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table one";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unexpected extra data/,
+ "invalid syntax: extra content after object identifier pattern");
+
+#########################################
+# Combined with --strict-names
+
+# First ensure that a matching filter works
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ "strict names with matching mattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "no table dumped");
+
+# Now append a pattern to the filter file which doesn't resolve
+open $inputfile, '>>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_nonexisting_name";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "inclusion of non-existing objects with --strict names");
+
+#########################################
+# pg_dumpall tests
+
+###########################
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude database postgres\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ "dump tables with exclusion of a database");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^\\connect postgres/m, "database postgres is not dumped");
+ok($dump =~ qr/^\\connect template1/m, "database template1 is dumped");
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: exclusion of non-existing object type"
+);
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table foo";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/The application "pg_dumpall" doesn't support filter for object type "table"/,
+ "invalid syntax: exclusion of unsupported object type"
+);
+
+#########################################
+# pg_restore tests
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', "$tempdir/filter_test.dump",
+ "-Fc", 'postgres'
+ ],
+ "dump all tables");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_two";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "wanted table restored");
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "unwanted tables are not restored");
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 83a3e40425..d3c7292203 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -451,6 +451,7 @@ sub mkvcbuild
$pgdumpall->AddIncludeDir('src/backend');
$pgdumpall->AddFile('src/bin/pg_dump/pg_dumpall.c');
$pgdumpall->AddFile('src/bin/pg_dump/dumputils.c');
+ $pgdumpall->AddFile('src/bin/pg_dump/filter.c');
$pgdumpall->AddLibrary('ws2_32.lib');
my $pgrestore = AddSimpleFrontend('pg_dump', 1);
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-03 04:09 Julien Rouhaud <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Julien Rouhaud @ 2022-11-03 04:09 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi,
On Wed, Oct 26, 2022 at 06:26:26AM +0200, Pavel Stehule wrote:
>
> út 18. 10. 2022 v 11:33 odesílatel Julien Rouhaud <[email protected]>
> napsal:
>
> >
> > I'm wondering if psql's parse_identifier() could be exported and reused
> > here
> > rather than creating yet another version.
> >
>
> I looked there, and I don't think this parser is usable for this purpose.
> It is very sensitive on white spaces, and doesn't support multi-lines. It
> is designed for support readline tab complete, it is designed for
> simplicity not for correctness.
Ah, sorry I should have checked more thoroughly. I guess it's ok to have
another identifier parser for the include file then, as this new one wouldn't
really fit the tab-completion use case.
> > also, is there any reason why this function doesn't call exit_nicely in
> > case of
> > error rather than letting each caller do it without any other cleanup?
> >
>
> It is commented few lines up
>
> /*
> * Following routines are called from pg_dump, pg_dumpall and pg_restore.
> * Unfortunatelly, implementation of exit_nicely in pg_dump and pg_restore
> * is different from implementation of this rutine in pg_dumpall. So instead
> * direct calling exit_nicely we have to return some error flag (in this
> * case NULL), and exit_nicelly will be executed from caller's routine.
> */
Oh right, I totally missed it sorry about that!
About the new version, I didn't find any problem with the feature itself so
it's a good thing!
I still have a few comments about the patch. First, about the behavior:
- is that ok to have just "data" pattern instead of "table_data" or something
like that, since it's supposed to match --exclude-table-data option?
- the error message are sometimes not super helpful. For instance:
$ echo "include data t1" | pg_dump --filter -
pg_dump: error: invalid format of filter on line 1: include filter is not allowed for this type of object
It would be nice if the error message mentioned "data" rather than a generic
"this type of object". Also, maybe we should quote "include" to outline that
we found this keyword?
About the patch itself:
filter.c:
+#include "postgres_fe.h"
+
+#include "filter.h"
+
+#include "common/logging.h"
the filter.h inclusion should be done with the rest of the includes, in
alphabetical order.
+#define is_keyword_str(cstr, str, bytes) \
+ ((strlen(cstr) == bytes) && (pg_strncasecmp(cstr, str, bytes) == 0))
nit: our guidline is to protect macro arguments with parenthesis. Some
arguments can be evaluated multiple times but I don't think it's worth adding a
comment for that.
+ * Unfortunatelly, implementation of exit_nicely in pg_dump and pg_restore
+ * is different from implementation of this rutine in pg_dumpall. So instead
+ * direct calling exit_nicely we have to return some error flag (in this
typos: s/Unfortunatelly/Unfortunately/ and s/rutine/routine/
Also, it would probably be better to say "instead of directly calling..."
+static const char *
+filter_object_type_name(FilterObjectType fot)
+{
+ switch (fot)
+ {
+ case FILTER_OBJECT_TYPE_NONE:
+ return "comment or empty line";
+[...]
+ }
+
+ return "unknown object type";
+}
I'm wondering if we should add a pg_unreachable() there, some compilers might
complain otherwise. See CreateDestReceiver() for instance for similar pattern.
+ * Emit error message "invalid format of filter file ..."
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_invalid_filter_format(FilterStateData *fstate, char *message)
nit: invalid format *in* filter file...?
+void
+log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname,
+ FilterObjectType fot)
+{
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str,
+ "The application \"%s\" doesn't support filter for object type \"%s\".",
nit: there shouldn't be uppercase in error messages, especially since this will
be appended to another message by log_invalid_filter_format. I would just just
drop "The application" entirely for brevity.
+/*
+ * Release allocated resources for filter
+ */
+void
+filter_free(FilterStateData *fstate)
nit: Release allocated resources for *the given* filter?
+ * Search for keywords (limited to ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL, when the buffer is empty or first
+ * char is not alpha. The length of the found keyword is returned in the size
+ * parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ [...]
+ if (isascii(*ptr) && isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
+ ptr++;
Is there any reason to test isascii()? isalpha() should already cover that and
should be cheaper to test anyway.
Also nit: "Returns NULL when the buffer..." (unnecessary comma), and the '_'
char is also allowed.
+filter_read_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ Assert(!fstate->is_error);
+
+ if (pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ char *str = fstate->linebuff.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno++;
+
+ (void) pg_strip_crlf(str);
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+[...]
+ keyword = filter_get_keyword((const char **) &str, &size);
Is there any interest with the initial pg_strip_crlf? AFAICT all the rest of
the code will ignore such caracters using isspace() so it wouldn't change
anything.
Dropping both pg_strip_crlf() would allow you to declare str as const rather
than doing it in function calls. It would require to add const qualifiers in a
few other places, but it seems like an improvement, as for instance right now
filter_get_pattern is free to rewrite the str (because it's also calling
pg_strip_crlf, but there's no guarantee that it doesn't do anything else).
+/*
+ * filter_get_pattern - Read an object identifier pattern from the buffer
+ *
+ * Parses an object identifier pattern from the passed in buffer and sets
+ * objname to a string with object identifier pattern. Returns pointer to the
+ * first character after the pattern. Returns NULL on error.
+ */
+static char *
+filter_get_pattern(FilterStateData *fstate,
nit: suggestion to reword the comment, maybe something like
/*
* filter_get_pattern - Identify an object identifier pattern
*
* Try to parse an object identifier pattern from the passed buffer. If one is
* found, it sets objname to a string with the object identifier pattern and
* returns a pointer to the first byte after the found pattern. Otherwise NULL
* is returned.
*/
+bool
+filter_read_item(FilterStateData *fstate,
Another suggestion for comment rewrite:
/*-------------------
* filter_read_item - Read command/type/pattern triplet from a filter file
*
* This will parse one filter item from the filter file, and while it is a
* row based format a pattern may span more than one line due to how object
* names can be constructed. The expected format of the filter file is:
*
* <command> <object_type> <pattern>
*
* command can be "include" or "exclude"
* object_type can one of: "table", "schema", "foreign_data", "data",
* "database", "function", "trigger" or "index"
* pattern can be any possibly-quoted and possibly-qualified identifier. It
* follows the same rules as other object include and exclude functions so it
* can also use wildcards.
*
* Returns true when one filter item was successfully read and parsed. When
* object name contains \n chars, then more than one line from input file can
* be processed. Returns false when the filter file reaches EOF. In case of
* error, the function will emit an appropriate error message before returning
* false.
*/
Note also that your original comment said:
+ * In case of
+ * errors, the function wont return but will exit with an appropriate error
+ * message.
But AFAICS that's not the case: it will indeed log an appropriate error message
but will return false. I'm assuming that the comment was outdated as the
calling code handles it just fine, so I just modified the comment.
filter.h:
+#ifndef FILTER_H
+#define FILTER_H
+#include "c.h"
It's definitely not ok to include .ch in frontend code. But AFAICS just
removing it doesn't cause any problem. Note also that there should be an empty
line after the #define FILTER_H per usual coding style.
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-03 21:22 Pavel Stehule <[email protected]>
parent: Julien Rouhaud <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2022-11-03 21:22 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
čt 3. 11. 2022 v 5:09 odesílatel Julien Rouhaud <[email protected]> napsal:
> Hi,
>
> On Wed, Oct 26, 2022 at 06:26:26AM +0200, Pavel Stehule wrote:
> >
> > út 18. 10. 2022 v 11:33 odesílatel Julien Rouhaud <[email protected]>
> > napsal:
> >
> > >
> > > I'm wondering if psql's parse_identifier() could be exported and reused
> > > here
> > > rather than creating yet another version.
> > >
> >
> > I looked there, and I don't think this parser is usable for this purpose.
> > It is very sensitive on white spaces, and doesn't support multi-lines.
> It
> > is designed for support readline tab complete, it is designed for
> > simplicity not for correctness.
>
> Ah, sorry I should have checked more thoroughly. I guess it's ok to have
> another identifier parser for the include file then, as this new one
> wouldn't
> really fit the tab-completion use case.
>
> > > also, is there any reason why this function doesn't call exit_nicely in
> > > case of
> > > error rather than letting each caller do it without any other cleanup?
> > >
> >
> > It is commented few lines up
> >
> > /*
> > * Following routines are called from pg_dump, pg_dumpall and pg_restore.
> > * Unfortunatelly, implementation of exit_nicely in pg_dump and
> pg_restore
> > * is different from implementation of this rutine in pg_dumpall. So
> instead
> > * direct calling exit_nicely we have to return some error flag (in this
> > * case NULL), and exit_nicelly will be executed from caller's routine.
> > */
>
> Oh right, I totally missed it sorry about that!
>
> About the new version, I didn't find any problem with the feature itself so
> it's a good thing!
>
> I still have a few comments about the patch. First, about the behavior:
>
> - is that ok to have just "data" pattern instead of "table_data" or
> something
> like that, since it's supposed to match --exclude-table-data option?
>
done
>
> - the error message are sometimes not super helpful. For instance:
>
> $ echo "include data t1" | pg_dump --filter -
> pg_dump: error: invalid format of filter on line 1: include filter is not
> allowed for this type of object
>
> It would be nice if the error message mentioned "data" rather than a
> generic
> "this type of object". Also, maybe we should quote "include" to outline
> that
> we found this keyword?
>
>
done
> About the patch itself:
> filter.c:
>
> +#include "postgres_fe.h"
> +
> +#include "filter.h"
> +
> +#include "common/logging.h"
>
> the filter.h inclusion should be done with the rest of the includes, in
> alphabetical order.
>
>
done
> +#define is_keyword_str(cstr, str, bytes) \
> + ((strlen(cstr) == bytes) && (pg_strncasecmp(cstr, str, bytes) ==
> 0))
>
> nit: our guidline is to protect macro arguments with parenthesis. Some
> arguments can be evaluated multiple times but I don't think it's worth
> adding a
> comment for that.
>
>
done
> + * Unfortunatelly, implementation of exit_nicely in pg_dump and pg_restore
> + * is different from implementation of this rutine in pg_dumpall. So
> instead
> + * direct calling exit_nicely we have to return some error flag (in this
>
> typos: s/Unfortunatelly/Unfortunately/ and s/rutine/routine/
> Also, it would probably be better to say "instead of directly calling..."
>
>
done
> +static const char *
> +filter_object_type_name(FilterObjectType fot)
> +{
> + switch (fot)
> + {
> + case FILTER_OBJECT_TYPE_NONE:
> + return "comment or empty line";
> +[...]
> + }
> +
> + return "unknown object type";
> +}
>
> I'm wondering if we should add a pg_unreachable() there, some compilers
> might
> complain otherwise. See CreateDestReceiver() for instance for similar
> pattern.
>
done
>
> + * Emit error message "invalid format of filter file ..."
> + *
> + * This is mostly a convenience routine to avoid duplicating file closing
> code
> + * in multiple callsites.
> + */
> +void
> +log_invalid_filter_format(FilterStateData *fstate, char *message)
>
> nit: invalid format *in* filter file...?
>
changed
>
> +void
> +log_unsupported_filter_object_type(FilterStateData *fstate,
> +
> const char *appname,
> +
> FilterObjectType fot)
> +{
> + PQExpBuffer str = createPQExpBuffer();
> +
> + printfPQExpBuffer(str,
> + "The application \"%s\" doesn't
> support filter for object type \"%s\".",
>
> nit: there shouldn't be uppercase in error messages, especially since this
> will
> be appended to another message by log_invalid_filter_format. I would just
> just
> drop "The application" entirely for brevity.
>
changed
>
> +/*
> + * Release allocated resources for filter
> + */
> +void
> +filter_free(FilterStateData *fstate)
>
> nit: Release allocated resources for *the given* filter?
>
changed
>
> + * Search for keywords (limited to ascii alphabetic characters) in
> + * the passed in line buffer. Returns NULL, when the buffer is empty or
> first
> + * char is not alpha. The length of the found keyword is returned in the
> size
> + * parameter.
> + */
> +static const char *
> +filter_get_keyword(const char **line, int *size)
> +{
> + [...]
> + if (isascii(*ptr) && isalpha(*ptr))
> + {
> + result = ptr++;
> +
> + while (isascii(*ptr) && (isalpha(*ptr) || *ptr == '_'))
> + ptr++;
>
> Is there any reason to test isascii()? isalpha() should already cover
> that and
> should be cheaper to test anyway.
>
changed. I wanted to limit keyword's char just for basic ascii alphabets,
but the benefit probably is not too strong, and the real effect can be
messy, so I removed isascii test
>
> Also nit: "Returns NULL when the buffer..." (unnecessary comma), and the
> '_'
> char is also allowed.
>
done
>
> +filter_read_item(FilterStateData *fstate,
> + bool *is_include,
> + char **objname,
> + FilterObjectType *objtype)
> +{
> + Assert(!fstate->is_error);
> +
> + if (pg_get_line_buf(fstate->fp, &fstate->linebuff))
> + {
> + char *str = fstate->linebuff.data;
> + const char *keyword;
> + int size;
> +
> + fstate->lineno++;
> +
> + (void) pg_strip_crlf(str);
> +
> + /* Skip initial white spaces */
> + while (isspace(*str))
> + str++;
> +[...]
> + keyword = filter_get_keyword((const char **) &str,
> &size);
>
> Is there any interest with the initial pg_strip_crlf? AFAICT all the rest
> of
> the code will ignore such caracters using isspace() so it wouldn't change
> anything.
>
I think reading multiline identifiers is a little bit easier, because I
don't need to check the ending \n and \r
When I read multiline identifiers, I cannot ignore white spaces.
> Dropping both pg_strip_crlf() would allow you to declare str as const
> rather
> than doing it in function calls. It would require to add const qualifiers
> in a
> few other places, but it seems like an improvement, as for instance right
> now
> filter_get_pattern is free to rewrite the str (because it's also calling
> pg_strip_crlf, but there's no guarantee that it doesn't do anything else).
>
> +/*
> + * filter_get_pattern - Read an object identifier pattern from the buffer
> + *
> + * Parses an object identifier pattern from the passed in buffer and sets
> + * objname to a string with object identifier pattern. Returns pointer to
> the
> + * first character after the pattern. Returns NULL on error.
> + */
> +static char *
> +filter_get_pattern(FilterStateData *fstate,
>
> nit: suggestion to reword the comment, maybe something like
>
> /*
> * filter_get_pattern - Identify an object identifier pattern
> *
> * Try to parse an object identifier pattern from the passed buffer. If
> one is
> * found, it sets objname to a string with the object identifier pattern
> and
> * returns a pointer to the first byte after the found pattern. Otherwise
> NULL
> * is returned.
> */
>
>
replaced
> +bool
> +filter_read_item(FilterStateData *fstate,
>
> Another suggestion for comment rewrite:
>
> /*-------------------
> * filter_read_item - Read command/type/pattern triplet from a filter file
> *
> * This will parse one filter item from the filter file, and while it is a
> * row based format a pattern may span more than one line due to how object
> * names can be constructed. The expected format of the filter file is:
> *
> * <command> <object_type> <pattern>
> *
> * command can be "include" or "exclude"
> * object_type can one of: "table", "schema", "foreign_data", "data",
> * "database", "function", "trigger" or "index"
> * pattern can be any possibly-quoted and possibly-qualified identifier.
> It
> * follows the same rules as other object include and exclude functions so
> it
> * can also use wildcards.
> *
> * Returns true when one filter item was successfully read and parsed.
> When
> * object name contains \n chars, then more than one line from input file
> can
> * be processed. Returns false when the filter file reaches EOF. In case
> of
> * error, the function will emit an appropriate error message before
> returning
> * false.
> */
>
>
replaced, thank you for the text
> Note also that your original comment said:
> + * In case of
> + * errors, the function wont return but will exit with an appropriate
> error
> + * message.
>
> But AFAICS that's not the case: it will indeed log an appropriate error
> message
> but will return false. I'm assuming that the comment was outdated as the
> calling code handles it just fine, so I just modified the comment.
>
yes
>
> filter.h:
>
> +#ifndef FILTER_H
> +#define FILTER_H
> +#include "c.h"
>
> It's definitely not ok to include .ch in frontend code. But AFAICS just
> removing it doesn't cause any problem. Note also that there should be an
> empty
> line after the #define FILTER_H per usual coding style.
>
fixed - it looks so it was some garbage
updated patch attached
big thanks for these comments and tips
Regards
Pavel
Attachments:
[text/x-patch] pg_dump-filter-20221103.patch (50.6K, ../../CAFj8pRAyyXBFiD9U6Gpo7M++Jq33B_L1gWJEnGJu=WcMk+McQA@mail.gmail.com/3-pg_dump-filter-20221103.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 8b9d9f4cad..d5a6e2c7ee 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | table_data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>table_data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index e62d05e5ab..9cad26bbe6 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpretted according to the same rules
+ as <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for excluding databases,
+ and can also be specified more than once for multiple filter files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 47bd7dbda0..ffeb564c52 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules as <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 9dc5a784dd..700e1400c8 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -26,6 +26,7 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filter.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -43,8 +44,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/filter.c b/src/bin/pg_dump/filter.c
new file mode 100644
index 0000000000..76e785d2f8
--- /dev/null
+++ b/src/bin/pg_dump/filter.c
@@ -0,0 +1,469 @@
+/*-------------------------------------------------------------------------
+ *
+ * Implementation of simple filter file parser
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include "common/fe_memutils.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "filter.h"
+#include "lib/stringinfo.h"
+#include "pqexpbuffer.h"
+
+#define is_keyword_str(cstr, str, bytes) \
+ ((strlen(cstr) == (bytes)) && (pg_strncasecmp((cstr), (str), (bytes)) == 0))
+
+/*
+ * Following routines are called from pg_dump, pg_dumpall and pg_restore.
+ * Unfortunately, implementation of exit_nicely in pg_dump and pg_restore
+ * is different from implementation of this routine in pg_dumpall. So instead
+ * of directly calling exit_nicely we have to return some error flag (in this
+ * case NULL), and exit_nicelly will be executed from caller's routine.
+ */
+
+/*
+ * Opens filter's file and initialize fstate structure.
+ * Returns true on success.
+ */
+bool
+filter_init(FilterStateData *fstate, const char *filename)
+{
+ fstate->filename = filename;
+ fstate->lineno = 0;
+ initStringInfo(&fstate->linebuff);
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate->fp = fopen(filename, "r");
+ if (!fstate->fp)
+ {
+ pg_log_error("could not open filter file \"%s\": %m", filename);
+ return false;
+ }
+ }
+ else
+ fstate->fp = stdin;
+
+ fstate->is_error = false;
+
+ return true;
+}
+
+/*
+ * Release allocated resources for the given filter.
+ */
+void
+filter_free(FilterStateData *fstate)
+{
+ free(fstate->linebuff.data);
+ fstate->linebuff.data = NULL;
+
+ if (fstate->fp && fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ pg_log_error("could not close filter file \"%s\": %m", fstate->filename);
+
+ fstate->fp = NULL;
+ }
+}
+
+/*
+ * Translate FilterObjectType enum to string. It is designed for formatting
+ * of error message in log_unsupported_filter_object_type routine.
+ */
+static const char *
+filter_object_type_name(FilterObjectType fot)
+{
+ switch (fot)
+ {
+ case FILTER_OBJECT_TYPE_NONE:
+ return "comment or empty line";
+ case FILTER_OBJECT_TYPE_TABLE_DATA:
+ return "table data";
+ case FILTER_OBJECT_TYPE_DATABASE:
+ return "database";
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ return "foreign data";
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ return "function";
+ case FILTER_OBJECT_TYPE_INDEX:
+ return "index";
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ return "schema";
+ case FILTER_OBJECT_TYPE_TABLE:
+ return "table";
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ return "trigger";
+ }
+
+ /* should never get here */
+ pg_unreachable();
+}
+
+/*
+ * Emit error message "invalid format in filter file ..."
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format in filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+ }
+ else
+ pg_log_error("invalid format in filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ fstate->is_error = true;
+}
+
+/*
+ * Emit error message "The application doesn't support filter for object type ..."
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname,
+ FilterObjectType fot)
+{
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str,
+ "\"%s\" doesn't support filter for object type \"%s\".",
+ appname,
+ filter_object_type_name(fot));
+
+ log_invalid_filter_format(fstate, str->data);
+}
+
+/*
+ * filter_get_keyword - read the next filter keyword from buffer
+ *
+ * Search for keywords (limited to ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL when the buffer is empty or first
+ * char is not alpha. The char '_' is allowed too (exclude first position).
+ * The length of the found keyword is returned in the size parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* Set returnlength preemptively in case no keyword is found */
+ *size = 0;
+
+ /* Skip initial whitespace */
+ while (isspace(*ptr))
+ ptr++;
+
+ if (isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isalpha(*ptr) || *ptr == '_')
+ ptr++;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * filter_get_pattern - Identify an object identifier pattern
+ *
+ * Try to parse an object identifier pattern from the passed buffer. If one is
+ * found, it sets objname to a string with the object identifier pattern and
+ * returns a pointer to the first byte after the found pattern. Otherwise NULL
+ * is returned.
+ */
+static char *
+filter_get_pattern(FilterStateData *fstate,
+ char *str,
+ char **objname)
+{
+ /* Skip whitespace */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ {
+ log_invalid_filter_format(fstate, "missing object name pattern");
+ return NULL;
+ }
+
+ /*
+ * If the object name pattern has been quoted, we must take care to parse
+ * out the entire quoted pattern, which may contain whitespace and can span
+ * many lines.
+ */
+ if (*str == '"')
+ {
+ PQExpBuffer quoted_name = createPQExpBuffer();
+
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ while (1)
+ {
+ if (*str == '\0')
+ {
+ Assert(fstate->linebuff.data);
+
+ if (!pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m",
+ fstate->filename);
+ fstate->is_error = true;
+ }
+ else
+ log_invalid_filter_format(fstate, "unexpected end of file");
+
+ return NULL;
+ }
+
+ str = fstate->linebuff.data;
+ (void) pg_strip_crlf(str);
+
+ appendPQExpBufferChar(quoted_name, '\n');
+ fstate->lineno++;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(quoted_name, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(quoted_name, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(quoted_name, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(quoted_name, *str++);
+ }
+
+ *objname = pg_strdup(quoted_name->data);
+ destroyPQExpBuffer(quoted_name);
+ }
+ else
+ {
+ char *startptr = str++;
+
+ /* Simple variant, read to EOL or to first whitespace */
+ while (*str && !isspace(*str))
+ str++;
+
+ *objname = pnstrdup(startptr, str - startptr);
+ }
+
+ return str;
+}
+
+/*-------------------
+ * filter_read_item - Read command/type/pattern triplet from a filter file
+ *
+ * This will parse one filter item from the filter file, and while it is a
+ * row based format a pattern may span more than one line due to how object
+ * names can be constructed. The expected format of the filter file is:
+ *
+ * <command> <object_type> <pattern>
+ *
+ * command can be "include" or "exclude"
+ * object_type can one of: "table", "schema", "foreign_data", "table_data",
+ * "database", "function", "trigger" or "index"
+ * pattern can be any possibly-quoted and possibly-qualified identifier. It
+ * follows the same rules as other object include and exclude functions so it
+ * can also use wildcards.
+ *
+ * Returns true when one filter item was successfully read and parsed. When
+ * object name contains \n chars, then more than one line from input file can
+ * be processed. Returns false when the filter file reaches EOF. In case of
+ * error, the function will emit an appropriate error message before returning
+ * false.
+ */
+bool
+filter_read_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ Assert(!fstate->is_error);
+
+ if (pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ char *str = fstate->linebuff.data;
+ const char *keyword;
+ int size;
+
+ fstate->lineno++;
+
+ (void) pg_strip_crlf(str);
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Skip empty lines or lines where the first non-whitespace character
+ * is a hash indicating a comment.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ /*
+ * First we expect sequence of two keywords, {include|exclude}
+ * followed by the object type to operate on.
+ */
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate,
+ "no filter command found (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ if (is_keyword_str("include", keyword, size))
+ *is_include = true;
+ else if (is_keyword_str("exclude", keyword, size))
+ *is_include = false;
+ else
+ {
+ log_invalid_filter_format(fstate,
+ "invalid filter command (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate, "missing filter object type");
+ return false;
+ }
+
+ if (is_keyword_str("table_data", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE_DATA;
+ else if (is_keyword_str("database", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_DATABASE;
+ else if (is_keyword_str("foreign_data",keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (is_keyword_str("function", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FUNCTION;
+ else if (is_keyword_str("index", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_INDEX;
+ else if (is_keyword_str("schema", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (is_keyword_str("table", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (is_keyword_str("trigger", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TRIGGER;
+ else
+ {
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str, "unsupported filter object type: \"%.*s\"", size, keyword);
+ log_invalid_filter_format(fstate, str->data);
+ return false;
+ }
+
+ str = filter_get_pattern(fstate, str, objname);
+ if (!str)
+ return false;
+
+ /* skip whitespaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * qualified identifier can be valid pattern. So repeat
+ * reading if char after pattern is dot.
+ */
+ if (*str == '.')
+ {
+ PQExpBuffer qual_name = createPQExpBuffer();
+
+ appendPQExpBufferStr(qual_name, *objname);
+ free(*objname);
+
+ do
+ {
+ str = filter_get_pattern(fstate, ++str, objname);
+ if (!str)
+ return false;
+
+ appendPQExpBufferChar(qual_name, '.');
+ appendPQExpBufferStr(qual_name, *objname);
+ free(*objname);
+
+ while (isspace(*str))
+ str++;
+ }
+ while (*str == '.');
+
+ *objname = qual_name->data;
+ }
+
+ /*
+ * Look for any content after the object identifier. Comments and
+ * whitespace are allowed, other content may indicate that the
+ * user needed to quote the object name so exit with an invalid
+ * format error.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ log_invalid_filter_format(fstate,
+ "unexpected extra data after pattern");
+ return false;
+ }
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m", fstate->filename);
+ fstate->is_error = true;
+ }
+
+ return false;
+}
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..251763ad8e
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,54 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+
+#include "lib/stringinfo.h"
+
+/*
+ * State data for reading filter items from stream
+ */
+typedef struct
+{
+ FILE *fp;
+ const char *filename;
+ int lineno;
+ StringInfoData linebuff;
+ bool is_error;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern bool filter_init(FilterStateData *fstate, const char *filename);
+extern void filter_free(FilterStateData *fstate);
+
+extern void log_invalid_filter_format(FilterStateData *fstate, char *message);
+extern void log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname, FilterObjectType fot);
+extern bool filter_read_item(FilterStateData *fstate, bool *is_include, char **objname, FilterObjectType *objtype);
+
+#endif
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index e66f632b54..c099d66450 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -1,6 +1,7 @@
pg_dump_common_sources = files(
'compress_io.c',
'dumputils.c',
+ 'filter.c',
'parallel.c',
'pg_backup_archiver.c',
'pg_backup_custom.c',
@@ -86,6 +87,7 @@ tests += {
't/001_basic.pl',
't/002_pg_dump.pl',
't/003_pg_dump_with_server.pl',
+ 't/004_pg_dump_filterfile.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index da427f4d4a..a231dc2bb3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -59,6 +59,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -318,6 +319,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AH);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void read_dump_filters(const char *filename, DumpOptions *dopt);
int
@@ -390,6 +392,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -623,6 +626,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ read_dump_filters(optarg, &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1028,6 +1035,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18198,3 +18207,89 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+/*
+ * read_dump_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_dump_filters(const char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments and empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_TABLE_DATA)
+ {
+ if (is_include)
+ {
+ log_invalid_filter_format(&fstate,
+ "\"include\" table data filter is not allowed");
+ break;
+ }
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" foreign data filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dump", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 083012ca39..4597ac3822 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -26,6 +26,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -81,6 +82,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void read_dumpall_filters(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -158,6 +160,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -360,6 +363,10 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ read_dumpall_filters(optarg, &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -653,6 +660,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1890,7 +1898,6 @@ executeCommand(PGconn *conn, const char *query)
PQclear(res);
}
-
/*
* dumpTimestamp
*/
@@ -1914,3 +1921,55 @@ hash_string_pointer(char *s)
return hash_bytes(ss, strlen(s));
}
+
+/*
+ * read_dumpall_filters - retrieve database identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ *
+ * In this moment only excluded databases can be filtered.
+ */
+static void
+read_dumpall_filters(const char *filename, SimpleStringList *pattern)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATABASE)
+ {
+ if (!is_include)
+ simple_string_list_append(pattern, objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"include\" database filter is not allowed");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dumpall", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..9a5cdfd6eb 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,11 +47,13 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
static void usage(const char *progname);
+static void read_restore_filters(const char *filename, RestoreOptions *dopt);
int
main(int argc, char **argv)
@@ -123,6 +125,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +289,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ read_restore_filters(optarg, opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +470,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +502,109 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * read_restore_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_restore_filters(const char *filename, RestoreOptions *opts)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments or empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_FUNCTION)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" function filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_INDEX)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" type filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ simple_string_list_append(&opts->schemaNames, objname);
+ else
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" table filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TRIGGER)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, optarg);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" trigger filter is not allowed");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_restore", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..e72d338372
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,514 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 69;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;;
+my $inputfile;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres',
+ 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three_one(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres',
+ "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three_one VALUES('*** TABLE THREE_ONE ***')");
+
+#
+# Test interaction of correctly specified filter file
+#
+my ($cmd, $stdout, $stderr, $result);
+
+# Empty filterfile
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "\n # a comment and nothing more\n\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "table one dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "table two dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "table three dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m, "table three one dumped");
+
+# Test various combinations of whitespace, comments and correct filters
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "\t\n";
+print $inputfile " \t# another comment\n";
+print $inputfile "exclude table_data table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter patterns as well as comments and whitespace");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public\.table_three/m, "table three not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_three_one/m,
+ "table three_one not dumped");
+ok( $dump !~ qr/^COPY public\.table_one/m,
+ "content of table one is not included");
+ok($dump =~ qr/^COPY public\.table_two/m, "content of table two is included");
+
+# Test dumping tables specified by qualified names
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table public.table_one\n";
+print $inputfile "include table \"public\".\"table_two\"\n";
+print $inputfile "include table \"public\". table_three\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with exclusion of a single table");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping tables with a wildcard pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_thre*\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with wildcard in pattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_two/m, "table two not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping table with multiline quoted tablename
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with multiline names requiring quoting");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding multiline quoted tablename from dump
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding an entire schema
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "exclude the public schema");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test including and excluding an entire schema by multiple filterfiles
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema public\n";
+close $inputfile;
+
+open my $alt_inputfile, '>', "$tempdir/inputfile2.txt"
+ or die "unable to open filterfile for writing";
+print $alt_inputfile "exclude schema public\n";
+close $alt_inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "--filter=$tempdir/inputfile2.txt", 'postgres'
+ ],
+ "exclude the public schema with multiple filters");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test dumping a table with a single leading newline on a row
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump nonexisting foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummyserver\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump foreign_data with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE SERVER dummyserver/m, "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude foreign_data dummy*\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/"exclude" foreign data filter is not allowed/,
+ "erroneously exclude foreign server");
+
+#########################################
+# Test broken input format
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: invalid object type specified, should be table, schema, foreign_data or data"
+);
+
+# Test missing object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/missing object name/,
+ "invalid syntax: missing object identifier pattern");
+
+# Test adding extra content after the object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table one";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unexpected extra data/,
+ "invalid syntax: extra content after object identifier pattern");
+
+#########################################
+# Combined with --strict-names
+
+# First ensure that a matching filter works
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ "strict names with matching mattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "no table dumped");
+
+# Now append a pattern to the filter file which doesn't resolve
+open $inputfile, '>>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_nonexisting_name";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "inclusion of non-existing objects with --strict names");
+
+#########################################
+# pg_dumpall tests
+
+###########################
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude database postgres\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ "dump tables with exclusion of a database");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^\\connect postgres/m, "database postgres is not dumped");
+ok($dump =~ qr/^\\connect template1/m, "database template1 is dumped");
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: exclusion of non-existing object type"
+);
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table foo";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/"pg_dumpall" doesn't support filter for object type "table"/,
+ "invalid syntax: exclusion of unsupported object type"
+);
+
+#########################################
+# pg_restore tests
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', "$tempdir/filter_test.dump",
+ "-Fc", 'postgres'
+ ],
+ "dump all tables");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_two";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "wanted table restored");
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "unwanted tables are not restored");
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 83a3e40425..d3c7292203 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -451,6 +451,7 @@ sub mkvcbuild
$pgdumpall->AddIncludeDir('src/backend');
$pgdumpall->AddFile('src/bin/pg_dump/pg_dumpall.c');
$pgdumpall->AddFile('src/bin/pg_dump/dumputils.c');
+ $pgdumpall->AddFile('src/bin/pg_dump/filter.c');
$pgdumpall->AddLibrary('ws2_32.lib');
my $pgrestore = AddSimpleFrontend('pg_dump', 1);
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-04 11:59 Julien Rouhaud <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Julien Rouhaud @ 2022-11-04 11:59 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Justin Pryzby <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Thu, Nov 03, 2022 at 10:22:15PM +0100, Pavel Stehule wrote:
> čt 3. 11. 2022 v 5:09 odesílatel Julien Rouhaud <[email protected]> napsal:
>
> >
> > Is there any interest with the initial pg_strip_crlf? AFAICT all the rest
> > of
> > the code will ignore such caracters using isspace() so it wouldn't change
> > anything.
> >
>
> I think reading multiline identifiers is a little bit easier, because I
> don't need to check the ending \n and \r
> When I read multiline identifiers, I cannot ignore white spaces.
Ok. I don't have a strong objection to it.
>
> updated patch attached
>
> big thanks for these comments and tips
Thanks for the updated patch! As far as I'm concerned the patch is in a good
shape, passes the CI and I don't have anything more to say so I'm marking it as
Ready for Committer!
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-04 12:19 Justin Pryzby <[email protected]>
parent: Julien Rouhaud <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Justin Pryzby @ 2022-11-04 12:19 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Pavel Stehule <[email protected]>; Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Fri, Nov 04, 2022 at 07:59:01PM +0800, Julien Rouhaud wrote:
> On Thu, Nov 03, 2022 at 10:22:15PM +0100, Pavel Stehule wrote:
> > čt 3. 11. 2022 v 5:09 odesílatel Julien Rouhaud <[email protected]> napsal:
> > updated patch attached
> >
> > big thanks for these comments and tips
>
> Thanks for the updated patch! As far as I'm concerned the patch is in a good
> shape, passes the CI and I don't have anything more to say so I'm marking it as
> Ready for Committer!
+1
I started looking to see if it's possible to simplify the patch at all,
but nothing to show yet.
But one thing I noticed is that "optarg" looks wrong here:
simple_string_list_append(&opts->triggerNames, optarg);
--
Justin
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-04 13:28 Julien Rouhaud <[email protected]>
parent: Justin Pryzby <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Julien Rouhaud @ 2022-11-04 13:28 UTC (permalink / raw)
To: Justin Pryzby <[email protected]>; +Cc: Pavel Stehule <[email protected]>; Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Fri, Nov 04, 2022 at 07:19:27AM -0500, Justin Pryzby wrote:
> On Fri, Nov 04, 2022 at 07:59:01PM +0800, Julien Rouhaud wrote:
> > On Thu, Nov 03, 2022 at 10:22:15PM +0100, Pavel Stehule wrote:
> > > čt 3. 11. 2022 v 5:09 odesílatel Julien Rouhaud <[email protected]> napsal:
> > > updated patch attached
> > >
> > > big thanks for these comments and tips
> >
> > Thanks for the updated patch! As far as I'm concerned the patch is in a good
> > shape, passes the CI and I don't have anything more to say so I'm marking it as
> > Ready for Committer!
>
> +1
>
> I started looking to see if it's possible to simplify the patch at all,
> but nothing to show yet.
>
> But one thing I noticed is that "optarg" looks wrong here:
>
> simple_string_list_append(&opts->triggerNames, optarg);
Ah indeed, good catch! Maybe there should be an explicit test for every
(include|exclude) / objtype combination? It would be a bit verbose (and
possibly hard to maintain).
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-04 13:37 Pavel Stehule <[email protected]>
parent: Julien Rouhaud <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2022-11-04 13:37 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
pá 4. 11. 2022 v 14:28 odesílatel Julien Rouhaud <[email protected]>
napsal:
> On Fri, Nov 04, 2022 at 07:19:27AM -0500, Justin Pryzby wrote:
> > On Fri, Nov 04, 2022 at 07:59:01PM +0800, Julien Rouhaud wrote:
> > > On Thu, Nov 03, 2022 at 10:22:15PM +0100, Pavel Stehule wrote:
> > > > čt 3. 11. 2022 v 5:09 odesílatel Julien Rouhaud <[email protected]>
> napsal:
> > > > updated patch attached
> > > >
> > > > big thanks for these comments and tips
> > >
> > > Thanks for the updated patch! As far as I'm concerned the patch is in
> a good
> > > shape, passes the CI and I don't have anything more to say so I'm
> marking it as
> > > Ready for Committer!
> >
> > +1
> >
> > I started looking to see if it's possible to simplify the patch at all,
> > but nothing to show yet.
> >
> > But one thing I noticed is that "optarg" looks wrong here:
> >
> > simple_string_list_append(&opts->triggerNames, optarg);
>
> Ah indeed, good catch! Maybe there should be an explicit test for every
> (include|exclude) / objtype combination? It would be a bit verbose (and
> possibly hard to maintain).
>
I'll do it
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-05 05:39 Julien Rouhaud <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Julien Rouhaud @ 2022-11-05 05:39 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Fri, Nov 04, 2022 at 02:37:05PM +0100, Pavel Stehule wrote:
> pá 4. 11. 2022 v 14:28 odesílatel Julien Rouhaud <[email protected]>
> napsal:
>
> > >
> > > But one thing I noticed is that "optarg" looks wrong here:
> > >
> > > simple_string_list_append(&opts->triggerNames, optarg);
> >
> > Ah indeed, good catch! Maybe there should be an explicit test for every
> > (include|exclude) / objtype combination? It would be a bit verbose (and
> > possibly hard to maintain).
> >
>
> I'll do it
Thanks a lot Pavel! I switched the CF entry back to "Waiting on Author" in the
meantime.
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-05 19:54 Pavel Stehule <[email protected]>
parent: Julien Rouhaud <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Pavel Stehule @ 2022-11-05 19:54 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
pá 4. 11. 2022 v 14:28 odesílatel Julien Rouhaud <[email protected]>
napsal:
> On Fri, Nov 04, 2022 at 07:19:27AM -0500, Justin Pryzby wrote:
> > On Fri, Nov 04, 2022 at 07:59:01PM +0800, Julien Rouhaud wrote:
> > > On Thu, Nov 03, 2022 at 10:22:15PM +0100, Pavel Stehule wrote:
> > > > čt 3. 11. 2022 v 5:09 odesílatel Julien Rouhaud <[email protected]>
> napsal:
> > > > updated patch attached
> > > >
> > > > big thanks for these comments and tips
> > >
> > > Thanks for the updated patch! As far as I'm concerned the patch is in
> a good
> > > shape, passes the CI and I don't have anything more to say so I'm
> marking it as
> > > Ready for Committer!
> >
> > +1
> >
> > I started looking to see if it's possible to simplify the patch at all,
> > but nothing to show yet.
> >
> > But one thing I noticed is that "optarg" looks wrong here:
> >
> > simple_string_list_append(&opts->triggerNames, optarg);
>
> Ah indeed, good catch! Maybe there should be an explicit test for every
> (include|exclude) / objtype combination? It would be a bit verbose (and
> possibly hard to maintain).
>
yes - pg_restore is not well covered by tests, fixed
I found another issue. The pg_restore requires a full signature of the
function and it is pretty sensitive on white spaces (pg_restore). I made a
mistake when I partially parsed patterns like SQL identifiers. It can work
for simple cases, but when I parse the function's signature it stops
working. So I rewrote the parsing pattern part. Now, I just read an input
string and I try to reduce spaces. Still multiline identifiers are
supported. Against the previous method of pattern parsing, I needed to
change just one regress test - now I am not able to detect garbage after
pattern :-/. It is possible to enter types like "double precision" or
"timestamp with time zone", without needing to check it on the server side.
When I wroted regress tests I found some issues of pg_restore filtering
options (not related to this patch)
* function's filtering doesn't support schema - when the name of function
is specified with schema, then the function is not found
* the function has to be specified with an argument type list - the
separator has to be exactly ", " string. Without space or with one space
more, the filtering doesn't work (new implementation of pattern parsing
reduces white spaces sensitivity). This is not a bug, but it is not well
documented.
* the trigger filtering is probably broken (on pg_restore side). The name
should be entered in form "tablename triggername"
attached updated patch
Regards
Pavel
Attachments:
[text/x-patch] pg_dump-filter-20221005.patch (55.5K, ../../CAFj8pRBB_9jdr8LoL9r-gQ9O1aRdPXXO0iH-HrYkC3jPgxY5Aw@mail.gmail.com/3-pg_dump-filter-20221005.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 8b9d9f4cad..d5a6e2c7ee 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | table_data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>table_data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index e62d05e5ab..9cad26bbe6 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpretted according to the same rules
+ as <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for excluding databases,
+ and can also be specified more than once for multiple filter files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 47bd7dbda0..ffeb564c52 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules as <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 9dc5a784dd..700e1400c8 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -26,6 +26,7 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filter.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -43,8 +44,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/filter.c b/src/bin/pg_dump/filter.c
new file mode 100644
index 0000000000..65ffb1f432
--- /dev/null
+++ b/src/bin/pg_dump/filter.c
@@ -0,0 +1,480 @@
+/*-------------------------------------------------------------------------
+ *
+ * Implementation of simple filter file parser
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include "common/fe_memutils.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "filter.h"
+#include "lib/stringinfo.h"
+#include "pqexpbuffer.h"
+
+#define is_keyword_str(cstr, str, bytes) \
+ ((strlen(cstr) == (bytes)) && (pg_strncasecmp((cstr), (str), (bytes)) == 0))
+
+/*
+ * Following routines are called from pg_dump, pg_dumpall and pg_restore.
+ * Unfortunately, implementation of exit_nicely in pg_dump and pg_restore
+ * is different from implementation of this routine in pg_dumpall. So instead
+ * of directly calling exit_nicely we have to return some error flag (in this
+ * case NULL), and exit_nicelly will be executed from caller's routine.
+ */
+
+/*
+ * Opens filter's file and initialize fstate structure.
+ * Returns true on success.
+ */
+bool
+filter_init(FilterStateData *fstate, const char *filename)
+{
+ fstate->filename = filename;
+ fstate->lineno = 0;
+ initStringInfo(&fstate->linebuff);
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate->fp = fopen(filename, "r");
+ if (!fstate->fp)
+ {
+ pg_log_error("could not open filter file \"%s\": %m", filename);
+ return false;
+ }
+ }
+ else
+ fstate->fp = stdin;
+
+ fstate->is_error = false;
+
+ return true;
+}
+
+/*
+ * Release allocated resources for the given filter.
+ */
+void
+filter_free(FilterStateData *fstate)
+{
+ free(fstate->linebuff.data);
+ fstate->linebuff.data = NULL;
+
+ if (fstate->fp && fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ pg_log_error("could not close filter file \"%s\": %m", fstate->filename);
+
+ fstate->fp = NULL;
+ }
+}
+
+/*
+ * Translate FilterObjectType enum to string. It is designed for formatting
+ * of error message in log_unsupported_filter_object_type routine.
+ */
+static const char *
+filter_object_type_name(FilterObjectType fot)
+{
+ switch (fot)
+ {
+ case FILTER_OBJECT_TYPE_NONE:
+ return "comment or empty line";
+ case FILTER_OBJECT_TYPE_TABLE_DATA:
+ return "table data";
+ case FILTER_OBJECT_TYPE_DATABASE:
+ return "database";
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ return "foreign data";
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ return "function";
+ case FILTER_OBJECT_TYPE_INDEX:
+ return "index";
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ return "schema";
+ case FILTER_OBJECT_TYPE_TABLE:
+ return "table";
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ return "trigger";
+ }
+
+ /* should never get here */
+ pg_unreachable();
+}
+
+/*
+ * Emit error message "invalid format in filter file ..."
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format in filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+ }
+ else
+ pg_log_error("invalid format in filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ fstate->is_error = true;
+}
+
+/*
+ * Emit error message "The application doesn't support filter for object type ..."
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname,
+ FilterObjectType fot)
+{
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str,
+ "\"%s\" doesn't support filter for object type \"%s\".",
+ appname,
+ filter_object_type_name(fot));
+
+ log_invalid_filter_format(fstate, str->data);
+}
+
+/*
+ * filter_get_keyword - read the next filter keyword from buffer
+ *
+ * Search for keywords (limited to ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL when the buffer is empty or first
+ * char is not alpha. The char '_' is allowed too (exclude first position).
+ * The length of the found keyword is returned in the size parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* Set returnlength preemptively in case no keyword is found */
+ *size = 0;
+
+ /* Skip initial whitespace */
+ while (isspace(*ptr))
+ ptr++;
+
+ if (isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isalpha(*ptr) || *ptr == '_')
+ ptr++;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * read_quoted_pattern - read quoted possibly multi lined string.
+ *
+ * Returns pointer to next char after ending double quotes or NULL on error.
+ */
+static const char *
+read_quoted_string(FilterStateData *fstate,
+ const char *str,
+ PQExpBuffer pattern)
+{
+ appendPQExpBufferChar(pattern, '"');
+ str++;
+
+ while (1)
+ {
+ /*
+ * We can ignore \r or \n chars because the string is read by
+ * pg_get_line_buf, so these chars should be just trailing chars.
+ */
+ if (*str == '\r' || *str == '\n')
+ {
+ str++;
+ continue;
+ }
+
+ if (*str == '\0')
+ {
+ Assert(fstate->linebuff.data);
+
+ if (!pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m",
+ fstate->filename);
+ fstate->is_error = true;
+ }
+ else
+ log_invalid_filter_format(fstate, "unexpected end of file");
+
+ return NULL;
+ }
+
+ str = fstate->linebuff.data;
+
+ appendPQExpBufferChar(pattern, '\n');
+ fstate->lineno++;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(pattern, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(pattern, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(pattern, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(pattern, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(pattern, *str++);
+ }
+
+ return str;
+}
+
+/*
+ * read_pattern - reads an pattern from input. The pattern can be mix of
+ * single line or multi line subpatterns. Single line subpattern starts first
+ * non white space char, and ending last non space char on line or by char
+ * '#'. The white spaces inside are removed (around char ".()"), or reformated
+ * around char ',' or reduced (the multiple spaces are replaced by one).
+ * Multiline subpattern starts by double quote and ending by this char too.
+ * The escape rules are same like for SQL quoted literal.
+ *
+ * Routine signalizes error by returning NULL. Otherwise returns pointer
+ * to next char after last processed char in input string.
+ */
+static const char *
+read_pattern(FilterStateData *fstate, const char *str, PQExpBuffer pattern)
+{
+ bool skip_space = true;
+ bool found_space = false;
+
+ /* Skip initial whitespace */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ {
+ log_invalid_filter_format(fstate, "missing object name pattern");
+ return NULL;
+ }
+
+ while (*str && *str != '#')
+ {
+ while (*str && !isspace(*str) && !strchr("#,.()\"", *str))
+ {
+ /*
+ * Append space only when it is allowed, and when it was found
+ * in original string.
+ */
+ if (!skip_space && found_space)
+ {
+ appendPQExpBufferChar(pattern, ' ');
+ skip_space = true;
+ }
+
+ appendPQExpBufferChar(pattern, *str++);
+ }
+
+ skip_space = false;
+
+ if (*str == '"')
+ {
+ if (found_space)
+ appendPQExpBufferChar(pattern, ' ');
+
+ str = read_quoted_string(fstate, str, pattern);
+ if (!str)
+ return NULL;
+ }
+ else if (*str == ',')
+ {
+ appendPQExpBufferStr(pattern, ", ");
+ skip_space = true;
+ str++;
+ }
+ else if (strchr(".()", *str))
+ {
+ appendPQExpBufferChar(pattern, *str++);
+ skip_space = true;
+ }
+
+ found_space = false;
+
+ /* skip ending whitespaces */
+ while (isspace(*str))
+ {
+ found_space = true;
+ str++;
+ }
+ }
+
+ return str;
+}
+
+/*
+ * filter_read_item - Read command/type/pattern triplet from a filter file
+ *
+ * This will parse one filter item from the filter file, and while it is a
+ * row based format a pattern may span more than one line due to how object
+ * names can be constructed. The expected format of the filter file is:
+ *
+ * <command> <object_type> <pattern>
+ *
+ * command can be "include" or "exclude"
+ * object_type can one of: "table", "schema", "foreign_data", "table_data",
+ * "database", "function", "trigger" or "index"
+ * pattern can be any possibly-quoted and possibly-qualified identifier. It
+ * follows the same rules as other object include and exclude functions so it
+ * can also use wildcards.
+ *
+ * Returns true when one filter item was successfully read and parsed. When
+ * object name contains \n chars, then more than one line from input file can
+ * be processed. Returns false when the filter file reaches EOF. In case of
+ * error, the function will emit an appropriate error message before returning
+ * false.
+ */
+bool
+filter_read_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ Assert(!fstate->is_error);
+
+ if (pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ const char *str = fstate->linebuff.data;
+ const char *keyword;
+ int size;
+ PQExpBuffer pattern;
+
+ fstate->lineno++;
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Skip empty lines or lines where the first non-whitespace character
+ * is a hash indicating a comment.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ /*
+ * First we expect sequence of two keywords, {include|exclude}
+ * followed by the object type to operate on.
+ */
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate,
+ "no filter command found (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ if (is_keyword_str("include", keyword, size))
+ *is_include = true;
+ else if (is_keyword_str("exclude", keyword, size))
+ *is_include = false;
+ else
+ {
+ log_invalid_filter_format(fstate,
+ "invalid filter command (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ keyword = filter_get_keyword((const char **) &str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate, "missing filter object type");
+ return false;
+ }
+
+ if (is_keyword_str("table_data", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE_DATA;
+ else if (is_keyword_str("database", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_DATABASE;
+ else if (is_keyword_str("foreign_data",keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (is_keyword_str("function", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FUNCTION;
+ else if (is_keyword_str("index", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_INDEX;
+ else if (is_keyword_str("schema", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (is_keyword_str("table", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (is_keyword_str("trigger", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TRIGGER;
+ else
+ {
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str, "unsupported filter object type: \"%.*s\"", size, keyword);
+ log_invalid_filter_format(fstate, str->data);
+ return false;
+ }
+
+ pattern = createPQExpBuffer();
+
+ str = read_pattern(fstate, str, pattern);
+ if (!str)
+ return false;
+
+ *objname = pattern->data;
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m", fstate->filename);
+ fstate->is_error = true;
+ }
+
+ return false;
+}
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..251763ad8e
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,54 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+
+#include "lib/stringinfo.h"
+
+/*
+ * State data for reading filter items from stream
+ */
+typedef struct
+{
+ FILE *fp;
+ const char *filename;
+ int lineno;
+ StringInfoData linebuff;
+ bool is_error;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern bool filter_init(FilterStateData *fstate, const char *filename);
+extern void filter_free(FilterStateData *fstate);
+
+extern void log_invalid_filter_format(FilterStateData *fstate, char *message);
+extern void log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname, FilterObjectType fot);
+extern bool filter_read_item(FilterStateData *fstate, bool *is_include, char **objname, FilterObjectType *objtype);
+
+#endif
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index e66f632b54..c099d66450 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -1,6 +1,7 @@
pg_dump_common_sources = files(
'compress_io.c',
'dumputils.c',
+ 'filter.c',
'parallel.c',
'pg_backup_archiver.c',
'pg_backup_custom.c',
@@ -86,6 +87,7 @@ tests += {
't/001_basic.pl',
't/002_pg_dump.pl',
't/003_pg_dump_with_server.pl',
+ 't/004_pg_dump_filterfile.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index da427f4d4a..a231dc2bb3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -59,6 +59,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -318,6 +319,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AH);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void read_dump_filters(const char *filename, DumpOptions *dopt);
int
@@ -390,6 +392,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -623,6 +626,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ read_dump_filters(optarg, &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1028,6 +1035,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18198,3 +18207,89 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+/*
+ * read_dump_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_dump_filters(const char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments and empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_TABLE_DATA)
+ {
+ if (is_include)
+ {
+ log_invalid_filter_format(&fstate,
+ "\"include\" table data filter is not allowed");
+ break;
+ }
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" foreign data filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dump", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 083012ca39..4597ac3822 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -26,6 +26,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -81,6 +82,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void read_dumpall_filters(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -158,6 +160,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -360,6 +363,10 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ read_dumpall_filters(optarg, &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -653,6 +660,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1890,7 +1898,6 @@ executeCommand(PGconn *conn, const char *query)
PQclear(res);
}
-
/*
* dumpTimestamp
*/
@@ -1914,3 +1921,55 @@ hash_string_pointer(char *s)
return hash_bytes(ss, strlen(s));
}
+
+/*
+ * read_dumpall_filters - retrieve database identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ *
+ * In this moment only excluded databases can be filtered.
+ */
+static void
+read_dumpall_filters(const char *filename, SimpleStringList *pattern)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATABASE)
+ {
+ if (!is_include)
+ simple_string_list_append(pattern, objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"include\" database filter is not allowed");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dumpall", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..1ebf573ac4 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,11 +47,13 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
static void usage(const char *progname);
+static void read_restore_filters(const char *filename, RestoreOptions *dopt);
int
main(int argc, char **argv)
@@ -123,6 +125,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +289,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ read_restore_filters(optarg, opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +470,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +502,109 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * read_restore_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_restore_filters(const char *filename, RestoreOptions *opts)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments or empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_FUNCTION)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" function filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_INDEX)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" type filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ simple_string_list_append(&opts->schemaNames, objname);
+ else
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" table filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TRIGGER)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" trigger filter is not allowed");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_restore", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..a1b06845a3
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,643 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 89;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;;
+my $inputfile;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres',
+ 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three_one(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres',
+ "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three_one VALUES('*** TABLE THREE_ONE ***')");
+
+$node->safe_psql('postgres', "CREATE DATABASE sourcedb");
+$node->safe_psql('postgres', "CREATE DATABASE targetdb");
+
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo1(a int) RETURNS int AS $$ select $1 $$ LANGUAGE sql');
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo2(a int) RETURNS int AS $$ select $1 $$ LANGUAGE sql');
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo3(a double precision, b int) RETURNS double precision AS $$ select $1 + $2 $$ LANGUAGE sql');
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo_trg() RETURNS trigger AS $$ BEGIN RETURN NEW; END $$ LANGUAGE plpgsql');
+$node->safe_psql('sourcedb', 'CREATE SCHEMA s1');
+$node->safe_psql('sourcedb', 'CREATE SCHEMA s2');
+$node->safe_psql('sourcedb', 'CREATE TABLE s1.t1(a int)');
+$node->safe_psql('sourcedb', 'CREATE SEQUENCE s1.s1');
+$node->safe_psql('sourcedb', 'CREATE TABLE s2.t2(a int)');
+$node->safe_psql('sourcedb', 'CREATE TABLE t1(a int, b int)');
+$node->safe_psql('sourcedb', 'CREATE TABLE t2(a int, b int)');
+$node->safe_psql('sourcedb', 'CREATE INDEX t1_idx1 ON t1(a)');
+$node->safe_psql('sourcedb', 'CREATE INDEX t1_idx2 ON t1(b)');
+$node->safe_psql('sourcedb', 'CREATE TRIGGER trg1 BEFORE INSERT ON t1 EXECUTE FUNCTION foo_trg()');
+$node->safe_psql('sourcedb', 'CREATE TRIGGER trg2 BEFORE INSERT ON t1 EXECUTE FUNCTION foo_trg()');
+
+#
+# Test interaction of correctly specified filter file
+#
+my ($cmd, $stdout, $stderr, $result);
+
+# Empty filterfile
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "\n # a comment and nothing more\n\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "table one dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "table two dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "table three dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m, "table three one dumped");
+
+# Test various combinations of whitespace, comments and correct filters
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "\t\n";
+print $inputfile " \t# another comment\n";
+print $inputfile "exclude table_data table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter patterns as well as comments and whitespace");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public\.table_three/m, "table three not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_three_one/m,
+ "table three_one not dumped");
+ok( $dump !~ qr/^COPY public\.table_one/m,
+ "content of table one is not included");
+ok($dump =~ qr/^COPY public\.table_two/m, "content of table two is included");
+
+# Test dumping tables specified by qualified names
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table public.table_one\n";
+print $inputfile "include table \"public\".\"table_two\"\n";
+print $inputfile "include table \"public\". table_three\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with exclusion of a single table");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping tables with a wildcard pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_thre*\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with wildcard in pattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_two/m, "table two not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping table with multiline quoted tablename
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with multiline names requiring quoting");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding multiline quoted tablename from dump
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding an entire schema
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "exclude the public schema");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test including and excluding an entire schema by multiple filterfiles
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema public\n";
+close $inputfile;
+
+open my $alt_inputfile, '>', "$tempdir/inputfile2.txt"
+ or die "unable to open filterfile for writing";
+print $alt_inputfile "exclude schema public\n";
+close $alt_inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "--filter=$tempdir/inputfile2.txt", 'postgres'
+ ],
+ "exclude the public schema with multiple filters");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test dumping a table with a single leading newline on a row
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump nonexisting foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummyserver\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump foreign_data with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE SERVER dummyserver/m, "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude foreign_data dummy*\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/"exclude" foreign data filter is not allowed/,
+ "erroneously exclude foreign server");
+
+#########################################
+# Test broken input format
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: invalid object type specified, should be table, schema, foreign_data or data"
+);
+
+# Test missing object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/missing object name/,
+ "invalid syntax: missing object identifier pattern");
+
+# Test adding extra content after the object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table one";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "invalid syntax: extra content after object identifier pattern");
+
+#########################################
+# Combined with --strict-names
+
+# First ensure that a matching filter works
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ "strict names with matching mattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "no table dumped");
+
+# Now append a pattern to the filter file which doesn't resolve
+open $inputfile, '>>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_nonexisting_name";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "inclusion of non-existing objects with --strict names");
+
+#########################################
+# pg_dumpall tests
+
+###########################
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude database postgres\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ "dump tables with exclusion of a database");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^\\connect postgres/m, "database postgres is not dumped");
+ok($dump =~ qr/^\\connect template1/m, "database template1 is dumped");
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: exclusion of non-existing object type"
+);
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table foo";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/"pg_dumpall" doesn't support filter for object type "table"/,
+ "invalid syntax: exclusion of unsupported object type"
+);
+
+#########################################
+# pg_restore tests
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', "$tempdir/filter_test.dump",
+ "-Fc", 'postgres'
+ ],
+ "dump all tables");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_two";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "wanted table restored");
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "unwanted table is not restored");
+
+#########################################
+# test restore of other objects
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', "$tempdir/filter_test.dump",
+ "-Fc", 'sourcedb'
+ ],
+ "dump all objects from sourcedb");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include function foo1(integer)";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE FUNCTION public\.foo1/m, "wanted function restored");
+ok($dump !~ qr/^CREATE TABLE public\.foo2/m, "unwanted function is not restored");
+
+# this should be white space tolerant (against the -P argument)
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include function foo3 ( double precision , integer) ";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE FUNCTION public\.foo3/m, "wanted function restored");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include index t1_idx1\n";
+
+# attention! this hit pg_restore bug - correct name of trigger is "trg1"
+# not "t1 trg1". Should be fixed when pg_restore will be fixed
+print $inputfile "include trigger t1 trg1\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE INDEX t1_idx1/m, "wanted index restored");
+ok($dump !~ qr/^CREATE INDEX t2_idx2/m, "unwanted index are not restored");
+ok($dump =~ qr/^CREATE TRIGGER trg1/m, "wanted trigger restored");
+ok($dump !~ qr/^CREATE TRIGGER trg2/m, "unwanted trigger is not restored");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema s1\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE s1\.t1/m, "wanted table from schema restored");
+ok($dump =~ qr/^CREATE SEQUENCE s1\.s1/m, "wanted sequence from schema restored");
+ok($dump !~ qr/^CREATE TABLE s2\t2/m, "unwanted table is not restored");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema s1\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !=~ qr/^CREATE TABLE s1\.t1/m, "unwanted table from schema is not restored");
+ok($dump !=~ qr/^CREATE SEQUENCE s1\.s1/m, "unwanted sequence from schema is not restored");
+ok($dump =~ qr/^CREATE TABLE s2\.t2/m, "wanted table restored");
+ok($dump =~ qr/^CREATE TABLE public\.t1/m, "wanted table restored");
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 83a3e40425..d3c7292203 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -451,6 +451,7 @@ sub mkvcbuild
$pgdumpall->AddIncludeDir('src/backend');
$pgdumpall->AddFile('src/bin/pg_dump/pg_dumpall.c');
$pgdumpall->AddFile('src/bin/pg_dump/dumputils.c');
+ $pgdumpall->AddFile('src/bin/pg_dump/filter.c');
$pgdumpall->AddLibrary('ws2_32.lib');
my $pgrestore = AddSimpleFrontend('pg_dump', 1);
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-11 08:11 Julien Rouhaud <[email protected]>
parent: Pavel Stehule <[email protected]>
0 siblings, 1 reply; 181+ messages in thread
From: Julien Rouhaud @ 2022-11-11 08:11 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi,
On Sat, Nov 05, 2022 at 08:54:57PM +0100, Pavel Stehule wrote:
>
> pá 4. 11. 2022 v 14:28 odesílatel Julien Rouhaud <[email protected]>
> napsal:
>
> > > But one thing I noticed is that "optarg" looks wrong here:
> > >
> > > simple_string_list_append(&opts->triggerNames, optarg);
> >
> > Ah indeed, good catch! Maybe there should be an explicit test for every
> > (include|exclude) / objtype combination? It would be a bit verbose (and
> > possibly hard to maintain).
> >
>
> yes - pg_restore is not well covered by tests, fixed
>
> I found another issue. The pg_restore requires a full signature of the
> function and it is pretty sensitive on white spaces (pg_restore).
Argh, indeed. It's a good thing to have expanded the regression tests :)
> I made a
> mistake when I partially parsed patterns like SQL identifiers. It can work
> for simple cases, but when I parse the function's signature it stops
> working. So I rewrote the parsing pattern part. Now, I just read an input
> string and I try to reduce spaces. Still multiline identifiers are
> supported. Against the previous method of pattern parsing, I needed to
> change just one regress test - now I am not able to detect garbage after
> pattern :-/.
I'm not sure it's really problematic. It looks POLA-violation compatible with
regular pg_dump options, for instance:
$ echo "include table t1()" | pg_dump --filter - | ag CREATE
CREATE TABLE public.t1 (
$ pg_dump -t "t1()" | ag CREATE
CREATE TABLE public.t1 (
$ echo "include table t1()blabla" | pg_dump --filter - | ag CREATE
pg_dump: error: no matching tables were found
$ pg_dump -t "t1()blabla" | ag CREATE
pg_dump: error: no matching tables were found
I don't think the file parsing code should try to be smart about checking the
found patterns.
> * function's filtering doesn't support schema - when the name of function
> is specified with schema, then the function is not found
Ah I didn't know that. Indeed it only expect a non-qualified identifier, and
would restore any function that matches the name (and arguments), possibly
multiple ones if there are variants in different schema. That's unrelated to
this patch though.
> * the function has to be specified with an argument type list - the
> separator has to be exactly ", " string. Without space or with one space
> more, the filtering doesn't work (new implementation of pattern parsing
> reduces white spaces sensitivity). This is not a bug, but it is not well
> documented.
Agreed.
> attached updated patch
It looks overall good to me! I just have a few minor nitpicking complaints:
- you removed the pg_strip_clrf() calls and declared everything as "const char
*", so there's no need to explicitly cast the filter_get_keyword() arguments
anymore
Note also that the code now relies on the fact that there are some non-zero
bytes after a pattern to know that no errors happened. It's not a problem as
you should find an EOF marker anyway if CLRF were stripped.
+ * Following routines are called from pg_dump, pg_dumpall and pg_restore.
+ * Unfortunately, implementation of exit_nicely in pg_dump and pg_restore
+ * is different from implementation of this routine in pg_dumpall. So instead
+ * of directly calling exit_nicely we have to return some error flag (in this
+ * case NULL), and exit_nicelly will be executed from caller's routine.
Slight improvement:
[...]
Unfortunately, the implementation of exit_nicely in pg_dump and pg_restore is
different from the one in pg_dumpall, so instead of...
+ * read_pattern - reads an pattern from input. The pattern can be mix of
+ * single line or multi line subpatterns. Single line subpattern starts first
+ * non white space char, and ending last non space char on line or by char
+ * '#'. The white spaces inside are removed (around char ".()"), or reformated
+ * around char ',' or reduced (the multiple spaces are replaced by one).
+ * Multiline subpattern starts by double quote and ending by this char too.
+ * The escape rules are same like for SQL quoted literal.
+ *
+ * Routine signalizes error by returning NULL. Otherwise returns pointer
+ * to next char after last processed char in input string.
typo: reads "a" pattern from input...
I don't fully understand the part about subpatterns, but is that necessary to
describe it? Simply saying that any valid and possibly-quoted identifier can
be parsed should make it clear that identifiers containing \n characters should
work too. Maybe also just mention that whitespaces are removed and special
care is taken to output routines in exactly the same way calling code will
expect it (that is comma-and-single-space type delimiter).
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-12 20:35 Pavel Stehule <[email protected]>
parent: Julien Rouhaud <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2022-11-12 20:35 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
pá 11. 11. 2022 v 9:11 odesílatel Julien Rouhaud <[email protected]>
napsal:
> Hi,
>
> On Sat, Nov 05, 2022 at 08:54:57PM +0100, Pavel Stehule wrote:
> >
> > pá 4. 11. 2022 v 14:28 odesílatel Julien Rouhaud <[email protected]>
> > napsal:
> >
> > > > But one thing I noticed is that "optarg" looks wrong here:
> > > >
> > > > simple_string_list_append(&opts->triggerNames, optarg);
> > >
> > > Ah indeed, good catch! Maybe there should be an explicit test for
> every
> > > (include|exclude) / objtype combination? It would be a bit verbose
> (and
> > > possibly hard to maintain).
> > >
> >
> > yes - pg_restore is not well covered by tests, fixed
> >
> > I found another issue. The pg_restore requires a full signature of the
> > function and it is pretty sensitive on white spaces (pg_restore).
>
> Argh, indeed. It's a good thing to have expanded the regression tests :)
>
> > I made a
> > mistake when I partially parsed patterns like SQL identifiers. It can
> work
> > for simple cases, but when I parse the function's signature it stops
> > working. So I rewrote the parsing pattern part. Now, I just read an input
> > string and I try to reduce spaces. Still multiline identifiers are
> > supported. Against the previous method of pattern parsing, I needed to
> > change just one regress test - now I am not able to detect garbage after
> > pattern :-/.
>
> I'm not sure it's really problematic. It looks POLA-violation compatible
> with
> regular pg_dump options, for instance:
>
> $ echo "include table t1()" | pg_dump --filter - | ag CREATE
> CREATE TABLE public.t1 (
>
> $ pg_dump -t "t1()" | ag CREATE
> CREATE TABLE public.t1 (
>
> $ echo "include table t1()blabla" | pg_dump --filter - | ag CREATE
> pg_dump: error: no matching tables were found
>
> $ pg_dump -t "t1()blabla" | ag CREATE
> pg_dump: error: no matching tables were found
>
> I don't think the file parsing code should try to be smart about checking
> the
> found patterns.
>
> > * function's filtering doesn't support schema - when the name of function
> > is specified with schema, then the function is not found
>
> Ah I didn't know that. Indeed it only expect a non-qualified identifier,
> and
> would restore any function that matches the name (and arguments), possibly
> multiple ones if there are variants in different schema. That's unrelated
> to
> this patch though.
>
> > * the function has to be specified with an argument type list - the
> > separator has to be exactly ", " string. Without space or with one space
> > more, the filtering doesn't work (new implementation of pattern parsing
> > reduces white spaces sensitivity). This is not a bug, but it is not well
> > documented.
>
> Agreed.
>
> > attached updated patch
>
> It looks overall good to me! I just have a few minor nitpicking
> complaints:
>
> - you removed the pg_strip_clrf() calls and declared everything as "const
> char
> *", so there's no need to explicitly cast the filter_get_keyword()
> arguments
> anymore
>
removed
>
> Note also that the code now relies on the fact that there are some non-zero
> bytes after a pattern to know that no errors happened. It's not a problem
> as
> you should find an EOF marker anyway if CLRF were stripped.
>
I am not sure if I understand this note well?
>
> + * Following routines are called from pg_dump, pg_dumpall and pg_restore.
> + * Unfortunately, implementation of exit_nicely in pg_dump and pg_restore
> + * is different from implementation of this routine in pg_dumpall. So
> instead
> + * of directly calling exit_nicely we have to return some error flag (in
> this
> + * case NULL), and exit_nicelly will be executed from caller's routine.
>
> Slight improvement:
> [...]
> Unfortunately, the implementation of exit_nicely in pg_dump and pg_restore
> is
> different from the one in pg_dumpall, so instead of...
>
> + * read_pattern - reads an pattern from input. The pattern can be mix of
> + * single line or multi line subpatterns. Single line subpattern starts
> first
> + * non white space char, and ending last non space char on line or by char
> + * '#'. The white spaces inside are removed (around char ".()"), or
> reformated
> + * around char ',' or reduced (the multiple spaces are replaced by one).
> + * Multiline subpattern starts by double quote and ending by this char
> too.
> + * The escape rules are same like for SQL quoted literal.
> + *
> + * Routine signalizes error by returning NULL. Otherwise returns pointer
> + * to next char after last processed char in input string.
>
>
> typo: reads "a" pattern from input...
>
fixed
>
> I don't fully understand the part about subpatterns, but is that necessary
> to
> describe it? Simply saying that any valid and possibly-quoted identifier
> can
> be parsed should make it clear that identifiers containing \n characters
> should
> work too. Maybe also just mention that whitespaces are removed and special
> care is taken to output routines in exactly the same way calling code will
> expect it (that is comma-and-single-space type delimiter).
>
In this case I hit the limits of my English language skills.
I rewrote this comment, but it needs more care. Please, can you look at it?
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-12 20:37 Pavel Stehule <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2022-11-12 20:37 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
and updated patch
Regards
Pavel
Attachments:
[text/x-patch] pg_dump-filter-20221112.patch (55.5K, ../../CAFj8pRDtKo5zGGpPeKe1v-jK8bhoF35CwFojRf_uvwFDjbdiYw@mail.gmail.com/3-pg_dump-filter-20221112.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 8b9d9f4cad..d5a6e2c7ee 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | table_data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>table_data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index e62d05e5ab..9cad26bbe6 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpretted according to the same rules
+ as <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for excluding databases,
+ and can also be specified more than once for multiple filter files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 47bd7dbda0..ffeb564c52 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules as <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 9dc5a784dd..700e1400c8 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -26,6 +26,7 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filter.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -43,8 +44,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/filter.c b/src/bin/pg_dump/filter.c
new file mode 100644
index 0000000000..830c9b2978
--- /dev/null
+++ b/src/bin/pg_dump/filter.c
@@ -0,0 +1,482 @@
+/*-------------------------------------------------------------------------
+ *
+ * Implementation of simple filter file parser
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include "common/fe_memutils.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "filter.h"
+#include "lib/stringinfo.h"
+#include "pqexpbuffer.h"
+
+#define is_keyword_str(cstr, str, bytes) \
+ ((strlen(cstr) == (bytes)) && (pg_strncasecmp((cstr), (str), (bytes)) == 0))
+
+/*
+ * Following routines are called from pg_dump, pg_dumpall and pg_restore.
+ * Unfortunately, the implementation of exit_nicely in pg_dump and pg_restore is
+ * different from the one in pg_dumpall, so instead of calling exit_nicely we
+ * have to return some error flag (in this case NULL), and exit_nicelly will be
+ * executed from caller's routine.
+ */
+
+/*
+ * Opens filter's file and initialize fstate structure.
+ * Returns true on success.
+ */
+bool
+filter_init(FilterStateData *fstate, const char *filename)
+{
+ fstate->filename = filename;
+ fstate->lineno = 0;
+ initStringInfo(&fstate->linebuff);
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate->fp = fopen(filename, "r");
+ if (!fstate->fp)
+ {
+ pg_log_error("could not open filter file \"%s\": %m", filename);
+ return false;
+ }
+ }
+ else
+ fstate->fp = stdin;
+
+ fstate->is_error = false;
+
+ return true;
+}
+
+/*
+ * Release allocated resources for the given filter.
+ */
+void
+filter_free(FilterStateData *fstate)
+{
+ free(fstate->linebuff.data);
+ fstate->linebuff.data = NULL;
+
+ if (fstate->fp && fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ pg_log_error("could not close filter file \"%s\": %m", fstate->filename);
+
+ fstate->fp = NULL;
+ }
+}
+
+/*
+ * Translate FilterObjectType enum to string. It is designed for formatting
+ * of error message in log_unsupported_filter_object_type routine.
+ */
+static const char *
+filter_object_type_name(FilterObjectType fot)
+{
+ switch (fot)
+ {
+ case FILTER_OBJECT_TYPE_NONE:
+ return "comment or empty line";
+ case FILTER_OBJECT_TYPE_TABLE_DATA:
+ return "table data";
+ case FILTER_OBJECT_TYPE_DATABASE:
+ return "database";
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ return "foreign data";
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ return "function";
+ case FILTER_OBJECT_TYPE_INDEX:
+ return "index";
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ return "schema";
+ case FILTER_OBJECT_TYPE_TABLE:
+ return "table";
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ return "trigger";
+ }
+
+ /* should never get here */
+ pg_unreachable();
+}
+
+/*
+ * Emit error message "invalid format in filter file ..."
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format in filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+ }
+ else
+ pg_log_error("invalid format in filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ fstate->is_error = true;
+}
+
+/*
+ * Emit error message "The application doesn't support filter for object type ..."
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname,
+ FilterObjectType fot)
+{
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str,
+ "\"%s\" doesn't support filter for object type \"%s\".",
+ appname,
+ filter_object_type_name(fot));
+
+ log_invalid_filter_format(fstate, str->data);
+}
+
+/*
+ * filter_get_keyword - read the next filter keyword from buffer
+ *
+ * Search for keywords (limited to ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL when the buffer is empty or first
+ * char is not alpha. The char '_' is allowed too (exclude first position).
+ * The length of the found keyword is returned in the size parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* Set returnlength preemptively in case no keyword is found */
+ *size = 0;
+
+ /* Skip initial whitespace */
+ while (isspace(*ptr))
+ ptr++;
+
+ if (isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isalpha(*ptr) || *ptr == '_')
+ ptr++;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * read_quoted_pattern - read quoted possibly multi lined string.
+ *
+ * Returns pointer to next char after ending double quotes or NULL on error.
+ */
+static const char *
+read_quoted_string(FilterStateData *fstate,
+ const char *str,
+ PQExpBuffer pattern)
+{
+ appendPQExpBufferChar(pattern, '"');
+ str++;
+
+ while (1)
+ {
+ /*
+ * We can ignore \r or \n chars because the string is read by
+ * pg_get_line_buf, so these chars should be just trailing chars.
+ */
+ if (*str == '\r' || *str == '\n')
+ {
+ str++;
+ continue;
+ }
+
+ if (*str == '\0')
+ {
+ Assert(fstate->linebuff.data);
+
+ if (!pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m",
+ fstate->filename);
+ fstate->is_error = true;
+ }
+ else
+ log_invalid_filter_format(fstate, "unexpected end of file");
+
+ return NULL;
+ }
+
+ str = fstate->linebuff.data;
+
+ appendPQExpBufferChar(pattern, '\n');
+ fstate->lineno++;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(pattern, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(pattern, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(pattern, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(pattern, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(pattern, *str++);
+ }
+
+ return str;
+}
+
+/*
+ * read_pattern - reads a pattern from input. Any valid (possibly quoted,
+ * possibly qualified) identifier (or function signature) should be parsed.
+ * An output pattern is reformatted. Initial and trailing spaces are removed.
+ * Inner spaces around chars with special meaning ",.()" are removed too.
+ * Other inner multi-spaces are replaced by single-space. The single-space is
+ * inserted after any comma char.
+ *
+ * Described reformatting is necessary, because backup filtering routines
+ * are sensitive on white spaces and strictly requires expected format.
+ *
+ * Routine signalizes error by returning NULL. Otherwise returns pointer
+ * to next char after last processed char in input string.
+ */
+static const char *
+read_pattern(FilterStateData *fstate, const char *str, PQExpBuffer pattern)
+{
+ bool skip_space = true;
+ bool found_space = false;
+
+ /* Skip initial whitespace */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ {
+ log_invalid_filter_format(fstate, "missing object name pattern");
+ return NULL;
+ }
+
+ while (*str && *str != '#')
+ {
+ while (*str && !isspace(*str) && !strchr("#,.()\"", *str))
+ {
+ /*
+ * Append space only when it is allowed, and when it was found
+ * in original string.
+ */
+ if (!skip_space && found_space)
+ {
+ appendPQExpBufferChar(pattern, ' ');
+ skip_space = true;
+ }
+
+ appendPQExpBufferChar(pattern, *str++);
+ }
+
+ skip_space = false;
+
+ if (*str == '"')
+ {
+ if (found_space)
+ appendPQExpBufferChar(pattern, ' ');
+
+ str = read_quoted_string(fstate, str, pattern);
+ if (!str)
+ return NULL;
+ }
+ else if (*str == ',')
+ {
+ appendPQExpBufferStr(pattern, ", ");
+ skip_space = true;
+ str++;
+ }
+ else if (strchr(".()", *str))
+ {
+ appendPQExpBufferChar(pattern, *str++);
+ skip_space = true;
+ }
+
+ found_space = false;
+
+ /* skip ending whitespaces */
+ while (isspace(*str))
+ {
+ found_space = true;
+ str++;
+ }
+ }
+
+ return str;
+}
+
+/*
+ * filter_read_item - Read command/type/pattern triplet from a filter file
+ *
+ * This will parse one filter item from the filter file, and while it is a
+ * row based format a pattern may span more than one line due to how object
+ * names can be constructed. The expected format of the filter file is:
+ *
+ * <command> <object_type> <pattern>
+ *
+ * command can be "include" or "exclude"
+ * object_type can one of: "table", "schema", "foreign_data", "table_data",
+ * "database", "function", "trigger" or "index"
+ * pattern can be any possibly-quoted and possibly-qualified identifier. It
+ * follows the same rules as other object include and exclude functions so it
+ * can also use wildcards.
+ *
+ * Returns true when one filter item was successfully read and parsed. When
+ * object name contains \n chars, then more than one line from input file can
+ * be processed. Returns false when the filter file reaches EOF. In case of
+ * error, the function will emit an appropriate error message before returning
+ * false.
+ */
+bool
+filter_read_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ Assert(!fstate->is_error);
+
+ if (pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ const char *str = fstate->linebuff.data;
+ const char *keyword;
+ int size;
+ PQExpBuffer pattern;
+
+ fstate->lineno++;
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Skip empty lines or lines where the first non-whitespace character
+ * is a hash indicating a comment.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ /*
+ * First we expect sequence of two keywords, {include|exclude}
+ * followed by the object type to operate on.
+ */
+ keyword = filter_get_keyword(&str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate,
+ "no filter command found (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ if (is_keyword_str("include", keyword, size))
+ *is_include = true;
+ else if (is_keyword_str("exclude", keyword, size))
+ *is_include = false;
+ else
+ {
+ log_invalid_filter_format(fstate,
+ "invalid filter command (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ keyword = filter_get_keyword(&str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate, "missing filter object type");
+ return false;
+ }
+
+ if (is_keyword_str("table_data", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE_DATA;
+ else if (is_keyword_str("database", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_DATABASE;
+ else if (is_keyword_str("foreign_data",keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (is_keyword_str("function", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FUNCTION;
+ else if (is_keyword_str("index", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_INDEX;
+ else if (is_keyword_str("schema", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (is_keyword_str("table", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (is_keyword_str("trigger", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TRIGGER;
+ else
+ {
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str, "unsupported filter object type: \"%.*s\"", size, keyword);
+ log_invalid_filter_format(fstate, str->data);
+ return false;
+ }
+
+ pattern = createPQExpBuffer();
+
+ str = read_pattern(fstate, str, pattern);
+ if (!str)
+ return false;
+
+ *objname = pattern->data;
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m", fstate->filename);
+ fstate->is_error = true;
+ }
+
+ return false;
+}
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..251763ad8e
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,54 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+
+#include "lib/stringinfo.h"
+
+/*
+ * State data for reading filter items from stream
+ */
+typedef struct
+{
+ FILE *fp;
+ const char *filename;
+ int lineno;
+ StringInfoData linebuff;
+ bool is_error;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern bool filter_init(FilterStateData *fstate, const char *filename);
+extern void filter_free(FilterStateData *fstate);
+
+extern void log_invalid_filter_format(FilterStateData *fstate, char *message);
+extern void log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname, FilterObjectType fot);
+extern bool filter_read_item(FilterStateData *fstate, bool *is_include, char **objname, FilterObjectType *objtype);
+
+#endif
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index e66f632b54..c099d66450 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -1,6 +1,7 @@
pg_dump_common_sources = files(
'compress_io.c',
'dumputils.c',
+ 'filter.c',
'parallel.c',
'pg_backup_archiver.c',
'pg_backup_custom.c',
@@ -86,6 +87,7 @@ tests += {
't/001_basic.pl',
't/002_pg_dump.pl',
't/003_pg_dump_with_server.pl',
+ 't/004_pg_dump_filterfile.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index da427f4d4a..a231dc2bb3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -59,6 +59,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -318,6 +319,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AH);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void read_dump_filters(const char *filename, DumpOptions *dopt);
int
@@ -390,6 +392,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -623,6 +626,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ read_dump_filters(optarg, &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1028,6 +1035,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18198,3 +18207,89 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+/*
+ * read_dump_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_dump_filters(const char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments and empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_TABLE_DATA)
+ {
+ if (is_include)
+ {
+ log_invalid_filter_format(&fstate,
+ "\"include\" table data filter is not allowed");
+ break;
+ }
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" foreign data filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dump", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 083012ca39..4597ac3822 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -26,6 +26,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -81,6 +82,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void read_dumpall_filters(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -158,6 +160,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -360,6 +363,10 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ read_dumpall_filters(optarg, &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -653,6 +660,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1890,7 +1898,6 @@ executeCommand(PGconn *conn, const char *query)
PQclear(res);
}
-
/*
* dumpTimestamp
*/
@@ -1914,3 +1921,55 @@ hash_string_pointer(char *s)
return hash_bytes(ss, strlen(s));
}
+
+/*
+ * read_dumpall_filters - retrieve database identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ *
+ * In this moment only excluded databases can be filtered.
+ */
+static void
+read_dumpall_filters(const char *filename, SimpleStringList *pattern)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATABASE)
+ {
+ if (!is_include)
+ simple_string_list_append(pattern, objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"include\" database filter is not allowed");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dumpall", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..1ebf573ac4 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,11 +47,13 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
static void usage(const char *progname);
+static void read_restore_filters(const char *filename, RestoreOptions *dopt);
int
main(int argc, char **argv)
@@ -123,6 +125,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +289,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ read_restore_filters(optarg, opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +470,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +502,109 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * read_restore_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_restore_filters(const char *filename, RestoreOptions *opts)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments or empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_FUNCTION)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" function filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_INDEX)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" type filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ simple_string_list_append(&opts->schemaNames, objname);
+ else
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" table filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TRIGGER)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" trigger filter is not allowed");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_restore", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..a1b06845a3
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,643 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 89;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;;
+my $inputfile;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres',
+ 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three_one(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres',
+ "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three_one VALUES('*** TABLE THREE_ONE ***')");
+
+$node->safe_psql('postgres', "CREATE DATABASE sourcedb");
+$node->safe_psql('postgres', "CREATE DATABASE targetdb");
+
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo1(a int) RETURNS int AS $$ select $1 $$ LANGUAGE sql');
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo2(a int) RETURNS int AS $$ select $1 $$ LANGUAGE sql');
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo3(a double precision, b int) RETURNS double precision AS $$ select $1 + $2 $$ LANGUAGE sql');
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo_trg() RETURNS trigger AS $$ BEGIN RETURN NEW; END $$ LANGUAGE plpgsql');
+$node->safe_psql('sourcedb', 'CREATE SCHEMA s1');
+$node->safe_psql('sourcedb', 'CREATE SCHEMA s2');
+$node->safe_psql('sourcedb', 'CREATE TABLE s1.t1(a int)');
+$node->safe_psql('sourcedb', 'CREATE SEQUENCE s1.s1');
+$node->safe_psql('sourcedb', 'CREATE TABLE s2.t2(a int)');
+$node->safe_psql('sourcedb', 'CREATE TABLE t1(a int, b int)');
+$node->safe_psql('sourcedb', 'CREATE TABLE t2(a int, b int)');
+$node->safe_psql('sourcedb', 'CREATE INDEX t1_idx1 ON t1(a)');
+$node->safe_psql('sourcedb', 'CREATE INDEX t1_idx2 ON t1(b)');
+$node->safe_psql('sourcedb', 'CREATE TRIGGER trg1 BEFORE INSERT ON t1 EXECUTE FUNCTION foo_trg()');
+$node->safe_psql('sourcedb', 'CREATE TRIGGER trg2 BEFORE INSERT ON t1 EXECUTE FUNCTION foo_trg()');
+
+#
+# Test interaction of correctly specified filter file
+#
+my ($cmd, $stdout, $stderr, $result);
+
+# Empty filterfile
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "\n # a comment and nothing more\n\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "table one dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "table two dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "table three dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m, "table three one dumped");
+
+# Test various combinations of whitespace, comments and correct filters
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "\t\n";
+print $inputfile " \t# another comment\n";
+print $inputfile "exclude table_data table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter patterns as well as comments and whitespace");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public\.table_three/m, "table three not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_three_one/m,
+ "table three_one not dumped");
+ok( $dump !~ qr/^COPY public\.table_one/m,
+ "content of table one is not included");
+ok($dump =~ qr/^COPY public\.table_two/m, "content of table two is included");
+
+# Test dumping tables specified by qualified names
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table public.table_one\n";
+print $inputfile "include table \"public\".\"table_two\"\n";
+print $inputfile "include table \"public\". table_three\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with exclusion of a single table");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping tables with a wildcard pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_thre*\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with wildcard in pattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_two/m, "table two not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping table with multiline quoted tablename
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with multiline names requiring quoting");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding multiline quoted tablename from dump
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding an entire schema
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "exclude the public schema");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test including and excluding an entire schema by multiple filterfiles
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema public\n";
+close $inputfile;
+
+open my $alt_inputfile, '>', "$tempdir/inputfile2.txt"
+ or die "unable to open filterfile for writing";
+print $alt_inputfile "exclude schema public\n";
+close $alt_inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "--filter=$tempdir/inputfile2.txt", 'postgres'
+ ],
+ "exclude the public schema with multiple filters");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test dumping a table with a single leading newline on a row
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump nonexisting foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummyserver\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump foreign_data with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE SERVER dummyserver/m, "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude foreign_data dummy*\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/"exclude" foreign data filter is not allowed/,
+ "erroneously exclude foreign server");
+
+#########################################
+# Test broken input format
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: invalid object type specified, should be table, schema, foreign_data or data"
+);
+
+# Test missing object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/missing object name/,
+ "invalid syntax: missing object identifier pattern");
+
+# Test adding extra content after the object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table one";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "invalid syntax: extra content after object identifier pattern");
+
+#########################################
+# Combined with --strict-names
+
+# First ensure that a matching filter works
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ "strict names with matching mattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "no table dumped");
+
+# Now append a pattern to the filter file which doesn't resolve
+open $inputfile, '>>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_nonexisting_name";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "inclusion of non-existing objects with --strict names");
+
+#########################################
+# pg_dumpall tests
+
+###########################
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude database postgres\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ "dump tables with exclusion of a database");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^\\connect postgres/m, "database postgres is not dumped");
+ok($dump =~ qr/^\\connect template1/m, "database template1 is dumped");
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: exclusion of non-existing object type"
+);
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table foo";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/"pg_dumpall" doesn't support filter for object type "table"/,
+ "invalid syntax: exclusion of unsupported object type"
+);
+
+#########################################
+# pg_restore tests
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', "$tempdir/filter_test.dump",
+ "-Fc", 'postgres'
+ ],
+ "dump all tables");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_two";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "wanted table restored");
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "unwanted table is not restored");
+
+#########################################
+# test restore of other objects
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', "$tempdir/filter_test.dump",
+ "-Fc", 'sourcedb'
+ ],
+ "dump all objects from sourcedb");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include function foo1(integer)";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE FUNCTION public\.foo1/m, "wanted function restored");
+ok($dump !~ qr/^CREATE TABLE public\.foo2/m, "unwanted function is not restored");
+
+# this should be white space tolerant (against the -P argument)
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include function foo3 ( double precision , integer) ";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE FUNCTION public\.foo3/m, "wanted function restored");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include index t1_idx1\n";
+
+# attention! this hit pg_restore bug - correct name of trigger is "trg1"
+# not "t1 trg1". Should be fixed when pg_restore will be fixed
+print $inputfile "include trigger t1 trg1\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE INDEX t1_idx1/m, "wanted index restored");
+ok($dump !~ qr/^CREATE INDEX t2_idx2/m, "unwanted index are not restored");
+ok($dump =~ qr/^CREATE TRIGGER trg1/m, "wanted trigger restored");
+ok($dump !~ qr/^CREATE TRIGGER trg2/m, "unwanted trigger is not restored");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema s1\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE s1\.t1/m, "wanted table from schema restored");
+ok($dump =~ qr/^CREATE SEQUENCE s1\.s1/m, "wanted sequence from schema restored");
+ok($dump !~ qr/^CREATE TABLE s2\t2/m, "unwanted table is not restored");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema s1\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !=~ qr/^CREATE TABLE s1\.t1/m, "unwanted table from schema is not restored");
+ok($dump !=~ qr/^CREATE SEQUENCE s1\.s1/m, "unwanted sequence from schema is not restored");
+ok($dump =~ qr/^CREATE TABLE s2\.t2/m, "wanted table restored");
+ok($dump =~ qr/^CREATE TABLE public\.t1/m, "wanted table restored");
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 83a3e40425..d3c7292203 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -451,6 +451,7 @@ sub mkvcbuild
$pgdumpall->AddIncludeDir('src/backend');
$pgdumpall->AddFile('src/bin/pg_dump/pg_dumpall.c');
$pgdumpall->AddFile('src/bin/pg_dump/dumputils.c');
+ $pgdumpall->AddFile('src/bin/pg_dump/filter.c');
$pgdumpall->AddLibrary('ws2_32.lib');
my $pgrestore = AddSimpleFrontend('pg_dump', 1);
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-13 08:58 Julien Rouhaud <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Julien Rouhaud @ 2022-11-13 08:58 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
On Sat, Nov 12, 2022 at 09:35:59PM +0100, Pavel Stehule wrote:
Thanks for the updated patch. Apart from the function comment it looks good to
me.
Justin, did you have any other comment on the patch?
> > I don't fully understand the part about subpatterns, but is that necessary
> > to
> > describe it? Simply saying that any valid and possibly-quoted identifier
> > can
> > be parsed should make it clear that identifiers containing \n characters
> > should
> > work too. Maybe also just mention that whitespaces are removed and special
> > care is taken to output routines in exactly the same way calling code will
> > expect it (that is comma-and-single-space type delimiter).
> >
>
> In this case I hit the limits of my English language skills.
>
> I rewrote this comment, but it needs more care. Please, can you look at it?
I'm also not a native English speaker so I'm far for writing perfect comments
myself :)
Maybe something like
/*
* read_pattern - reads on object pattern from input
*
* This function will parse any valid identifier (quoted or not, qualified or
* not), which can also includes the full signature for routines.
* Note that this function takes special care to sanitize the detected
* identifier (removing extraneous whitespaces or other unnecessary
* characters). This is necessary as most backup/restore filtering functions
* only recognize identifiers if they are written exactly way as they are
* regenerated.
* Returns a pointer to next character after the found identifier, or NULL on
* error.
*/
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-13 19:32 Pavel Stehule <[email protected]>
parent: Julien Rouhaud <[email protected]>
0 siblings, 2 replies; 181+ messages in thread
From: Pavel Stehule @ 2022-11-13 19:32 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
ne 13. 11. 2022 v 9:58 odesílatel Julien Rouhaud <[email protected]>
napsal:
> On Sat, Nov 12, 2022 at 09:35:59PM +0100, Pavel Stehule wrote:
>
> Thanks for the updated patch. Apart from the function comment it looks
> good to
> me.
>
> Justin, did you have any other comment on the patch?
>
> > > I don't fully understand the part about subpatterns, but is that
> necessary
> > > to
> > > describe it? Simply saying that any valid and possibly-quoted
> identifier
> > > can
> > > be parsed should make it clear that identifiers containing \n
> characters
> > > should
> > > work too. Maybe also just mention that whitespaces are removed and
> special
> > > care is taken to output routines in exactly the same way calling code
> will
> > > expect it (that is comma-and-single-space type delimiter).
> > >
> >
> > In this case I hit the limits of my English language skills.
> >
> > I rewrote this comment, but it needs more care. Please, can you look at
> it?
>
> I'm also not a native English speaker so I'm far for writing perfect
> comments
> myself :)
>
far better than mine :)
Thank you very much
updated patch attached
Regards
Pavel
>
> Maybe something like
>
> /*
> * read_pattern - reads on object pattern from input
> *
> * This function will parse any valid identifier (quoted or not, qualified
> or
> * not), which can also includes the full signature for routines.
> * Note that this function takes special care to sanitize the detected
> * identifier (removing extraneous whitespaces or other unnecessary
> * characters). This is necessary as most backup/restore filtering
> functions
> * only recognize identifiers if they are written exactly way as they are
> * regenerated.
> * Returns a pointer to next character after the found identifier, or NULL
> on
> * error.
> */
>
Attachments:
[text/x-patch] pg_dump-filter-20221113.patch (55.4K, ../../CAFj8pRA_1rroP3TvM7xnpUb=U4Z085ZHL6HC5TjaETnj0ttirA@mail.gmail.com/3-pg_dump-filter-20221113.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 8b9d9f4cad..d5a6e2c7ee 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | table_data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>table_data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index e62d05e5ab..9cad26bbe6 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpretted according to the same rules
+ as <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for excluding databases,
+ and can also be specified more than once for multiple filter files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 47bd7dbda0..ffeb564c52 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules as <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 9dc5a784dd..700e1400c8 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -26,6 +26,7 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filter.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -43,8 +44,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/filter.c b/src/bin/pg_dump/filter.c
new file mode 100644
index 0000000000..3998312b33
--- /dev/null
+++ b/src/bin/pg_dump/filter.c
@@ -0,0 +1,481 @@
+/*-------------------------------------------------------------------------
+ *
+ * Implementation of simple filter file parser
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include "common/fe_memutils.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "filter.h"
+#include "lib/stringinfo.h"
+#include "pqexpbuffer.h"
+
+#define is_keyword_str(cstr, str, bytes) \
+ ((strlen(cstr) == (bytes)) && (pg_strncasecmp((cstr), (str), (bytes)) == 0))
+
+/*
+ * Following routines are called from pg_dump, pg_dumpall and pg_restore.
+ * Unfortunately, the implementation of exit_nicely in pg_dump and pg_restore is
+ * different from the one in pg_dumpall, so instead of calling exit_nicely we
+ * have to return some error flag (in this case NULL), and exit_nicelly will be
+ * executed from caller's routine.
+ */
+
+/*
+ * Opens filter's file and initialize fstate structure.
+ * Returns true on success.
+ */
+bool
+filter_init(FilterStateData *fstate, const char *filename)
+{
+ fstate->filename = filename;
+ fstate->lineno = 0;
+ initStringInfo(&fstate->linebuff);
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate->fp = fopen(filename, "r");
+ if (!fstate->fp)
+ {
+ pg_log_error("could not open filter file \"%s\": %m", filename);
+ return false;
+ }
+ }
+ else
+ fstate->fp = stdin;
+
+ fstate->is_error = false;
+
+ return true;
+}
+
+/*
+ * Release allocated resources for the given filter.
+ */
+void
+filter_free(FilterStateData *fstate)
+{
+ free(fstate->linebuff.data);
+ fstate->linebuff.data = NULL;
+
+ if (fstate->fp && fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ pg_log_error("could not close filter file \"%s\": %m", fstate->filename);
+
+ fstate->fp = NULL;
+ }
+}
+
+/*
+ * Translate FilterObjectType enum to string. It is designed for formatting
+ * of error message in log_unsupported_filter_object_type routine.
+ */
+static const char *
+filter_object_type_name(FilterObjectType fot)
+{
+ switch (fot)
+ {
+ case FILTER_OBJECT_TYPE_NONE:
+ return "comment or empty line";
+ case FILTER_OBJECT_TYPE_TABLE_DATA:
+ return "table data";
+ case FILTER_OBJECT_TYPE_DATABASE:
+ return "database";
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ return "foreign data";
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ return "function";
+ case FILTER_OBJECT_TYPE_INDEX:
+ return "index";
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ return "schema";
+ case FILTER_OBJECT_TYPE_TABLE:
+ return "table";
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ return "trigger";
+ }
+
+ /* should never get here */
+ pg_unreachable();
+}
+
+/*
+ * Emit error message "invalid format in filter file ..."
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format in filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+ }
+ else
+ pg_log_error("invalid format in filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ fstate->is_error = true;
+}
+
+/*
+ * Emit error message "The application doesn't support filter for object type ..."
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname,
+ FilterObjectType fot)
+{
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str,
+ "\"%s\" doesn't support filter for object type \"%s\".",
+ appname,
+ filter_object_type_name(fot));
+
+ log_invalid_filter_format(fstate, str->data);
+}
+
+/*
+ * filter_get_keyword - read the next filter keyword from buffer
+ *
+ * Search for keywords (limited to ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL when the buffer is empty or first
+ * char is not alpha. The char '_' is allowed too (exclude first position).
+ * The length of the found keyword is returned in the size parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* Set returnlength preemptively in case no keyword is found */
+ *size = 0;
+
+ /* Skip initial whitespace */
+ while (isspace(*ptr))
+ ptr++;
+
+ if (isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isalpha(*ptr) || *ptr == '_')
+ ptr++;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * read_quoted_pattern - read quoted possibly multi lined string.
+ *
+ * Returns pointer to next char after ending double quotes or NULL on error.
+ */
+static const char *
+read_quoted_string(FilterStateData *fstate,
+ const char *str,
+ PQExpBuffer pattern)
+{
+ appendPQExpBufferChar(pattern, '"');
+ str++;
+
+ while (1)
+ {
+ /*
+ * We can ignore \r or \n chars because the string is read by
+ * pg_get_line_buf, so these chars should be just trailing chars.
+ */
+ if (*str == '\r' || *str == '\n')
+ {
+ str++;
+ continue;
+ }
+
+ if (*str == '\0')
+ {
+ Assert(fstate->linebuff.data);
+
+ if (!pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m",
+ fstate->filename);
+ fstate->is_error = true;
+ }
+ else
+ log_invalid_filter_format(fstate, "unexpected end of file");
+
+ return NULL;
+ }
+
+ str = fstate->linebuff.data;
+
+ appendPQExpBufferChar(pattern, '\n');
+ fstate->lineno++;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(pattern, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(pattern, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(pattern, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(pattern, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(pattern, *str++);
+ }
+
+ return str;
+}
+
+/*
+ * read_pattern - reads on object pattern from input
+ *
+ * This function will parse any valid identifier (quoted or not, qualified or
+ * not), which can also includes the full signature for routines.
+ * Note that this function takes special care to sanitize the detected
+ * identifier (removing extraneous whitespaces or other unnecessary
+ * characters). This is necessary as most backup/restore filtering functions
+ * only recognize identifiers if they are written exactly way as they are
+ * regenerated.
+ * Returns a pointer to next character after the found identifier, or NULL on
+ * error.
+ */
+static const char *
+read_pattern(FilterStateData *fstate, const char *str, PQExpBuffer pattern)
+{
+ bool skip_space = true;
+ bool found_space = false;
+
+ /* Skip initial whitespace */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ {
+ log_invalid_filter_format(fstate, "missing object name pattern");
+ return NULL;
+ }
+
+ while (*str && *str != '#')
+ {
+ while (*str && !isspace(*str) && !strchr("#,.()\"", *str))
+ {
+ /*
+ * Append space only when it is allowed, and when it was found
+ * in original string.
+ */
+ if (!skip_space && found_space)
+ {
+ appendPQExpBufferChar(pattern, ' ');
+ skip_space = true;
+ }
+
+ appendPQExpBufferChar(pattern, *str++);
+ }
+
+ skip_space = false;
+
+ if (*str == '"')
+ {
+ if (found_space)
+ appendPQExpBufferChar(pattern, ' ');
+
+ str = read_quoted_string(fstate, str, pattern);
+ if (!str)
+ return NULL;
+ }
+ else if (*str == ',')
+ {
+ appendPQExpBufferStr(pattern, ", ");
+ skip_space = true;
+ str++;
+ }
+ else if (strchr(".()", *str))
+ {
+ appendPQExpBufferChar(pattern, *str++);
+ skip_space = true;
+ }
+
+ found_space = false;
+
+ /* skip ending whitespaces */
+ while (isspace(*str))
+ {
+ found_space = true;
+ str++;
+ }
+ }
+
+ return str;
+}
+
+/*
+ * filter_read_item - Read command/type/pattern triplet from a filter file
+ *
+ * This will parse one filter item from the filter file, and while it is a
+ * row based format a pattern may span more than one line due to how object
+ * names can be constructed. The expected format of the filter file is:
+ *
+ * <command> <object_type> <pattern>
+ *
+ * command can be "include" or "exclude"
+ * object_type can one of: "table", "schema", "foreign_data", "table_data",
+ * "database", "function", "trigger" or "index"
+ * pattern can be any possibly-quoted and possibly-qualified identifier. It
+ * follows the same rules as other object include and exclude functions so it
+ * can also use wildcards.
+ *
+ * Returns true when one filter item was successfully read and parsed. When
+ * object name contains \n chars, then more than one line from input file can
+ * be processed. Returns false when the filter file reaches EOF. In case of
+ * error, the function will emit an appropriate error message before returning
+ * false.
+ */
+bool
+filter_read_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ Assert(!fstate->is_error);
+
+ if (pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ const char *str = fstate->linebuff.data;
+ const char *keyword;
+ int size;
+ PQExpBuffer pattern;
+
+ fstate->lineno++;
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Skip empty lines or lines where the first non-whitespace character
+ * is a hash indicating a comment.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ /*
+ * First we expect sequence of two keywords, {include|exclude}
+ * followed by the object type to operate on.
+ */
+ keyword = filter_get_keyword(&str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate,
+ "no filter command found (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ if (is_keyword_str("include", keyword, size))
+ *is_include = true;
+ else if (is_keyword_str("exclude", keyword, size))
+ *is_include = false;
+ else
+ {
+ log_invalid_filter_format(fstate,
+ "invalid filter command (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ keyword = filter_get_keyword(&str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate, "missing filter object type");
+ return false;
+ }
+
+ if (is_keyword_str("table_data", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE_DATA;
+ else if (is_keyword_str("database", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_DATABASE;
+ else if (is_keyword_str("foreign_data",keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (is_keyword_str("function", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FUNCTION;
+ else if (is_keyword_str("index", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_INDEX;
+ else if (is_keyword_str("schema", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (is_keyword_str("table", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (is_keyword_str("trigger", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TRIGGER;
+ else
+ {
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str, "unsupported filter object type: \"%.*s\"", size, keyword);
+ log_invalid_filter_format(fstate, str->data);
+ return false;
+ }
+
+ pattern = createPQExpBuffer();
+
+ str = read_pattern(fstate, str, pattern);
+ if (!str)
+ return false;
+
+ *objname = pattern->data;
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m", fstate->filename);
+ fstate->is_error = true;
+ }
+
+ return false;
+}
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..251763ad8e
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,54 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+
+#include "lib/stringinfo.h"
+
+/*
+ * State data for reading filter items from stream
+ */
+typedef struct
+{
+ FILE *fp;
+ const char *filename;
+ int lineno;
+ StringInfoData linebuff;
+ bool is_error;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern bool filter_init(FilterStateData *fstate, const char *filename);
+extern void filter_free(FilterStateData *fstate);
+
+extern void log_invalid_filter_format(FilterStateData *fstate, char *message);
+extern void log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname, FilterObjectType fot);
+extern bool filter_read_item(FilterStateData *fstate, bool *is_include, char **objname, FilterObjectType *objtype);
+
+#endif
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index e66f632b54..c099d66450 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -1,6 +1,7 @@
pg_dump_common_sources = files(
'compress_io.c',
'dumputils.c',
+ 'filter.c',
'parallel.c',
'pg_backup_archiver.c',
'pg_backup_custom.c',
@@ -86,6 +87,7 @@ tests += {
't/001_basic.pl',
't/002_pg_dump.pl',
't/003_pg_dump_with_server.pl',
+ 't/004_pg_dump_filterfile.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index da427f4d4a..a231dc2bb3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -59,6 +59,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -318,6 +319,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AH);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void read_dump_filters(const char *filename, DumpOptions *dopt);
int
@@ -390,6 +392,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -623,6 +626,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ read_dump_filters(optarg, &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1028,6 +1035,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18198,3 +18207,89 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+/*
+ * read_dump_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_dump_filters(const char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments and empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_TABLE_DATA)
+ {
+ if (is_include)
+ {
+ log_invalid_filter_format(&fstate,
+ "\"include\" table data filter is not allowed");
+ break;
+ }
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" foreign data filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dump", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 083012ca39..4597ac3822 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -26,6 +26,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -81,6 +82,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void read_dumpall_filters(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -158,6 +160,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -360,6 +363,10 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ read_dumpall_filters(optarg, &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -653,6 +660,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1890,7 +1898,6 @@ executeCommand(PGconn *conn, const char *query)
PQclear(res);
}
-
/*
* dumpTimestamp
*/
@@ -1914,3 +1921,55 @@ hash_string_pointer(char *s)
return hash_bytes(ss, strlen(s));
}
+
+/*
+ * read_dumpall_filters - retrieve database identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ *
+ * In this moment only excluded databases can be filtered.
+ */
+static void
+read_dumpall_filters(const char *filename, SimpleStringList *pattern)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATABASE)
+ {
+ if (!is_include)
+ simple_string_list_append(pattern, objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"include\" database filter is not allowed");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dumpall", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..1ebf573ac4 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,11 +47,13 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
static void usage(const char *progname);
+static void read_restore_filters(const char *filename, RestoreOptions *dopt);
int
main(int argc, char **argv)
@@ -123,6 +125,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +289,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ read_restore_filters(optarg, opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +470,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +502,109 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * read_restore_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_restore_filters(const char *filename, RestoreOptions *opts)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments or empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_FUNCTION)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" function filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_INDEX)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" type filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ simple_string_list_append(&opts->schemaNames, objname);
+ else
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" table filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TRIGGER)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" trigger filter is not allowed");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_restore", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..a1b06845a3
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,643 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 89;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;;
+my $inputfile;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres',
+ 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three_one(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres',
+ "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three_one VALUES('*** TABLE THREE_ONE ***')");
+
+$node->safe_psql('postgres', "CREATE DATABASE sourcedb");
+$node->safe_psql('postgres', "CREATE DATABASE targetdb");
+
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo1(a int) RETURNS int AS $$ select $1 $$ LANGUAGE sql');
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo2(a int) RETURNS int AS $$ select $1 $$ LANGUAGE sql');
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo3(a double precision, b int) RETURNS double precision AS $$ select $1 + $2 $$ LANGUAGE sql');
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo_trg() RETURNS trigger AS $$ BEGIN RETURN NEW; END $$ LANGUAGE plpgsql');
+$node->safe_psql('sourcedb', 'CREATE SCHEMA s1');
+$node->safe_psql('sourcedb', 'CREATE SCHEMA s2');
+$node->safe_psql('sourcedb', 'CREATE TABLE s1.t1(a int)');
+$node->safe_psql('sourcedb', 'CREATE SEQUENCE s1.s1');
+$node->safe_psql('sourcedb', 'CREATE TABLE s2.t2(a int)');
+$node->safe_psql('sourcedb', 'CREATE TABLE t1(a int, b int)');
+$node->safe_psql('sourcedb', 'CREATE TABLE t2(a int, b int)');
+$node->safe_psql('sourcedb', 'CREATE INDEX t1_idx1 ON t1(a)');
+$node->safe_psql('sourcedb', 'CREATE INDEX t1_idx2 ON t1(b)');
+$node->safe_psql('sourcedb', 'CREATE TRIGGER trg1 BEFORE INSERT ON t1 EXECUTE FUNCTION foo_trg()');
+$node->safe_psql('sourcedb', 'CREATE TRIGGER trg2 BEFORE INSERT ON t1 EXECUTE FUNCTION foo_trg()');
+
+#
+# Test interaction of correctly specified filter file
+#
+my ($cmd, $stdout, $stderr, $result);
+
+# Empty filterfile
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "\n # a comment and nothing more\n\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "table one dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "table two dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "table three dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m, "table three one dumped");
+
+# Test various combinations of whitespace, comments and correct filters
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "\t\n";
+print $inputfile " \t# another comment\n";
+print $inputfile "exclude table_data table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter patterns as well as comments and whitespace");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public\.table_three/m, "table three not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_three_one/m,
+ "table three_one not dumped");
+ok( $dump !~ qr/^COPY public\.table_one/m,
+ "content of table one is not included");
+ok($dump =~ qr/^COPY public\.table_two/m, "content of table two is included");
+
+# Test dumping tables specified by qualified names
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table public.table_one\n";
+print $inputfile "include table \"public\".\"table_two\"\n";
+print $inputfile "include table \"public\". table_three\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with exclusion of a single table");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping tables with a wildcard pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_thre*\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with wildcard in pattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_two/m, "table two not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping table with multiline quoted tablename
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with multiline names requiring quoting");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding multiline quoted tablename from dump
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding an entire schema
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "exclude the public schema");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test including and excluding an entire schema by multiple filterfiles
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema public\n";
+close $inputfile;
+
+open my $alt_inputfile, '>', "$tempdir/inputfile2.txt"
+ or die "unable to open filterfile for writing";
+print $alt_inputfile "exclude schema public\n";
+close $alt_inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "--filter=$tempdir/inputfile2.txt", 'postgres'
+ ],
+ "exclude the public schema with multiple filters");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test dumping a table with a single leading newline on a row
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump nonexisting foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummyserver\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump foreign_data with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE SERVER dummyserver/m, "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude foreign_data dummy*\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/"exclude" foreign data filter is not allowed/,
+ "erroneously exclude foreign server");
+
+#########################################
+# Test broken input format
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: invalid object type specified, should be table, schema, foreign_data or data"
+);
+
+# Test missing object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/missing object name/,
+ "invalid syntax: missing object identifier pattern");
+
+# Test adding extra content after the object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table one";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "invalid syntax: extra content after object identifier pattern");
+
+#########################################
+# Combined with --strict-names
+
+# First ensure that a matching filter works
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ "strict names with matching mattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "no table dumped");
+
+# Now append a pattern to the filter file which doesn't resolve
+open $inputfile, '>>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_nonexisting_name";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "inclusion of non-existing objects with --strict names");
+
+#########################################
+# pg_dumpall tests
+
+###########################
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude database postgres\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ "dump tables with exclusion of a database");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^\\connect postgres/m, "database postgres is not dumped");
+ok($dump =~ qr/^\\connect template1/m, "database template1 is dumped");
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: exclusion of non-existing object type"
+);
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table foo";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/"pg_dumpall" doesn't support filter for object type "table"/,
+ "invalid syntax: exclusion of unsupported object type"
+);
+
+#########################################
+# pg_restore tests
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', "$tempdir/filter_test.dump",
+ "-Fc", 'postgres'
+ ],
+ "dump all tables");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_two";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "wanted table restored");
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "unwanted table is not restored");
+
+#########################################
+# test restore of other objects
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', "$tempdir/filter_test.dump",
+ "-Fc", 'sourcedb'
+ ],
+ "dump all objects from sourcedb");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include function foo1(integer)";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE FUNCTION public\.foo1/m, "wanted function restored");
+ok($dump !~ qr/^CREATE TABLE public\.foo2/m, "unwanted function is not restored");
+
+# this should be white space tolerant (against the -P argument)
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include function foo3 ( double precision , integer) ";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE FUNCTION public\.foo3/m, "wanted function restored");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include index t1_idx1\n";
+
+# attention! this hit pg_restore bug - correct name of trigger is "trg1"
+# not "t1 trg1". Should be fixed when pg_restore will be fixed
+print $inputfile "include trigger t1 trg1\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE INDEX t1_idx1/m, "wanted index restored");
+ok($dump !~ qr/^CREATE INDEX t2_idx2/m, "unwanted index are not restored");
+ok($dump =~ qr/^CREATE TRIGGER trg1/m, "wanted trigger restored");
+ok($dump !~ qr/^CREATE TRIGGER trg2/m, "unwanted trigger is not restored");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema s1\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE s1\.t1/m, "wanted table from schema restored");
+ok($dump =~ qr/^CREATE SEQUENCE s1\.s1/m, "wanted sequence from schema restored");
+ok($dump !~ qr/^CREATE TABLE s2\t2/m, "unwanted table is not restored");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema s1\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !=~ qr/^CREATE TABLE s1\.t1/m, "unwanted table from schema is not restored");
+ok($dump !=~ qr/^CREATE SEQUENCE s1\.s1/m, "unwanted sequence from schema is not restored");
+ok($dump =~ qr/^CREATE TABLE s2\.t2/m, "wanted table restored");
+ok($dump =~ qr/^CREATE TABLE public\.t1/m, "wanted table restored");
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 83a3e40425..d3c7292203 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -451,6 +451,7 @@ sub mkvcbuild
$pgdumpall->AddIncludeDir('src/backend');
$pgdumpall->AddFile('src/bin/pg_dump/pg_dumpall.c');
$pgdumpall->AddFile('src/bin/pg_dump/dumputils.c');
+ $pgdumpall->AddFile('src/bin/pg_dump/filter.c');
$pgdumpall->AddLibrary('ws2_32.lib');
my $pgrestore = AddSimpleFrontend('pg_dump', 1);
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-22 05:25 Julien Rouhaud <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 1 reply; 181+ messages in thread
From: Julien Rouhaud @ 2022-11-22 05:25 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi,
On Sun, Nov 13, 2022 at 08:32:47PM +0100, Pavel Stehule wrote:
>
> updated patch attached
Thanks!
Some enhancement could probably be done by a native english speaker, but apart
from that it looks good to me, so hearing no other complaints I'm marking the
CF entry as Ready for Committer!
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-22 05:30 Pavel Stehule <[email protected]>
parent: Julien Rouhaud <[email protected]>
0 siblings, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2022-11-22 05:30 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; +Cc: Justin Pryzby <[email protected]>; Andres Freund <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
út 22. 11. 2022 v 6:26 odesílatel Julien Rouhaud <[email protected]>
napsal:
> Hi,
>
> On Sun, Nov 13, 2022 at 08:32:47PM +0100, Pavel Stehule wrote:
> >
> > updated patch attached
>
> Thanks!
>
> Some enhancement could probably be done by a native english speaker, but
> apart
> from that it looks good to me, so hearing no other complaints I'm marking
> the
> CF entry as Ready for Committer!
>
Thank you very much for check and help
Regards
Pavel
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-22 07:39 Andres Freund <[email protected]>
parent: Pavel Stehule <[email protected]>
1 sibling, 2 replies; 181+ messages in thread
From: Andres Freund @ 2022-11-22 07:39 UTC (permalink / raw)
To: Pavel Stehule <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Justin Pryzby <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi,
On 2022-11-13 20:32:47 +0100, Pavel Stehule wrote:
> updated patch attached
It fails with address sanitizer that's now part of CI:
https://cirrus-ci.com/task/6031397744279552?logs=test_world#L2659
[06:33:11.271] # ==31965==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x619000000480 at pc 0x559f1ac40822 bp 0x7ffea83e1ad0 sp 0x7ffea83e1ac8
[06:33:11.271] # READ of size 1 at 0x619000000480 thread T0
[06:33:11.271] # #0 0x559f1ac40821 in read_pattern /tmp/cirrus-ci-build/src/bin/pg_dump/filter.c:302
[06:33:11.271] # #1 0x559f1ac40e4d in filter_read_item /tmp/cirrus-ci-build/src/bin/pg_dump/filter.c:459
[06:33:11.271] # #2 0x559f1abe6fa5 in read_dump_filters /tmp/cirrus-ci-build/src/bin/pg_dump/pg_dump.c:18229
[06:33:11.271] # #3 0x559f1ac2bb1b in main /tmp/cirrus-ci-build/src/bin/pg_dump/pg_dump.c:630
[06:33:11.271] # #4 0x7fd91fabfd09 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x23d09)
[06:33:11.271] # #5 0x559f1abe5d29 in _start (/tmp/cirrus-ci-build/tmp_install/usr/local/pgsql/bin/pg_dump+0x39d29)
[06:33:11.271] #
[06:33:11.271] # 0x619000000480 is located 0 bytes to the right of 1024-byte region [0x619000000080,0x619000000480)
[06:33:11.271] # allocated by thread T0 here:
[06:33:11.271] # #0 0x7fd91fe14e8f in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
[06:33:11.271] # #1 0x559f1ac69f35 in pg_malloc_internal /tmp/cirrus-ci-build/src/common/fe_memutils.c:30
[06:33:11.271] # #2 0x559f1ac69f35 in palloc /tmp/cirrus-ci-build/src/common/fe_memutils.c:117
[06:33:11.271] #
[06:33:11.271] # SUMMARY: AddressSanitizer: heap-buffer-overflow /tmp/cirrus-ci-build/src/bin/pg_dump/filter.c:302 in read_pattern
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-22 07:41 Pavel Stehule <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2022-11-22 07:41 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Justin Pryzby <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
út 22. 11. 2022 v 8:39 odesílatel Andres Freund <[email protected]> napsal:
> Hi,
>
> On 2022-11-13 20:32:47 +0100, Pavel Stehule wrote:
> > updated patch attached
>
> It fails with address sanitizer that's now part of CI:
>
> https://cirrus-ci.com/task/6031397744279552?logs=test_world#L2659
>
> [06:33:11.271] # ==31965==ERROR: AddressSanitizer: heap-buffer-overflow on
> address 0x619000000480 at pc 0x559f1ac40822 bp 0x7ffea83e1ad0 sp
> 0x7ffea83e1ac8
> [06:33:11.271] # READ of size 1 at 0x619000000480 thread T0
> [06:33:11.271] # #0 0x559f1ac40821 in read_pattern
> /tmp/cirrus-ci-build/src/bin/pg_dump/filter.c:302
> [06:33:11.271] # #1 0x559f1ac40e4d in filter_read_item
> /tmp/cirrus-ci-build/src/bin/pg_dump/filter.c:459
> [06:33:11.271] # #2 0x559f1abe6fa5 in read_dump_filters
> /tmp/cirrus-ci-build/src/bin/pg_dump/pg_dump.c:18229
> [06:33:11.271] # #3 0x559f1ac2bb1b in main
> /tmp/cirrus-ci-build/src/bin/pg_dump/pg_dump.c:630
> [06:33:11.271] # #4 0x7fd91fabfd09 in __libc_start_main
> (/lib/x86_64-linux-gnu/libc.so.6+0x23d09)
> [06:33:11.271] # #5 0x559f1abe5d29 in _start
> (/tmp/cirrus-ci-build/tmp_install/usr/local/pgsql/bin/pg_dump+0x39d29)
> [06:33:11.271] #
> [06:33:11.271] # 0x619000000480 is located 0 bytes to the right of
> 1024-byte region [0x619000000080,0x619000000480)
> [06:33:11.271] # allocated by thread T0 here:
> [06:33:11.271] # #0 0x7fd91fe14e8f in __interceptor_malloc
> ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
> [06:33:11.271] # #1 0x559f1ac69f35 in pg_malloc_internal
> /tmp/cirrus-ci-build/src/common/fe_memutils.c:30
> [06:33:11.271] # #2 0x559f1ac69f35 in palloc
> /tmp/cirrus-ci-build/src/common/fe_memutils.c:117
> [06:33:11.271] #
> [06:33:11.271] # SUMMARY: AddressSanitizer: heap-buffer-overflow
> /tmp/cirrus-ci-build/src/bin/pg_dump/filter.c:302 in read_pattern
>
I'll check it
>
>
> Greetings,
>
> Andres Freund
>
^ permalink raw reply [nested|flat] 181+ messages in thread
* Re: proposal: possibility to read dumped table's name from file
@ 2022-11-22 17:02 Pavel Stehule <[email protected]>
parent: Andres Freund <[email protected]>
1 sibling, 0 replies; 181+ messages in thread
From: Pavel Stehule @ 2022-11-22 17:02 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Justin Pryzby <[email protected]>; Daniel Gustafsson <[email protected]>; Andrew Dunstan <[email protected]>; John Naylor <[email protected]>; Erik Rijkers <[email protected]>; Tom Lane <[email protected]>; Alvaro Herrera <[email protected]>; Tomas Vondra <[email protected]>; Dean Rasheed <[email protected]>; Stephen Frost <[email protected]>; Surafel Temesgen <[email protected]>; vignesh C <[email protected]>; pgsql-hackers
Hi
út 22. 11. 2022 v 8:39 odesílatel Andres Freund <[email protected]> napsal:
> Hi,
>
> On 2022-11-13 20:32:47 +0100, Pavel Stehule wrote:
> > updated patch attached
>
> It fails with address sanitizer that's now part of CI:
>
> https://cirrus-ci.com/task/6031397744279552?logs=test_world#L2659
>
> [06:33:11.271] # ==31965==ERROR: AddressSanitizer: heap-buffer-overflow on
> address 0x619000000480 at pc 0x559f1ac40822 bp 0x7ffea83e1ad0 sp
> 0x7ffea83e1ac8
> [06:33:11.271] # READ of size 1 at 0x619000000480 thread T0
> [06:33:11.271] # #0 0x559f1ac40821 in read_pattern
> /tmp/cirrus-ci-build/src/bin/pg_dump/filter.c:302
> [06:33:11.271] # #1 0x559f1ac40e4d in filter_read_item
> /tmp/cirrus-ci-build/src/bin/pg_dump/filter.c:459
> [06:33:11.271] # #2 0x559f1abe6fa5 in read_dump_filters
> /tmp/cirrus-ci-build/src/bin/pg_dump/pg_dump.c:18229
> [06:33:11.271] # #3 0x559f1ac2bb1b in main
> /tmp/cirrus-ci-build/src/bin/pg_dump/pg_dump.c:630
> [06:33:11.271] # #4 0x7fd91fabfd09 in __libc_start_main
> (/lib/x86_64-linux-gnu/libc.so.6+0x23d09)
> [06:33:11.271] # #5 0x559f1abe5d29 in _start
> (/tmp/cirrus-ci-build/tmp_install/usr/local/pgsql/bin/pg_dump+0x39d29)
> [06:33:11.271] #
> [06:33:11.271] # 0x619000000480 is located 0 bytes to the right of
> 1024-byte region [0x619000000080,0x619000000480)
> [06:33:11.271] # allocated by thread T0 here:
> [06:33:11.271] # #0 0x7fd91fe14e8f in __interceptor_malloc
> ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
> [06:33:11.271] # #1 0x559f1ac69f35 in pg_malloc_internal
> /tmp/cirrus-ci-build/src/common/fe_memutils.c:30
> [06:33:11.271] # #2 0x559f1ac69f35 in palloc
> /tmp/cirrus-ci-build/src/common/fe_memutils.c:117
> [06:33:11.271] #
> [06:33:11.271] # SUMMARY: AddressSanitizer: heap-buffer-overflow
> /tmp/cirrus-ci-build/src/bin/pg_dump/filter.c:302 in read_pattern
>
>
should be fixed in attached patch
I found and fix small memleak 24bytes per filter row (PQExpBufferData)
Regards
Pavel
>
> Greetings,
>
> Andres Freund
>
Attachments:
[text/x-patch] pg_dump-filter-20221122.patch (55.4K, ../../CAFj8pRDC5zVgh4YY_bmfp9GPTeqRgTTXK5ZnY4-5oJsa9qvKug@mail.gmail.com/3-pg_dump-filter-20221122.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index 8b9d9f4cad..d5a6e2c7ee 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -779,6 +779,80 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects to include
+ or exclude from the dump. The patterns are interpreted according to the
+ same rules as the corresponding options:
+ <option>-t</option>/<option>--table</option> for tables,
+ <option>-n</option>/<option>--schema</option> for schemas,
+ <option>--include-foreign-data</option> for data on foreign servers and
+ <option>--exclude-table-data</option> for table data.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one object pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { table | schema | foreign_data | table_data } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+
+ <para>
+ The first keyword specifies whether the objects matched by the pattern
+ are to be included or excluded. The second keyword specifies the type
+ of object to be filtered using the pattern:
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>table</literal>: tables, works like
+ <option>-t</option>/<option>--table</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>schema</literal>: schemas, works like
+ <option>-n</option>/<option>--schema</option>
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>foreign_data</literal>: data on foreign servers, works like
+ <option>--include-foreign-data</option>. This keyword can only be
+ used with the <literal>include</literal> keyword.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>table_data</literal>: table data, works like
+ <option>--exclude-table-data</option>. This keyword can only be
+ used with the <literal>exclude</literal> keyword.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Lines starting with <literal>#</literal> are considered comments and
+ ignored. Comments can be placed after filter as well. Blank lines
+ are also ignored. See <xref linkend="app-psql-patterns"/> for how to
+ perform quoting in patterns.
+ </para>
+
+ <para>
+ Example files are listed below in the <xref linkend="pg-dump-examples"/>
+ section.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--if-exists</option></term>
<listitem>
@@ -1119,6 +1193,7 @@ PostgreSQL documentation
schema (<option>-n</option>/<option>--schema</option>) and
table (<option>-t</option>/<option>--table</option>) qualifier
match at least one extension/schema/table in the database to be dumped.
+ This also applies to filters used with <option>--filter</option>.
Note that if none of the extension/schema/table qualifiers find
matches, <application>pg_dump</application> will generate an error
even without <option>--strict-names</option>.
@@ -1528,6 +1603,19 @@ CREATE DATABASE foo WITH TEMPLATE template0;
<screen>
<prompt>$</prompt> <userinput>pg_dump -t "\"MixedCaseName\"" mydb > mytab.sql</userinput>
+</screen></para>
+
+ <para>
+ To dump all tables with names starting with mytable, except for table
+ <literal>mytable2</literal>, specify a filter file
+ <filename>filter.txt</filename> like:
+<programlisting>
+include table mytable*
+exclude table mytable2
+</programlisting>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dump --filter=filter.txt mydb > db.sql</userinput>
</screen></para>
</refsect1>
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index e62d05e5ab..9cad26bbe6 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -122,6 +122,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for databases excluded
+ from dump. The patterns are interpretted according to the same rules
+ as <option>--exclude-database</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for excluding databases,
+ and can also be specified more than once for multiple filter files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+exclude database <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 47bd7dbda0..ffeb564c52 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -188,6 +188,31 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify a filename from which to read patterns for objects excluded
+ or included from restore. The patterns are interpretted according to the
+ same rules as <option>--schema</option>, <option>--exclude-schema</option>,
+ <option>--function</option>, <option>--index</option>, <option>--table</option>
+ or <option>--trigger</option>.
+ To read from <literal>STDIN</literal>, use <filename>-</filename> as the
+ filename. The <option>--filter</option> option can be specified in
+ conjunction with the above listed options for including or excluding
+ objects, and can also be specified more than once for multiple filter
+ files.
+ </para>
+
+ <para>
+ The file lists one database pattern per row, with the following format:
+<synopsis>
+{ include | exclude } { function | index | schema | table | trigger } <replaceable class="parameter">PATTERN</replaceable>
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 9dc5a784dd..700e1400c8 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -26,6 +26,7 @@ OBJS = \
$(WIN32RES) \
compress_io.o \
dumputils.o \
+ filter.o \
parallel.o \
pg_backup_archiver.o \
pg_backup_custom.o \
@@ -43,8 +44,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/filter.c b/src/bin/pg_dump/filter.c
new file mode 100644
index 0000000000..500db870c2
--- /dev/null
+++ b/src/bin/pg_dump/filter.c
@@ -0,0 +1,481 @@
+/*-------------------------------------------------------------------------
+ *
+ * Implementation of simple filter file parser
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres_fe.h"
+
+#include "common/fe_memutils.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "filter.h"
+#include "lib/stringinfo.h"
+#include "pqexpbuffer.h"
+
+#define is_keyword_str(cstr, str, bytes) \
+ ((strlen(cstr) == (bytes)) && (pg_strncasecmp((cstr), (str), (bytes)) == 0))
+
+/*
+ * Following routines are called from pg_dump, pg_dumpall and pg_restore.
+ * Unfortunately, the implementation of exit_nicely in pg_dump and pg_restore is
+ * different from the one in pg_dumpall, so instead of calling exit_nicely we
+ * have to return some error flag (in this case NULL), and exit_nicelly will be
+ * executed from caller's routine.
+ */
+
+/*
+ * Opens filter's file and initialize fstate structure.
+ * Returns true on success.
+ */
+bool
+filter_init(FilterStateData *fstate, const char *filename)
+{
+ fstate->filename = filename;
+ fstate->lineno = 0;
+ initStringInfo(&fstate->linebuff);
+
+ if (strcmp(filename, "-") != 0)
+ {
+ fstate->fp = fopen(filename, "r");
+ if (!fstate->fp)
+ {
+ pg_log_error("could not open filter file \"%s\": %m", filename);
+ return false;
+ }
+ }
+ else
+ fstate->fp = stdin;
+
+ fstate->is_error = false;
+
+ return true;
+}
+
+/*
+ * Release allocated resources for the given filter.
+ */
+void
+filter_free(FilterStateData *fstate)
+{
+ free(fstate->linebuff.data);
+ fstate->linebuff.data = NULL;
+
+ if (fstate->fp && fstate->fp != stdin)
+ {
+ if (fclose(fstate->fp) != 0)
+ pg_log_error("could not close filter file \"%s\": %m", fstate->filename);
+
+ fstate->fp = NULL;
+ }
+}
+
+/*
+ * Translate FilterObjectType enum to string. It is designed for formatting
+ * of error message in log_unsupported_filter_object_type routine.
+ */
+static const char *
+filter_object_type_name(FilterObjectType fot)
+{
+ switch (fot)
+ {
+ case FILTER_OBJECT_TYPE_NONE:
+ return "comment or empty line";
+ case FILTER_OBJECT_TYPE_TABLE_DATA:
+ return "table data";
+ case FILTER_OBJECT_TYPE_DATABASE:
+ return "database";
+ case FILTER_OBJECT_TYPE_FOREIGN_DATA:
+ return "foreign data";
+ case FILTER_OBJECT_TYPE_FUNCTION:
+ return "function";
+ case FILTER_OBJECT_TYPE_INDEX:
+ return "index";
+ case FILTER_OBJECT_TYPE_SCHEMA:
+ return "schema";
+ case FILTER_OBJECT_TYPE_TABLE:
+ return "table";
+ case FILTER_OBJECT_TYPE_TRIGGER:
+ return "trigger";
+ }
+
+ /* should never get here */
+ pg_unreachable();
+}
+
+/*
+ * Emit error message "invalid format in filter file ..."
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_invalid_filter_format(FilterStateData *fstate, char *message)
+{
+ if (fstate->fp != stdin)
+ {
+ pg_log_error("invalid format in filter file \"%s\" on line %d: %s",
+ fstate->filename,
+ fstate->lineno,
+ message);
+ }
+ else
+ pg_log_error("invalid format in filter on line %d: %s",
+ fstate->lineno,
+ message);
+
+ fstate->is_error = true;
+}
+
+/*
+ * Emit error message "The application doesn't support filter for object type ..."
+ *
+ * This is mostly a convenience routine to avoid duplicating file closing code
+ * in multiple callsites.
+ */
+void
+log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname,
+ FilterObjectType fot)
+{
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str,
+ "\"%s\" doesn't support filter for object type \"%s\".",
+ appname,
+ filter_object_type_name(fot));
+
+ log_invalid_filter_format(fstate, str->data);
+}
+
+/*
+ * filter_get_keyword - read the next filter keyword from buffer
+ *
+ * Search for keywords (limited to ascii alphabetic characters) in
+ * the passed in line buffer. Returns NULL when the buffer is empty or first
+ * char is not alpha. The char '_' is allowed too (exclude first position).
+ * The length of the found keyword is returned in the size parameter.
+ */
+static const char *
+filter_get_keyword(const char **line, int *size)
+{
+ const char *ptr = *line;
+ const char *result = NULL;
+
+ /* Set returnlength preemptively in case no keyword is found */
+ *size = 0;
+
+ /* Skip initial whitespace */
+ while (isspace(*ptr))
+ ptr++;
+
+ if (isalpha(*ptr))
+ {
+ result = ptr++;
+
+ while (isalpha(*ptr) || *ptr == '_')
+ ptr++;
+
+ *size = ptr - result;
+ }
+
+ *line = ptr;
+
+ return result;
+}
+
+/*
+ * read_quoted_pattern - read quoted possibly multi lined string.
+ *
+ * Returns pointer to next char after ending double quotes or NULL on error.
+ */
+static const char *
+read_quoted_string(FilterStateData *fstate,
+ const char *str,
+ PQExpBuffer pattern)
+{
+ appendPQExpBufferChar(pattern, '"');
+ str++;
+
+ while (1)
+ {
+ /*
+ * We can ignore \r or \n chars because the string is read by
+ * pg_get_line_buf, so these chars should be just trailing chars.
+ */
+ if (*str == '\r' || *str == '\n')
+ {
+ str++;
+ continue;
+ }
+
+ if (*str == '\0')
+ {
+ Assert(fstate->linebuff.data);
+
+ if (!pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m",
+ fstate->filename);
+ fstate->is_error = true;
+ }
+ else
+ log_invalid_filter_format(fstate, "unexpected end of file");
+
+ return NULL;
+ }
+
+ str = fstate->linebuff.data;
+
+ appendPQExpBufferChar(pattern, '\n');
+ fstate->lineno++;
+ }
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(pattern, '"');
+ str++;
+
+ if (*str == '"')
+ {
+ appendPQExpBufferChar(pattern, '"');
+ str++;
+ }
+ else
+ break;
+ }
+ else if (*str == '\\')
+ {
+ str++;
+ if (*str == 'n')
+ appendPQExpBufferChar(pattern, '\n');
+ else if (*str == '\\')
+ appendPQExpBufferChar(pattern, '\\');
+
+ str++;
+ }
+ else
+ appendPQExpBufferChar(pattern, *str++);
+ }
+
+ return str;
+}
+
+/*
+ * read_pattern - reads on object pattern from input
+ *
+ * This function will parse any valid identifier (quoted or not, qualified or
+ * not), which can also includes the full signature for routines.
+ * Note that this function takes special care to sanitize the detected
+ * identifier (removing extraneous whitespaces or other unnecessary
+ * characters). This is necessary as most backup/restore filtering functions
+ * only recognize identifiers if they are written exactly way as they are
+ * regenerated.
+ * Returns a pointer to next character after the found identifier, or NULL on
+ * error.
+ */
+static const char *
+read_pattern(FilterStateData *fstate, const char *str, PQExpBuffer pattern)
+{
+ bool skip_space = true;
+ bool found_space = false;
+
+ /* Skip initial whitespace */
+ while (isspace(*str))
+ str++;
+
+ if (*str == '\0')
+ {
+ log_invalid_filter_format(fstate, "missing object name pattern");
+ return NULL;
+ }
+
+ while (*str && *str != '#')
+ {
+ while (*str && !isspace(*str) && !strchr("#,.()\"", *str))
+ {
+ /*
+ * Append space only when it is allowed, and when it was found
+ * in original string.
+ */
+ if (!skip_space && found_space)
+ {
+ appendPQExpBufferChar(pattern, ' ');
+ skip_space = true;
+ }
+
+ appendPQExpBufferChar(pattern, *str++);
+ }
+
+ skip_space = false;
+
+ if (*str == '"')
+ {
+ if (found_space)
+ appendPQExpBufferChar(pattern, ' ');
+
+ str = read_quoted_string(fstate, str, pattern);
+ if (!str)
+ return NULL;
+ }
+ else if (*str == ',')
+ {
+ appendPQExpBufferStr(pattern, ", ");
+ skip_space = true;
+ str++;
+ }
+ else if (*str && strchr(".()", *str))
+ {
+ appendPQExpBufferChar(pattern, *str++);
+ skip_space = true;
+ }
+
+ found_space = false;
+
+ /* skip ending whitespaces */
+ while (isspace(*str))
+ {
+ found_space = true;
+ str++;
+ }
+ }
+
+ return str;
+}
+
+/*
+ * filter_read_item - Read command/type/pattern triplet from a filter file
+ *
+ * This will parse one filter item from the filter file, and while it is a
+ * row based format a pattern may span more than one line due to how object
+ * names can be constructed. The expected format of the filter file is:
+ *
+ * <command> <object_type> <pattern>
+ *
+ * command can be "include" or "exclude"
+ * object_type can one of: "table", "schema", "foreign_data", "table_data",
+ * "database", "function", "trigger" or "index"
+ * pattern can be any possibly-quoted and possibly-qualified identifier. It
+ * follows the same rules as other object include and exclude functions so it
+ * can also use wildcards.
+ *
+ * Returns true when one filter item was successfully read and parsed. When
+ * object name contains \n chars, then more than one line from input file can
+ * be processed. Returns false when the filter file reaches EOF. In case of
+ * error, the function will emit an appropriate error message before returning
+ * false.
+ */
+bool
+filter_read_item(FilterStateData *fstate,
+ bool *is_include,
+ char **objname,
+ FilterObjectType *objtype)
+{
+ Assert(!fstate->is_error);
+
+ if (pg_get_line_buf(fstate->fp, &fstate->linebuff))
+ {
+ const char *str = fstate->linebuff.data;
+ const char *keyword;
+ int size;
+ PQExpBufferData pattern;
+
+ fstate->lineno++;
+
+ /* Skip initial white spaces */
+ while (isspace(*str))
+ str++;
+
+ /*
+ * Skip empty lines or lines where the first non-whitespace character
+ * is a hash indicating a comment.
+ */
+ if (*str != '\0' && *str != '#')
+ {
+ /*
+ * First we expect sequence of two keywords, {include|exclude}
+ * followed by the object type to operate on.
+ */
+ keyword = filter_get_keyword(&str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate,
+ "no filter command found (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ if (is_keyword_str("include", keyword, size))
+ *is_include = true;
+ else if (is_keyword_str("exclude", keyword, size))
+ *is_include = false;
+ else
+ {
+ log_invalid_filter_format(fstate,
+ "invalid filter command (expected \"include\" or \"exclude\")");
+ return false;
+ }
+
+ keyword = filter_get_keyword(&str, &size);
+ if (!keyword)
+ {
+ log_invalid_filter_format(fstate, "missing filter object type");
+ return false;
+ }
+
+ if (is_keyword_str("table_data", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE_DATA;
+ else if (is_keyword_str("database", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_DATABASE;
+ else if (is_keyword_str("foreign_data",keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FOREIGN_DATA;
+ else if (is_keyword_str("function", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_FUNCTION;
+ else if (is_keyword_str("index", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_INDEX;
+ else if (is_keyword_str("schema", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_SCHEMA;
+ else if (is_keyword_str("table", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TABLE;
+ else if (is_keyword_str("trigger", keyword, size))
+ *objtype = FILTER_OBJECT_TYPE_TRIGGER;
+ else
+ {
+ PQExpBuffer str = createPQExpBuffer();
+
+ printfPQExpBuffer(str, "unsupported filter object type: \"%.*s\"", size, keyword);
+ log_invalid_filter_format(fstate, str->data);
+ return false;
+ }
+
+ initPQExpBuffer(&pattern);
+
+ str = read_pattern(fstate, str, &pattern);
+ if (!str)
+ return false;
+
+ *objname = pattern.data;
+ }
+ else
+ {
+ *objname = NULL;
+ *objtype = FILTER_OBJECT_TYPE_NONE;
+ }
+
+ return true;
+ }
+
+ if (ferror(fstate->fp))
+ {
+ pg_log_error("could not read from filter file \"%s\": %m", fstate->filename);
+ fstate->is_error = true;
+ }
+
+ return false;
+}
diff --git a/src/bin/pg_dump/filter.h b/src/bin/pg_dump/filter.h
new file mode 100644
index 0000000000..251763ad8e
--- /dev/null
+++ b/src/bin/pg_dump/filter.h
@@ -0,0 +1,54 @@
+/*-------------------------------------------------------------------------
+ *
+ * filter.h
+ * Common header file for the parser of filter file
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/bin/pg_dump/filter.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef FILTER_H
+#define FILTER_H
+
+#include "lib/stringinfo.h"
+
+/*
+ * State data for reading filter items from stream
+ */
+typedef struct
+{
+ FILE *fp;
+ const char *filename;
+ int lineno;
+ StringInfoData linebuff;
+ bool is_error;
+} FilterStateData;
+
+/*
+ * List of objects that can be specified in filter file
+ */
+typedef enum
+{
+ FILTER_OBJECT_TYPE_NONE,
+ FILTER_OBJECT_TYPE_TABLE_DATA,
+ FILTER_OBJECT_TYPE_DATABASE,
+ FILTER_OBJECT_TYPE_FOREIGN_DATA,
+ FILTER_OBJECT_TYPE_FUNCTION,
+ FILTER_OBJECT_TYPE_INDEX,
+ FILTER_OBJECT_TYPE_SCHEMA,
+ FILTER_OBJECT_TYPE_TABLE,
+ FILTER_OBJECT_TYPE_TRIGGER,
+} FilterObjectType;
+
+extern bool filter_init(FilterStateData *fstate, const char *filename);
+extern void filter_free(FilterStateData *fstate);
+
+extern void log_invalid_filter_format(FilterStateData *fstate, char *message);
+extern void log_unsupported_filter_object_type(FilterStateData *fstate,
+ const char *appname, FilterObjectType fot);
+extern bool filter_read_item(FilterStateData *fstate, bool *is_include, char **objname, FilterObjectType *objtype);
+
+#endif
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index e66f632b54..c099d66450 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -1,6 +1,7 @@
pg_dump_common_sources = files(
'compress_io.c',
'dumputils.c',
+ 'filter.c',
'parallel.c',
'pg_backup_archiver.c',
'pg_backup_custom.c',
@@ -86,6 +87,7 @@ tests += {
't/001_basic.pl',
't/002_pg_dump.pl',
't/003_pg_dump_with_server.pl',
+ 't/004_pg_dump_filterfile.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index da427f4d4a..a231dc2bb3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -59,6 +59,7 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "libpq/libpq-fs.h"
#include "parallel.h"
@@ -318,6 +319,7 @@ static void appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
static char *get_synchronized_snapshot(Archive *fout);
static void setupDumpWorker(Archive *AH);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
+static void read_dump_filters(const char *filename, DumpOptions *dopt);
int
@@ -390,6 +392,7 @@ main(int argc, char **argv)
{"enable-row-security", no_argument, &dopt.enable_row_security, 1},
{"exclude-table-data", required_argument, NULL, 4},
{"extra-float-digits", required_argument, NULL, 8},
+ {"filter", required_argument, NULL, 12},
{"if-exists", no_argument, &dopt.if_exists, 1},
{"inserts", no_argument, NULL, 9},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -623,6 +626,10 @@ main(int argc, char **argv)
optarg);
break;
+ case 12: /* object filters from file */
+ read_dump_filters(optarg, &dopt);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1028,6 +1035,8 @@ help(const char *progname)
" access to)\n"));
printf(_(" --exclude-table-data=PATTERN do NOT dump data for the specified table(s)\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME dump objects and data based on the filter expressions\n"
+ " in specified file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --include-foreign-data=PATTERN\n"
" include data of foreign tables on foreign\n"
@@ -18198,3 +18207,89 @@ appendReloptionsArrayAH(PQExpBuffer buffer, const char *reloptions,
if (!res)
pg_log_warning("could not parse %s array", "reloptions");
}
+
+/*
+ * read_dump_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_dump_filters(const char *filename, DumpOptions *dopt)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments and empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_TABLE_DATA)
+ {
+ if (is_include)
+ {
+ log_invalid_filter_format(&fstate,
+ "\"include\" table data filter is not allowed");
+ break;
+ }
+ else
+ simple_string_list_append(&tabledata_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_FOREIGN_DATA)
+ {
+ if (is_include)
+ simple_string_list_append(&foreign_servers_include_patterns,
+ objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" foreign data filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&schema_include_patterns,
+ objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&schema_exclude_patterns,
+ objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ simple_string_list_append(&table_include_patterns, objname);
+ dopt->include_everything = false;
+ }
+ else
+ simple_string_list_append(&table_exclude_patterns, objname);
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dump", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index a87262e333..6e99e4da5d 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -26,6 +26,7 @@
#include "common/string.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
@@ -81,6 +82,7 @@ static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
+static void read_dumpall_filters(const char *filename, SimpleStringList *patterns);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -158,6 +160,7 @@ main(int argc, char *argv[])
{"disable-triggers", no_argument, &disable_triggers, 1},
{"exclude-database", required_argument, NULL, 6},
{"extra-float-digits", required_argument, NULL, 5},
+ {"filter", required_argument, NULL, 8},
{"if-exists", no_argument, &if_exists, 1},
{"inserts", no_argument, &inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
@@ -360,6 +363,10 @@ main(int argc, char *argv[])
appendShellString(pgdumpopts, optarg);
break;
+ case 8:
+ read_dumpall_filters(optarg, &database_exclude_patterns);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -653,6 +660,7 @@ help(void)
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --exclude-database=PATTERN exclude databases whose name matches PATTERN\n"));
printf(_(" --extra-float-digits=NUM override default setting for extra_float_digits\n"));
+ printf(_(" --filter=FILENAME exclude databases specified in filter file\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --load-via-partition-root load partitions via the root table\n"));
@@ -1900,7 +1908,6 @@ executeCommand(PGconn *conn, const char *query)
PQclear(res);
}
-
/*
* dumpTimestamp
*/
@@ -1924,3 +1931,55 @@ hash_string_pointer(char *s)
return hash_bytes(ss, strlen(s));
}
+
+/*
+ * read_dumpall_filters - retrieve database identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ *
+ * In this moment only excluded databases can be filtered.
+ */
+static void
+read_dumpall_filters(const char *filename, SimpleStringList *pattern)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_DATABASE)
+ {
+ if (!is_include)
+ simple_string_list_append(pattern, objname);
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"include\" database filter is not allowed");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_dumpall", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 049a100634..1ebf573ac4 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -47,11 +47,13 @@
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
static void usage(const char *progname);
+static void read_restore_filters(const char *filename, RestoreOptions *dopt);
int
main(int argc, char **argv)
@@ -123,6 +125,7 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
+ {"filter", required_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -286,6 +289,10 @@ main(int argc, char **argv)
set_dump_section(optarg, &(opts->dumpSections));
break;
+ case 4:
+ read_restore_filters(optarg, opts);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -463,6 +470,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --filter=FILE restore objects based on filter expressions\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
printf(_(" --no-comments do not restore comments\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
@@ -494,3 +502,109 @@ usage(const char *progname)
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
}
+
+/*
+ * read_restore_filters - retrieve object identifer patterns from file
+ *
+ * Parse the specified filter file for include and exclude patterns, and add
+ * them to the relevant lists. If the filename is "-" then filters will be
+ * read from STDIN rather than a file.
+ */
+static void
+read_restore_filters(const char *filename, RestoreOptions *opts)
+{
+ FilterStateData fstate;
+ bool is_include;
+ char *objname;
+ FilterObjectType objtype;
+
+ if (!filter_init(&fstate, filename))
+ exit_nicely(1);
+
+ while (filter_read_item(&fstate, &is_include, &objname, &objtype))
+ {
+ /* ignore comments or empty lines */
+ if (objtype == FILTER_OBJECT_TYPE_NONE)
+ continue;
+
+ if (objtype == FILTER_OBJECT_TYPE_FUNCTION)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selFunction = 1;
+ simple_string_list_append(&opts->functionNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" function filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_INDEX)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selIndex = 1;
+ simple_string_list_append(&opts->indexNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" type filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_SCHEMA)
+ {
+ if (is_include)
+ simple_string_list_append(&opts->schemaNames, objname);
+ else
+ simple_string_list_append(&opts->schemaExcludeNames, objname);
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TABLE)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTable = 1;
+ simple_string_list_append(&opts->tableNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" table filter is not allowed");
+ break;
+ }
+ }
+ else if (objtype == FILTER_OBJECT_TYPE_TRIGGER)
+ {
+ if (is_include)
+ {
+ opts->selTypes = 1;
+ opts->selTrigger = 1;
+ simple_string_list_append(&opts->triggerNames, objname);
+ }
+ else
+ {
+ log_invalid_filter_format(&fstate,
+ "\"exclude\" trigger filter is not allowed");
+ break;
+ }
+ }
+ else
+ {
+ log_unsupported_filter_object_type(&fstate, "pg_restore", objtype);
+ break;
+ }
+
+ if (objname)
+ free(objname);
+ }
+
+ filter_free(&fstate);
+ if (fstate.is_error)
+ exit_nicely(1);
+}
diff --git a/src/bin/pg_dump/t/004_pg_dump_filterfile.pl b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
new file mode 100644
index 0000000000..a1b06845a3
--- /dev/null
+++ b/src/bin/pg_dump/t/004_pg_dump_filterfile.pl
@@ -0,0 +1,643 @@
+
+# Copyright (c) 2021, PostgreSQL Global Development Group
+
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 89;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;;
+my $inputfile;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+my $port = $node->port;
+my $backupdir = $node->backup_dir;
+my $plainfile = "$backupdir/plain.sql";
+
+$node->init;
+$node->start;
+
+# Generate test objects
+$node->safe_psql('postgres', 'CREATE FOREIGN DATA WRAPPER dummy;');
+$node->safe_psql('postgres',
+ 'CREATE SERVER dummyserver FOREIGN DATA WRAPPER dummy;');
+
+$node->safe_psql('postgres', "CREATE TABLE table_one(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_two(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three(a varchar)");
+$node->safe_psql('postgres', "CREATE TABLE table_three_one(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"strange aaa
+name\"(a varchar)");
+$node->safe_psql(
+ 'postgres', "CREATE TABLE \"
+t
+t
+\"(a int)");
+
+$node->safe_psql('postgres',
+ "INSERT INTO table_one VALUES('*** TABLE ONE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_two VALUES('*** TABLE TWO ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three VALUES('*** TABLE THREE ***')");
+$node->safe_psql('postgres',
+ "INSERT INTO table_three_one VALUES('*** TABLE THREE_ONE ***')");
+
+$node->safe_psql('postgres', "CREATE DATABASE sourcedb");
+$node->safe_psql('postgres', "CREATE DATABASE targetdb");
+
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo1(a int) RETURNS int AS $$ select $1 $$ LANGUAGE sql');
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo2(a int) RETURNS int AS $$ select $1 $$ LANGUAGE sql');
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo3(a double precision, b int) RETURNS double precision AS $$ select $1 + $2 $$ LANGUAGE sql');
+$node->safe_psql('sourcedb', 'CREATE FUNCTION foo_trg() RETURNS trigger AS $$ BEGIN RETURN NEW; END $$ LANGUAGE plpgsql');
+$node->safe_psql('sourcedb', 'CREATE SCHEMA s1');
+$node->safe_psql('sourcedb', 'CREATE SCHEMA s2');
+$node->safe_psql('sourcedb', 'CREATE TABLE s1.t1(a int)');
+$node->safe_psql('sourcedb', 'CREATE SEQUENCE s1.s1');
+$node->safe_psql('sourcedb', 'CREATE TABLE s2.t2(a int)');
+$node->safe_psql('sourcedb', 'CREATE TABLE t1(a int, b int)');
+$node->safe_psql('sourcedb', 'CREATE TABLE t2(a int, b int)');
+$node->safe_psql('sourcedb', 'CREATE INDEX t1_idx1 ON t1(a)');
+$node->safe_psql('sourcedb', 'CREATE INDEX t1_idx2 ON t1(b)');
+$node->safe_psql('sourcedb', 'CREATE TRIGGER trg1 BEFORE INSERT ON t1 EXECUTE FUNCTION foo_trg()');
+$node->safe_psql('sourcedb', 'CREATE TRIGGER trg2 BEFORE INSERT ON t1 EXECUTE FUNCTION foo_trg()');
+
+#
+# Test interaction of correctly specified filter file
+#
+my ($cmd, $stdout, $stderr, $result);
+
+# Empty filterfile
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "\n # a comment and nothing more\n\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+my $dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "table one dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "table two dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "table three dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m, "table three one dumped");
+
+# Test various combinations of whitespace, comments and correct filters
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile " include table table_one #comment\n";
+print $inputfile "include table table_two\n";
+print $inputfile "# skip this line\n";
+print $inputfile "\n";
+print $inputfile "\t\n";
+print $inputfile " \t# another comment\n";
+print $inputfile "exclude table_data table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter patterns as well as comments and whitespace");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump !~ qr/^CREATE TABLE public\.table_three/m, "table three not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_three_one/m,
+ "table three_one not dumped");
+ok( $dump !~ qr/^COPY public\.table_one/m,
+ "content of table one is not included");
+ok($dump =~ qr/^COPY public\.table_two/m, "content of table two is included");
+
+# Test dumping tables specified by qualified names
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table public.table_one\n";
+print $inputfile "include table \"public\".\"table_two\"\n";
+print $inputfile "include table \"public\". table_three\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "filter file without patterns");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "dumped table one");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with exclusion of a single table");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "dumped table two");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping tables with a wildcard pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_thre*\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with wildcard in pattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "table one not dumped");
+ok($dump !~ qr/^CREATE TABLE public\.table_two/m, "table two not dumped");
+ok($dump =~ qr/^CREATE TABLE public\.table_three/m, "dumped table three");
+ok($dump =~ qr/^CREATE TABLE public\.table_three_one/m,
+ "dumped table three_one");
+
+# Test dumping table with multiline quoted tablename
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"strange aaa
+name\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with multiline names requiring quoting");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding multiline quoted tablename from dump
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table \"strange aaa\\nname\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE public.\"strange aaa/m,
+ "dump table with new line in name");
+
+# Test excluding an entire schema
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema public\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "exclude the public schema");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test including and excluding an entire schema by multiple filterfiles
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema public\n";
+close $inputfile;
+
+open my $alt_inputfile, '>', "$tempdir/inputfile2.txt"
+ or die "unable to open filterfile for writing";
+print $alt_inputfile "exclude schema public\n";
+close $alt_inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "--filter=$tempdir/inputfile2.txt", 'postgres'
+ ],
+ "exclude the public schema with multiple filters");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^CREATE TABLE/m, "no table dumped");
+
+# Test dumping a table with a single leading newline on a row
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"
+t
+t
+\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table \"\\nt\\nt\\n\"";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public.\"\nt\nt\n\" \($/ms,
+ "dump table with multiline strange name");
+
+#########################################
+# Test foreign_data
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include foreign_data doesnt_exists\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/pg_dump: error: no matching foreign servers were found for pattern/,
+ "dump nonexisting foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile, "include foreign_data dummyserver\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ "dump foreign_data with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE SERVER dummyserver/m, "dump foreign server");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude foreign_data dummy*\n";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/"exclude" foreign data filter is not allowed/,
+ "erroneously exclude foreign server");
+
+#########################################
+# Test broken input format
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: invalid object type specified, should be table, schema, foreign_data or data"
+);
+
+# Test missing object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/missing object name/,
+ "invalid syntax: missing object identifier pattern");
+
+# Test adding extra content after the object identifier pattern
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table one";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt", 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "invalid syntax: extra content after object identifier pattern");
+
+#########################################
+# Combined with --strict-names
+
+# First ensure that a matching filter works
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_one\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ "strict names with matching mattern");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_one/m, "no table dumped");
+
+# Now append a pattern to the filter file which doesn't resolve
+open $inputfile, '>>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_nonexisting_name";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dump', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ '--strict-names', 'postgres'
+ ],
+ qr/no matching tables were found/,
+ "inclusion of non-existing objects with --strict names");
+
+#########################################
+# pg_dumpall tests
+
+###########################
+# Test dumping all tables except one
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude database postgres\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ "dump tables with exclusion of a database");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !~ qr/^\\connect postgres/m, "database postgres is not dumped");
+ok($dump =~ qr/^\\connect template1/m, "database template1 is dumped");
+
+# Test invalid filter command
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "k";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/invalid filter command/,
+ "invalid syntax: incorrect filter command");
+
+# Test invalid object type
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude xxx";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/unsupported filter object type: "xxx"/,
+ "invalid syntax: exclusion of non-existing object type"
+);
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude table foo";
+close $inputfile;
+
+command_fails_like(
+ [
+ 'pg_dumpall', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt"
+ ],
+ qr/"pg_dumpall" doesn't support filter for object type "table"/,
+ "invalid syntax: exclusion of unsupported object type"
+);
+
+#########################################
+# pg_restore tests
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', "$tempdir/filter_test.dump",
+ "-Fc", 'postgres'
+ ],
+ "dump all tables");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include table table_two";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore tables with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE public\.table_two/m, "wanted table restored");
+ok($dump !~ qr/^CREATE TABLE public\.table_one/m, "unwanted table is not restored");
+
+#########################################
+# test restore of other objects
+
+command_ok(
+ [
+ 'pg_dump', '-p', $port, '-f', "$tempdir/filter_test.dump",
+ "-Fc", 'sourcedb'
+ ],
+ "dump all objects from sourcedb");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include function foo1(integer)";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE FUNCTION public\.foo1/m, "wanted function restored");
+ok($dump !~ qr/^CREATE TABLE public\.foo2/m, "unwanted function is not restored");
+
+# this should be white space tolerant (against the -P argument)
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include function foo3 ( double precision , integer) ";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE FUNCTION public\.foo3/m, "wanted function restored");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include index t1_idx1\n";
+
+# attention! this hit pg_restore bug - correct name of trigger is "trg1"
+# not "t1 trg1". Should be fixed when pg_restore will be fixed
+print $inputfile "include trigger t1 trg1\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE INDEX t1_idx1/m, "wanted index restored");
+ok($dump !~ qr/^CREATE INDEX t2_idx2/m, "unwanted index are not restored");
+ok($dump =~ qr/^CREATE TRIGGER trg1/m, "wanted trigger restored");
+ok($dump !~ qr/^CREATE TRIGGER trg2/m, "unwanted trigger is not restored");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "include schema s1\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump =~ qr/^CREATE TABLE s1\.t1/m, "wanted table from schema restored");
+ok($dump =~ qr/^CREATE SEQUENCE s1\.s1/m, "wanted sequence from schema restored");
+ok($dump !~ qr/^CREATE TABLE s2\t2/m, "unwanted table is not restored");
+
+open $inputfile, '>', "$tempdir/inputfile.txt"
+ or die "unable to open filterfile for writing";
+print $inputfile "exclude schema s1\n";
+close $inputfile;
+
+command_ok(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--filter=$tempdir/inputfile.txt",
+ "-Fc", "$tempdir/filter_test.dump"
+ ],
+ "restore function with filter");
+
+$dump = slurp_file($plainfile);
+
+ok($dump !=~ qr/^CREATE TABLE s1\.t1/m, "unwanted table from schema is not restored");
+ok($dump !=~ qr/^CREATE SEQUENCE s1\.s1/m, "unwanted sequence from schema is not restored");
+ok($dump =~ qr/^CREATE TABLE s2\.t2/m, "wanted table restored");
+ok($dump =~ qr/^CREATE TABLE public\.t1/m, "wanted table restored");
diff --git a/src/tools/msvc/Mkvcbuild.pm b/src/tools/msvc/Mkvcbuild.pm
index 83a3e40425..d3c7292203 100644
--- a/src/tools/msvc/Mkvcbuild.pm
+++ b/src/tools/msvc/Mkvcbuild.pm
@@ -451,6 +451,7 @@ sub mkvcbuild
$pgdumpall->AddIncludeDir('src/backend');
$pgdumpall->AddFile('src/bin/pg_dump/pg_dumpall.c');
$pgdumpall->AddFile('src/bin/pg_dump/dumputils.c');
+ $pgdumpall->AddFile('src/bin/pg_dump/filter.c');
$pgdumpall->AddLibrary('ws2_32.lib');
my $pgrestore = AddSimpleFrontend('pg_dump', 1);
^ permalink raw reply [nested|flat] 181+ messages in thread
end of thread, other threads:[~2022-11-22 17:02 UTC | newest]
Thread overview: 181+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-05-29 14:21 proposal: possibility to read dumped table's name from file Pavel Stehule <[email protected]>
2020-05-29 14:28 ` Tom Lane <[email protected]>
2020-05-29 14:47 ` Pavel Stehule <[email protected]>
2020-05-29 15:28 ` David Fetter <[email protected]>
2020-05-29 16:03 ` Tom Lane <[email protected]>
2020-05-29 16:14 ` Pavel Stehule <[email protected]>
2020-05-29 18:25 ` Justin Pryzby <[email protected]>
2020-06-08 17:18 ` Pavel Stehule <[email protected]>
2020-06-08 21:29 ` Justin Pryzby <[email protected]>
2020-06-09 09:46 ` Pavel Stehule <[email protected]>
2020-06-09 22:30 ` Justin Pryzby <[email protected]>
2020-06-10 03:03 ` Pavel Stehule <[email protected]>
2020-06-10 03:57 ` Justin Pryzby <[email protected]>
2020-06-11 07:36 ` Pavel Stehule <[email protected]>
2020-06-27 12:55 ` vignesh C <[email protected]>
2020-07-05 19:50 ` Pavel Stehule <[email protected]>
2020-07-01 21:24 ` Justin Pryzby <[email protected]>
2020-07-05 20:08 ` Pavel Stehule <[email protected]>
2020-09-03 19:48 ` Justin Pryzby <[email protected]>
2020-07-05 20:31 ` Justin Pryzby <[email protected]>
2020-07-05 20:37 ` Pavel Stehule <[email protected]>
2020-07-06 04:34 ` Pavel Stehule <[email protected]>
2020-07-11 23:06 ` Justin Pryzby <[email protected]>
2020-07-13 06:15 ` Pavel Stehule <[email protected]>
2020-07-13 17:59 ` Justin Pryzby <[email protected]>
2020-07-14 06:31 ` Pavel Stehule <[email protected]>
2020-07-12 01:43 ` vignesh C <[email protected]>
2020-07-13 08:20 ` Pavel Stehule <[email protected]>
2020-07-13 10:04 ` Daniel Gustafsson <[email protected]>
2020-07-13 11:02 ` Pavel Stehule <[email protected]>
2020-07-13 14:57 ` Daniel Gustafsson <[email protected]>
2020-07-13 15:33 ` Pavel Stehule <[email protected]>
2020-07-13 14:32 ` Justin Pryzby <[email protected]>
2020-07-13 13:32 ` vignesh C <[email protected]>
2020-07-14 06:32 ` Pavel Stehule <[email protected]>
2020-07-25 13:26 ` vignesh C <[email protected]>
2020-07-26 19:10 ` Justin Pryzby <[email protected]>
2020-07-27 05:28 ` Pavel Stehule <[email protected]>
2020-07-27 05:25 ` Pavel Stehule <[email protected]>
2020-08-13 21:41 ` Justin Pryzby <[email protected]>
2020-09-03 20:08 ` Alvaro Herrera <[email protected]>
2020-09-04 00:15 ` Tom Lane <[email protected]>
2020-09-04 03:21 ` Pavel Stehule <[email protected]>
2020-09-07 05:08 ` Pavel Stehule <[email protected]>
2020-09-07 12:14 ` Surafel Temesgen <[email protected]>
2020-09-11 08:50 ` Pavel Stehule <[email protected]>
2020-09-12 08:12 ` Pavel Stehule <[email protected]>
2020-09-24 17:47 ` Pavel Stehule <[email protected]>
2020-11-10 20:09 ` Stephen Frost <[email protected]>
2020-11-11 05:32 ` Pavel Stehule <[email protected]>
2020-11-11 05:49 ` Pavel Stehule <[email protected]>
2020-11-17 21:53 ` Justin Pryzby <[email protected]>
2020-11-18 14:46 ` Stephen Frost <[email protected]>
2020-11-19 19:51 ` Pavel Stehule <[email protected]>
2020-11-19 19:57 ` Pavel Stehule <[email protected]>
2020-11-25 18:25 ` Dean Rasheed <[email protected]>
2020-11-25 19:29 ` Pavel Stehule <[email protected]>
2020-11-25 20:00 ` Tom Lane <[email protected]>
2020-11-26 06:42 ` Pavel Stehule <[email protected]>
2020-11-26 11:50 ` Dean Rasheed <[email protected]>
2020-11-26 16:02 ` Tom Lane <[email protected]>
2020-11-27 16:56 ` Stephen Frost <[email protected]>
2020-11-28 20:14 ` Pavel Stehule <[email protected]>
2020-11-28 23:49 ` Justin Pryzby <[email protected]>
2020-11-29 14:09 ` Pavel Stehule <[email protected]>
2021-02-16 19:32 ` Pavel Stehule <[email protected]>
2021-04-11 07:48 ` Pavel Stehule <[email protected]>
2021-05-12 06:22 ` Pavel Stehule <[email protected]>
2021-07-10 15:47 ` Tomas Vondra <[email protected]>
2021-07-12 22:08 ` Daniel Gustafsson <[email protected]>
2021-07-12 22:47 ` Tomas Vondra <[email protected]>
2021-07-12 22:59 ` Alvaro Herrera <[email protected]>
2021-07-12 23:16 ` Tom Lane <[email protected]>
2021-07-13 13:40 ` Stephen Frost <[email protected]>
2021-07-13 16:14 ` Tomas Vondra <[email protected]>
2021-07-13 20:44 ` Daniel Gustafsson <[email protected]>
2021-07-13 20:55 ` Stephen Frost <[email protected]>
2021-07-13 21:57 ` Tomas Vondra <[email protected]>
2021-07-13 22:32 ` Stephen Frost <[email protected]>
2021-07-13 23:00 ` Alvaro Herrera <[email protected]>
2021-07-14 00:18 ` Stephen Frost <[email protected]>
2021-07-14 05:00 ` Pavel Stehule <[email protected]>
2021-07-14 10:08 ` Tomas Vondra <[email protected]>
2021-07-28 07:28 ` Pavel Stehule <[email protected]>
2021-09-13 13:01 ` Daniel Gustafsson <[email protected]>
2021-09-15 17:31 ` Pavel Stehule <[email protected]>
2021-09-16 04:45 ` Pavel Stehule <[email protected]>
2021-09-17 11:18 ` Daniel Gustafsson <[email protected]>
2021-09-17 11:42 ` Pavel Stehule <[email protected]>
2022-07-13 20:49 ` Andrew Dunstan <[email protected]>
2022-07-14 04:54 ` Pavel Stehule <[email protected]>
2022-07-17 06:20 ` Pavel Stehule <[email protected]>
2022-07-17 14:01 ` Justin Pryzby <[email protected]>
2022-07-18 18:48 ` Pavel Stehule <[email protected]>
2022-08-22 09:52 ` Pavel Stehule <[email protected]>
2022-09-07 19:45 ` Daniel Gustafsson <[email protected]>
2022-09-08 10:00 ` Erik Rijkers <[email protected]>
2022-09-08 11:38 ` Daniel Gustafsson <[email protected]>
2022-09-08 11:44 ` Julien Rouhaud <[email protected]>
2022-09-08 12:32 ` Daniel Gustafsson <[email protected]>
2022-09-09 07:53 ` John Naylor <[email protected]>
2022-09-09 09:00 ` Andrew Dunstan <[email protected]>
2022-09-12 07:58 ` Daniel Gustafsson <[email protected]>
2022-09-12 13:09 ` Pavel Stehule <[email protected]>
2022-09-13 08:46 ` John Naylor <[email protected]>
2022-09-13 09:36 ` Pavel Stehule <[email protected]>
2022-09-12 14:00 ` Erik Rijkers <[email protected]>
2022-09-12 14:18 ` Erik Rijkers <[email protected]>
2022-09-22 15:10 ` Andres Freund <[email protected]>
2022-10-02 06:56 ` Andres Freund <[email protected]>
2022-10-02 07:19 ` Andres Freund <[email protected]>
2022-10-02 16:04 ` Andres Freund <[email protected]>
2022-10-02 20:52 ` Daniel Gustafsson <[email protected]>
2022-10-02 21:02 ` Tom Lane <[email protected]>
2022-10-02 21:11 ` Andres Freund <[email protected]>
2022-10-03 04:00 ` Pavel Stehule <[email protected]>
2022-10-03 04:34 ` Julien Rouhaud <[email protected]>
2022-10-07 05:26 ` Pavel Stehule <[email protected]>
2022-10-07 14:03 ` Julien Rouhaud <[email protected]>
2022-10-07 16:48 ` Pavel Stehule <[email protected]>
2022-10-13 18:46 ` Andres Freund <[email protected]>
2022-10-18 09:33 ` Julien Rouhaud <[email protected]>
2022-10-26 04:26 ` Pavel Stehule <[email protected]>
2022-11-03 04:09 ` Julien Rouhaud <[email protected]>
2022-11-03 21:22 ` Pavel Stehule <[email protected]>
2022-11-04 11:59 ` Julien Rouhaud <[email protected]>
2022-11-04 12:19 ` Justin Pryzby <[email protected]>
2022-11-04 13:28 ` Julien Rouhaud <[email protected]>
2022-11-04 13:37 ` Pavel Stehule <[email protected]>
2022-11-05 05:39 ` Julien Rouhaud <[email protected]>
2022-11-05 19:54 ` Pavel Stehule <[email protected]>
2022-11-11 08:11 ` Julien Rouhaud <[email protected]>
2022-11-12 20:35 ` Pavel Stehule <[email protected]>
2022-11-12 20:37 ` Pavel Stehule <[email protected]>
2022-11-13 08:58 ` Julien Rouhaud <[email protected]>
2022-11-13 19:32 ` Pavel Stehule <[email protected]>
2022-11-22 05:25 ` Julien Rouhaud <[email protected]>
2022-11-22 05:30 ` Pavel Stehule <[email protected]>
2022-11-22 07:39 ` Andres Freund <[email protected]>
2022-11-22 07:41 ` Pavel Stehule <[email protected]>
2022-11-22 17:02 ` Pavel Stehule <[email protected]>
2022-09-08 15:03 ` Pavel Stehule <[email protected]>
2021-09-17 11:42 ` Daniel Gustafsson <[email protected]>
2021-09-17 11:51 ` Pavel Stehule <[email protected]>
2021-09-17 11:56 ` Daniel Gustafsson <[email protected]>
2021-09-17 11:59 ` Pavel Stehule <[email protected]>
2021-09-17 12:06 ` Stephen Frost <[email protected]>
2021-09-17 12:07 ` Daniel Gustafsson <[email protected]>
2021-09-17 12:09 ` Stephen Frost <[email protected]>
2021-09-17 12:16 ` Pavel Stehule <[email protected]>
2021-09-17 13:06 ` Pavel Stehule <[email protected]>
2021-09-20 12:10 ` Daniel Gustafsson <[email protected]>
2021-09-21 06:50 ` Pavel Stehule <[email protected]>
2021-09-21 12:37 ` Daniel Gustafsson <[email protected]>
2021-09-21 12:46 ` Pavel Stehule <[email protected]>
2021-09-21 13:28 ` Alvaro Herrera <[email protected]>
2021-09-21 16:06 ` Tomas Vondra <[email protected]>
2021-09-21 16:20 ` Pavel Stehule <[email protected]>
2021-09-24 03:59 ` Pavel Stehule <[email protected]>
2021-10-01 13:19 ` Daniel Gustafsson <[email protected]>
2021-10-01 16:00 ` Pavel Stehule <[email protected]>
2021-10-01 16:19 ` Erik Rijkers <[email protected]>
2021-10-02 06:18 ` Erik Rijkers <[email protected]>
2021-10-04 12:54 ` Daniel Gustafsson <[email protected]>
2021-10-05 12:30 ` Daniel Gustafsson <[email protected]>
2021-10-05 12:37 ` Pavel Stehule <[email protected]>
2021-10-05 14:39 ` Erik Rijkers <[email protected]>
2021-10-27 09:15 ` Pavel Stehule <[email protected]>
2021-09-13 13:11 ` Justin Pryzby <[email protected]>
2021-09-15 18:30 ` Pavel Stehule <[email protected]>
2021-07-13 20:55 ` Daniel Gustafsson <[email protected]>
2021-07-28 04:18 ` Pavel Stehule <[email protected]>
2020-11-27 18:45 ` Stephen Frost <[email protected]>
2020-11-28 21:14 ` Pavel Stehule <[email protected]>
2020-11-11 07:09 ` Pavel Stehule <[email protected]>
2020-11-11 15:17 ` Alvaro Herrera <[email protected]>
2020-11-12 07:45 ` Pavel Stehule <[email protected]>
2020-11-11 05:54 ` Pavel Stehule <[email protected]>
2020-07-13 08:20 [PATCH 1/2] proposal: possibility to read dumped table's name from file Pavel Stehule <[email protected]>
2020-11-11 05:54 [PATCH 1/2] proposal: possibility to read dumped table's name from file Pavel Stehule <[email protected]>
2020-11-28 21:53 [PATCH 1/4] Re: proposal: possibility to read dumped table's name from file Justin Pryzby <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox