public inbox for [email protected]
help / color / mirror / Atom feedRe: Non-text mode for pg_dumpall
36+ messages / 3 participants
[nested] [flat]
* Re: Non-text mode for pg_dumpall
@ 2025-01-20 16:01 jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: jian he @ 2025-01-20 16:01 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Guillaume Lelarge <[email protected]>; Nathan Bossart <[email protected]>; Magnus Hagander <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>
hi.
some minor issues come to my mind when I look at it again.
looking at set_null_conf,
i think "if (archDumpFormat != archNull)" can be:
if (archDumpFormat != archNull)
{
OPF = fopen(toc_path, "w");
if (!OPF)
pg_fatal("could not open global.dat file: \"%s\" for writing: %m",
toc_path);
}
some places we use ``fopen(filename, PG_BINARY_W)``,
some places we use ``fopen(filename, "w");``
kind of inconsistent...
+ printf(_(" -F, --format=c|d|t|p output file format
(custom, directory, tar,\n"
+ " plain text (default))\n"));
this indentation level is not right?
if we look closely at the surrounding output of `pg_dumpall --help`.
pg_dump.sgml --create option description:
This option is ignored when emitting an archive (non-text) output file. For the
archive formats, you can specify the option when you call pg_restore.
in runPgDump, we have:
/*
* If this is non-plain format dump, then append file name and dump
* format to the pg_dump command to get archive dump.
*/
if (archDumpFormat != archNull)
{
printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
dbfile, create_opts);
...
}
so in here, create_opts is not necessary per pg_dump.sgml above description.
we can simplify it as:
if (archDumpFormat != archNull)
{
printfPQExpBuffer(&cmd, "\"%s\" --file=%s", pg_dump_bin, dbfile);
}
?
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-01-21 04:05 ` jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: jian he @ 2025-01-21 04:05 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Guillaume Lelarge <[email protected]>; Nathan Bossart <[email protected]>; Magnus Hagander <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>
hi.
$BIN10/pg_restore --globals-only --verbose --file=test.sql x.dump
it will create a "test.sql" file, but it should create file test.sql
(no double quotes).
------<>>>>------
if (archDumpFormat != archNull &&
(!filename || strcmp(filename, "") == 0))
{
pg_log_error("options -F/--format=d|c|t requires option
-f/--filename with non-empty string");
...
}
here, it should be
pg_log_error("options -F/--format=d|c|t requires option -f/--file with
non-empty string");
------<>>>>------
the following pg_dumpall, pg_restore not working.
$BIN10/pg_dumpall --format=custom --file=x1.dump --globals-only
$BIN10/pg_restore --file=3.sql x1.dump
ERROR: pg_restore: error: directory "x1.dump" does not appear to be a
valid archive ("toc.dat" does not exist)
these two also not working:
$BIN10/pg_dumpall --format=custom --file=x1.dump --verbose --globals-only
$BIN10/pg_restore --file=3.sql --format=custom x1.dump
error message:
pg_restore: error: could not read from input file: Is a directory
------<>>>>------
IsFileExistsInDirectory function is the same as _fileExistsInDirectory.
Can we make _fileExistsInDirectory extern function?
+ /* If global.dat and map.dat are exist, then proces them. */
+ if (IsFileExistsInDirectory(pg_strdup(inputFileSpec), "global.dat")
+ && IsFileExistsInDirectory(pg_strdup(inputFileSpec),
"map.dat"))
+ {
comment typo, "proces" should "process".
here, we don't need pg_strdup?
------<>>>>------
# pg_restore tests
+command_fails_like(
+ [
+ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--exclude-database=grabadge",
+ '--globals-only'
+ ],
+ qr/\Qg_restore: error: option --exclude-database cannot be used
together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together
with -g/--globals-only');
We can put the above test on src/bin/pg_dump/t/001_basic.pl,
since validating these conflict options don't need a cluster to be set up.
typedef struct SimpleDatabaseOidListCell
and
typedef struct SimpleDatabaseOidList
need also put into src/tools/pgindent/typedefs.list
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-01-21 09:29 ` jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: jian he @ 2025-01-21 09:29 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Guillaume Lelarge <[email protected]>; Nathan Bossart <[email protected]>; Magnus Hagander <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>
hi.
+ printfPQExpBuffer(query,
+ "SELECT substring ( "
+ " '%s' , "
+ " '%s' ) ", str, ptrn);
+ result = executeQuery(conn, query->data);
+ if (PQresultStatus(result) == PGRES_TUPLES_OK)
+ {
+ if (PQntuples(result) == 1)
+ {
+ const char *outstr;
+
+ outstr = PQgetvalue(result, 0, 0);
i think here you should use PQgetisnull(result, 0, 0)
?
example: pg_dumpall and pg_restore:
$BIN10/pg_dumpall --verbose --format=custom --file=x12.dump
$BIN10/pg_restore --verbose --dbname=src10 x12.dump
some log message for the above command:
pg_restore: found dbname as : "template1" and db_oid:1 in map.dat file
while restoring
pg_restore: found dbname as : "s1" and db_oid:17960 in map.dat file
while restoring
pg_restore: found dbname as : "src10" and db_oid:5 in map.dat file
while restoring
pg_restore: found total 3 database names in map.dat file
pg_restore: needs to restore 3 databases out of 3 databases
pg_restore: restoring dump of pg_dumpall without -C option, there
might be multiple databases in directory.
pg_restore: restoring database "template1"
pg_restore: connecting to database for restore
pg_restore: implied data-only restore
pg_restore: restoring database "s1"
pg_restore: connecting to database for restore
pg_restore: processing data for table "public.t"
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 3376; 0 17961 TABLE DATA t jian
pg_restore: error: could not execute query: ERROR: relation
"public.t" does not exist
Command was: COPY public.t (a) FROM stdin;
1. message: "pg_restore: implied data-only restore"
Normally pg_dump and pg_restore will dump the schema and the data,
then when we are connecting to the same database with pg_restore,
there will be lots of schema elements already exists ERROR.
but the above command case, pg_restore only restores the content/data
not schema, that's why there is very little error happening.
so here pg_restore not restore schema seems not ok?
2. pg_dumpall with non-text mode, we don't have \connect command in
file global.dat or map.dat
I have database "s1" with table "public.t".
if I create a table src10.public.t (database.schema.table) with column a.
then pg_restore will restore content of s1.public.t (database s1) to
src10.public.t (database src10).
in ConnectDatabase(Archive *AHX,
const ConnParams *cparams,
bool isReconnect)
i added
if (cparams->dbname)
fprintf(stderr, "pg_backup_db.c:%d %s called connecting to %s
now\n", __LINE__, __func__, cparams->dbname);
to confirm that we are connecting the same database "src10", while
dumping all the contents in x12.dump.
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-01-21 18:56 ` Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Mahendra Singh Thalor @ 2025-01-21 18:56 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Guillaume Lelarge <[email protected]>; Nathan Bossart <[email protected]>; Magnus Hagander <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>
Thanks Jian for the detailed review and testing.
On Mon, 20 Jan 2025 at 21:32, jian he <[email protected]> wrote:
>
> hi.
> some minor issues come to my mind when I look at it again.
>
> looking at set_null_conf,
> i think "if (archDumpFormat != archNull)" can be:
>
> if (archDumpFormat != archNull)
> {
> OPF = fopen(toc_path, "w");
> if (!OPF)
> pg_fatal("could not open global.dat file: \"%s\" for writing: %m",
> toc_path);
> }
>
> some places we use ``fopen(filename, PG_BINARY_W)``,
> some places we use ``fopen(filename, "w");``
> kind of inconsistent...
Fixed. We should use PG_BINARY_W/PG_BINARY_R.
>
>
> + printf(_(" -F, --format=c|d|t|p output file format
> (custom, directory, tar,\n"
> + " plain text (default))\n"));
> this indentation level is not right?
> if we look closely at the surrounding output of `pg_dumpall --help`.
Fixed.
>
>
> pg_dump.sgml --create option description:
> This option is ignored when emitting an archive (non-text) output file. For the
> archive formats, you can specify the option when you call pg_restore.
>
> in runPgDump, we have:
> /*
> * If this is non-plain format dump, then append file name and dump
> * format to the pg_dump command to get archive dump.
> */
> if (archDumpFormat != archNull)
> {
> printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
> dbfile, create_opts);
> ...
> }
>
> so in here, create_opts is not necessary per pg_dump.sgml above description.
> we can simplify it as:
>
> if (archDumpFormat != archNull)
> {
> printfPQExpBuffer(&cmd, "\"%s\" --file=%s", pg_dump_bin, dbfile);
> }
> ?
We are already using the same code without this patch also. I haven't
tested this without create_opts. I think, if your theory is right,
then
you can submit a patch for this change in another thread.
On Tue, 21 Jan 2025 at 09:37, jian he <[email protected]> wrote:
>
> hi.
>
> $BIN10/pg_restore --globals-only --verbose --file=test.sql x.dump
> it will create a "test.sql" file, but it should create file test.sql
> (no double quotes).
Fixed.
>
> ------<>>>>------
> if (archDumpFormat != archNull &&
> (!filename || strcmp(filename, "") == 0))
> {
> pg_log_error("options -F/--format=d|c|t requires option
> -f/--filename with non-empty string");
> ...
> }
> here, it should be
> pg_log_error("options -F/--format=d|c|t requires option -f/--file with
> non-empty string");
Fixed.
> ------<>>>>------
> the following pg_dumpall, pg_restore not working.
> $BIN10/pg_dumpall --format=custom --file=x1.dump --globals-only
> $BIN10/pg_restore --file=3.sql x1.dump
>
> ERROR: pg_restore: error: directory "x1.dump" does not appear to be a
> valid archive ("toc.dat" does not exist)
Fixed.
>
> these two also not working:
> $BIN10/pg_dumpall --format=custom --file=x1.dump --verbose --globals-only
> $BIN10/pg_restore --file=3.sql --format=custom x1.dump
Fixed.
>
> error message:
> pg_restore: error: could not read from input file: Is a directory
Fixed.
> ------<>>>>------
> IsFileExistsInDirectory function is the same as _fileExistsInDirectory.
> Can we make _fileExistsInDirectory extern function?
No, we can't make it as we are using this function in different-2 modules.
>
> + /* If global.dat and map.dat are exist, then proces them. */
> + if (IsFileExistsInDirectory(pg_strdup(inputFileSpec), "global.dat")
> + && IsFileExistsInDirectory(pg_strdup(inputFileSpec),
> "map.dat"))
> + {
> comment typo, "proces" should "process".
Fixed.
> here, we don't need pg_strdup?
In most places, we are dumping strings so I kept the same here also.
> ------<>>>>------
> # pg_restore tests
> +command_fails_like(
> + [
> + 'pg_restore', '-p', $port, '-f', $plainfile,
> + "--exclude-database=grabadge",
> + '--globals-only'
> + ],
> + qr/\Qg_restore: error: option --exclude-database cannot be used
> together with -g\/--globals-only\E/,
> + 'pg_restore: option --exclude-database cannot be used together
> with -g/--globals-only');
>
> We can put the above test on src/bin/pg_dump/t/001_basic.pl,
> since validating these conflict options don't need a cluster to be set up.
Done.
>
>
> typedef struct SimpleDatabaseOidListCell
> and
> typedef struct SimpleDatabaseOidList
> need also put into src/tools/pgindent/typedefs.list
Fixed.
On Tue, 21 Jan 2025 at 15:00, jian he <[email protected]> wrote:
>
> hi.
>
> + printfPQExpBuffer(query,
> + "SELECT substring ( "
> + " '%s' , "
> + " '%s' ) ", str, ptrn);
> + result = executeQuery(conn, query->data);
> + if (PQresultStatus(result) == PGRES_TUPLES_OK)
> + {
> + if (PQntuples(result) == 1)
> + {
> + const char *outstr;
> +
> + outstr = PQgetvalue(result, 0, 0);
> i think here you should use PQgetisnull(result, 0, 0)
Fixed.
>
> example: pg_dumpall and pg_restore:
> $BIN10/pg_dumpall --verbose --format=custom --file=x12.dump
> $BIN10/pg_restore --verbose --dbname=src10 x12.dump
>
> some log message for the above command:
> pg_restore: found dbname as : "template1" and db_oid:1 in map.dat file
> while restoring
> pg_restore: found dbname as : "s1" and db_oid:17960 in map.dat file
> while restoring
> pg_restore: found dbname as : "src10" and db_oid:5 in map.dat file
> while restoring
> pg_restore: found total 3 database names in map.dat file
> pg_restore: needs to restore 3 databases out of 3 databases
> pg_restore: restoring dump of pg_dumpall without -C option, there
> might be multiple databases in directory.
> pg_restore: restoring database "template1"
> pg_restore: connecting to database for restore
> pg_restore: implied data-only restore
> pg_restore: restoring database "s1"
> pg_restore: connecting to database for restore
> pg_restore: processing data for table "public.t"
> pg_restore: while PROCESSING TOC:
> pg_restore: from TOC entry 3376; 0 17961 TABLE DATA t jian
> pg_restore: error: could not execute query: ERROR: relation
> "public.t" does not exist
> Command was: COPY public.t (a) FROM stdin;
>
>
> 1. message: "pg_restore: implied data-only restore"
> Normally pg_dump and pg_restore will dump the schema and the data,
> then when we are connecting to the same database with pg_restore,
> there will be lots of schema elements already exists ERROR.
> but the above command case, pg_restore only restores the content/data
> not schema, that's why there is very little error happening.
> so here pg_restore not restore schema seems not ok?
>
>
> 2. pg_dumpall with non-text mode, we don't have \connect command in
> file global.dat or map.dat
> I have database "s1" with table "public.t".
> if I create a table src10.public.t (database.schema.table) with column a.
> then pg_restore will restore content of s1.public.t (database s1) to
> src10.public.t (database src10).
>
> in ConnectDatabase(Archive *AHX,
> const ConnParams *cparams,
> bool isReconnect)
> i added
> if (cparams->dbname)
> fprintf(stderr, "pg_backup_db.c:%d %s called connecting to %s
> now\n", __LINE__, __func__, cparams->dbname);
> to confirm that we are connecting the same database "src10", while
> dumping all the contents in x12.dump.
I will do some more study for this and will update. As of now, I added
the "--create" option in the dump.
Here, I am attaching an updated patch for review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v11_pg_dumpall-with-directory-tar-custom-format-21-jan.patch (70.6K, 2-v11_pg_dumpall-with-directory-tar-custom-format-21-jan.patch)
download | inline diff:
From bca342933c95a6739e195d3332584737dd4f64cc Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Wed, 22 Jan 2025 00:08:22 +0530
Subject: [PATCH] pg_dumpall with directory|tar|custom format and restore it
by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat and map.dat to restore all databases. If both files are exists in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list.
TODO1: We need to think for --exclude-database=PATTERN for pg_restore.
TODO2: We need to make changes for exit_nicely as we are one entry for each database while
restoring. MAX_ON_EXIT_NICELY
TODO3: some more test cases for new added options.
TODO4: We can dump and restore databases in parallel mode.
This needs more study
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
---
doc/src/sgml/ref/pg_dumpall.sgml | 78 ++-
doc/src/sgml/ref/pg_restore.sgml | 29 +
src/bin/pg_dump/Makefile | 8 +-
src/bin/pg_dump/common_dumpall_restore.c | 314 ++++++++++
src/bin/pg_dump/common_dumpall_restore.h | 26 +
src/bin/pg_dump/meson.build | 2 +
src/bin/pg_dump/pg_backup.h | 4 +-
src/bin/pg_dump/pg_backup_archiver.c | 15 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 516 +++++++----------
src/bin/pg_dump/pg_restore.c | 701 ++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 11 +
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 1373 insertions(+), 340 deletions(-)
create mode 100644 src/bin/pg_dump/common_dumpall_restore.c
create mode 100644 src/bin/pg_dump/common_dumpall_restore.h
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 014f2792589..8ca49a65977 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -121,7 +121,83 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- </para>
+ Note: This option can be omitted only when <option>--format</option> is plain
+ </para>
+ </listitem>
+ </varlistentry>
+
+<varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index b8b27e1719e..ba2913b3356 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -166,6 +166,25 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +334,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..a4e557d62c7 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -47,11 +47,11 @@ all: pg_dump pg_restore pg_dumpall
pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dump.o common.o pg_dump_sort.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_restore: pg_restore.o common_dumpall_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_restore.o common_dumpall_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/common_dumpall_restore.c b/src/bin/pg_dump/common_dumpall_restore.c
new file mode 100644
index 00000000000..ace5077085c
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.c
@@ -0,0 +1,314 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.c
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * this is a common file for pg_dumpall and pg_restore.
+ * src/bin/pg_dump/common_dumpall_restore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+#define exit_nicely(code) exit(code)
+
+/*
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the global variable 'connstr' is set to a connection string
+ * containing the options used.
+ */
+PGconn *
+connectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ static char *password = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 6;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. Remember the options used, in the form of a
+ * connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If needed, then copy server version to outer function. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/* ----------
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ * ----------
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/common_dumpall_restore.h b/src/bin/pg_dump/common_dumpall_restore.h
new file mode 100644
index 00000000000..a27c3e9fb89
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.h
+ * Common header file for pg_dumpall and pg_restore
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/common_dumpall_restore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef COMMON_DUMPALL_RESTORE_H
+#define COMMON_DUMPALL_RESTORE_H
+
+#include "pg_backup.h"
+
+extern PGconn *connectDatabase(const char *dbname,
+ const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+extern ArchiveFormat parseDumpFormat(const char *format);
+#endif /* COMMON_DUMPALL_RESTORE_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..ddecac5cf09 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -49,6 +49,7 @@ bin_targets += pg_dump
pg_dumpall_sources = files(
'pg_dumpall.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
@@ -68,6 +69,7 @@ bin_targets += pg_dumpall
pg_restore_sources = files(
'pg_restore.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index f0f19bb0b29..65000e5a083 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -306,7 +306,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
@@ -319,7 +319,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
DataDirSyncMethod sync_method);
/* The --list option */
-extern void PrintTOCSummary(Archive *AHX);
+extern void PrintTOCSummary(Archive *AHX, bool append_data);
extern RestoreOptions *NewRestoreOptions(void);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 707a3fc844c..7153d4a40b6 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -82,7 +82,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -333,7 +333,7 @@ ProcessArchiveRestoreOptions(Archive *AHX)
/* Public */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -450,7 +450,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1263,7 +1263,7 @@ ArchiveEntry(Archive *AHX, CatalogId catalogId, DumpId dumpId,
/* Public */
void
-PrintTOCSummary(Archive *AHX)
+PrintTOCSummary(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -1279,7 +1279,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, append_data);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1658,7 +1658,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1678,7 +1679,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..f70ea9233fe 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -21,7 +21,8 @@
/* Globals exported by this file */
const char *progname = NULL;
-#define MAX_ON_EXIT_NICELY 20
+/* TODO: increasing this to keep 100 db restoring by single pg_restore command. */
+#define MAX_ON_EXIT_NICELY 100
static struct
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 8f73a5df956..eae626f6213 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1147,7 +1147,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 396f79781c5..5915b1b0516 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -24,14 +25,17 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
+#define exit_nicely(code) exit(code)
typedef struct
{
@@ -64,28 +68,24 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -107,7 +107,7 @@ static int no_subscriptions = 0;
static int no_toast_compression = 0;
static int no_unlogged_table_data = 0;
static int no_role_passwords = 0;
-static int server_version;
+static int server_version;
static int load_via_partition_root = 0;
static int on_conflict_do_nothing = 0;
@@ -121,8 +121,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -147,6 +145,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -188,6 +187,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -237,7 +238,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -265,7 +266,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -414,6 +417,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If non-plain format is specified then we must provide the
+ * file name to create one main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -460,6 +478,33 @@ main(int argc, char *argv[])
if (on_conflict_do_nothing)
appendPQExpBufferStr(pgdumpopts, " --on-conflict-do-nothing");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char toc_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -468,7 +513,8 @@ main(int argc, char *argv[])
if (pgdb)
{
conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
@@ -476,10 +522,12 @@ main(int argc, char *argv[])
else
{
conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ prompt_password, true,
+ progname, &connstr, &server_version);
if (!conn)
{
@@ -496,19 +544,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -607,7 +642,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -620,7 +655,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -637,6 +672,8 @@ help(void)
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1487,10 +1524,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1504,7 +1544,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1512,9 +1552,33 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_log_error("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1529,6 +1593,18 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is non-plain dump format, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1547,9 +1623,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1558,19 +1642,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char toc_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(toc_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ toc_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1580,7 +1675,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1589,17 +1685,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is non-plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1649,256 +1764,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
@@ -1994,3 +1859,58 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if(!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 88ae39d938a..79f61395ae3 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,27 +41,67 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/connect.h"
+#include "compress_io.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
+#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
+typedef struct SimpleDatabaseOidListCell
+{
+ struct SimpleDatabaseOidListCell *next;
+ Oid db_oid;
+ const char *db_name;
+} SimpleDatabaseOidListCell;
+
+typedef struct SimpleDatabaseOidList
+{
+ SimpleDatabaseOidListCell *head;
+ SimpleDatabaseOidListCell *tail;
+} SimpleDatabaseOidList;
+
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid, const char *dbname);
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static bool restoreOneDatabase(const char *inputFileSpec,
+ RestoreOptions *opts, int numWorkers, bool append_data);
+static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static void execute_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static int filter_dbnames_for_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list, SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleDatabaseOidList *dbname_oid_list);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname);
+static bool is_full_pattern(PGconn *conn, const char *str, const char *ptrn);
+static void simple_string_full_list_delete(SimpleStringList *list);
+static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
+static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell, SimpleDatabaseOidListCell *prev);
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
static int disable_triggers = 0;
static int enable_row_security = 0;
@@ -77,11 +117,14 @@ main(int argc, char **argv)
static int strict_names = 0;
bool data_only = false;
bool schema_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
+ bool globals_only = false;
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -128,6 +171,7 @@ main(int argc, char **argv)
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -156,7 +200,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "aAcCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -183,11 +227,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -302,6 +349,10 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6:
+ /* list of databases patterns those needs to skip while restoring */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -329,6 +380,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -383,28 +441,80 @@ main(int argc, char **argv)
if (opts->formatName)
{
- switch (opts->formatName[0])
+ opts->format = parseDumpFormat(opts->formatName);
+
+ /* Plain format is not supported for pg_restore. */
+ if (opts->format == archNull)
{
- case 'c':
- case 'C':
- opts->format = archCustom;
- break;
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", or \"t\"",
+ opts->formatName);
+ }
+ }
- case 'd':
- case 'D':
- opts->format = archDirectory;
- break;
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ !IsFileExistsInDirectory(inputFileSpec, "toc.dat"))
+ {
+ /* If global.dat is exist, then process it. */
+ if (IsFileExistsInDirectory(pg_strdup(inputFileSpec), "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql commands. */
+ int exit_code = 0;
- case 't':
- case 'T':
- opts->format = archTar;
- break;
+ /* Connect to database to execute global sql commands from global.dat file. */
+ if (opts->cparams.dbname)
+ {
+ conn = connectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT, false,
+ progname, NULL, NULL);
- default:
- pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", or \"t\"",
- opts->formatName);
- }
- }
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ execute_global_sql_commands(conn, inputFileSpec, opts->filename);
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ if (conn)
+ PQfinish(conn);
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ exit_code = restoreAllDatabases(conn, inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_full_list_delete(&db_exclude_patterns);
+
+ return exit_code;
+ }/* end if */
+ }/* end if */
+
+ return restoreOneDatabase(inputFileSpec, opts, numWorkers, false);
+}
+
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ */
+static bool
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data)
+{
+ Archive *AH;
+ bool exit_code;
AH = OpenArchive(inputFileSpec, opts->format);
@@ -431,11 +541,11 @@ main(int argc, char **argv)
AH->numWorkers = numWorkers;
if (opts->tocSummary)
- PrintTOCSummary(AH);
+ PrintTOCSummary(AH, append_data);
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
/* done, print a summary of ignored errors */
@@ -471,6 +581,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -483,6 +594,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -621,3 +733,542 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * IsFileExistsInDirectory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+IsFileExistsInDirectory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.sql file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+ReadOneStatement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(inBuf, (char) c);
+
+ if (c == '\n')
+ {
+ if(inBuf->len > 1 &&
+ inBuf->data[inBuf->len - 2] == ';')
+ break;
+ else
+ continue;
+ }
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * filter_dbnames_for_restore
+ *
+ * This will remove names from all dblist that are given with exclude-database
+ * option.
+ *
+ * returns number of dbnames those will be restored.
+ */
+static int
+filter_dbnames_for_restore(PGconn *conn, SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
+ SimpleDatabaseOidListCell *dboidprecell = NULL;
+ int count_db = 0;
+
+ /* Return 0 if there is no db to restore. */
+ if (dboid_cell == NULL)
+ return 0;
+
+ /* Process one by one all dbnames and if needs to skip restoring. */
+ while (dboid_cell != NULL)
+ {
+ bool skip_db_restore = false;
+ SimpleDatabaseOidListCell *next = dboid_cell->next;
+
+ /* Now match this dbname with exclude-database list. */
+ for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ {
+ if ((conn && is_full_pattern(conn, dboid_cell->db_name, celldb->val)) ||
+ (!conn && pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0))
+ {
+ /*
+ * As we need to skip this dbname so set flag to remove it from
+ * list.
+ *
+ * Note: we can't remove this pattern from skip list as we
+ * might have multiple database name with this same pattern.
+ */
+ skip_db_restore = true;
+ break;
+ }
+ }
+
+ /* Increment count if db needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
+ simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ }
+ else
+ {
+ count_db++; /* Increment db couter. */
+ dboidprecell = dboid_cell;
+ }
+
+ /* Process next dbname from dbname list. */
+ dboid_cell = next;
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and correspoding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ {
+ pg_log_info("map.dat file is not present in dump of pg_dumpall, so nothing to restore.");
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid;
+ char db_oid_str[MAXPGPATH + 1];
+ char dbname[MAXPGPATH + 1];
+
+ /* Extract dboid. */
+ sscanf(line, "%u" , &db_oid);
+ sscanf(line, "%s" , db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found dbname as : \"%s\" and db_oid:%u in map.dat file while restoring", dbname, db_oid);
+
+ /* Report error if file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
+ SimpleDatabaseOidListCell *dboid_cell;
+ int exit_code = 0;
+ int num_db_restore;
+ int num_total_db;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /* If map.dat has no entry, return from here. */
+ if (dbname_oid_list.head == NULL)
+ return 0;
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect postgres database to dump into out file");
+
+ conn = connectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT, false,
+ progname, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect template1 database as failed to connect to postgres to dump into out file");
+
+ conn = connectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT, false,
+ progname, NULL, NULL);
+
+ if (!conn)
+ pg_log_info("there is no database connection so consider pattern as simple name for exclude-database");
+ }
+ }
+
+ /*
+ * TODO: To skip databases, we need to make a design.
+ *
+ * Skip any explicitly excluded database. If there is no database
+ * connection, then just consider pattern as simple name to compare.
+ */
+ num_db_restore = filter_dbnames_for_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL)
+ return 0;
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * To restore multiple databases, -C (create database) option should be specified
+ * or all databases should be created before pg_restore.
+ */
+ if (opts->createDB != 1)
+ pg_log_info("restoring dump of pg_dumpall without -C option, there might be multiple databases in directory.");
+
+ /* TODO: MAX_ON_EXIT_NICELY is 100 now... max AH handle register on exit .*/
+ if (num_db_restore > 100)
+ {
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+ pg_fatal("cound not restore more than 100 databases by single pg_restore, here total db:%d", num_db_restore);
+ }
+
+ /*
+ * XXX: TODO till now, we made a list of databases, those needs to be restored
+ * after skipping names of exclude-database. Now we can launch parallel
+ * workers to restore these databases.
+ */
+ dboid_cell = dbname_oid_list.head;
+
+ while(dboid_cell != NULL)
+ {
+ char subdirpath[MAXPGPATH];
+ int dbexit_code;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored into
+ * already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+
+ pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+
+ dbexit_code = restoreOneDatabase(subdirpath, opts, numWorkers, true);
+
+ /* Store exit_code to report it back. */
+ if (exit_code == 0 && dbexit_code != 0)
+ exit_code = dbexit_code;
+
+ dboid_cell = dboid_cell->next;
+ } /* end while */
+
+ /* Log number of processed databases.*/
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+
+ return exit_code;
+}
+
+/*
+ * execute_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ */
+static void
+execute_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Now open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ char out_file_path[MAXPGPATH];
+ FILE *ofile;
+ int c;
+
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+
+ ofile = fopen(out_file_path, PG_BINARY_W);
+
+ if (ofile == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", out_file_path);
+ }
+
+ /* Now append global.dat inot outfile. */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ fputc(c, ofile);
+ }
+
+ fclose(pfile);
+ fclose(ofile);
+ return;
+ }
+
+ /* Init sqlstatement to append commands */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements */
+ while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ {
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ fclose(pfile);
+}
+
+/*
+ * simple_db_oid_list_append
+ *
+ * appends a node to the list in the end.
+ */
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid, const char *dbname)
+{
+ SimpleDatabaseOidListCell *cell;
+
+ cell = pg_malloc_object(SimpleDatabaseOidListCell);
+
+ cell->next = NULL;
+ cell->db_oid = db_oid;
+ cell->db_name = pg_strdup(dbname);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * simple_db_oid_full_list_delete
+ *
+ * delete all cell from dbname and dboid list.
+ */
+static void
+simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
+{
+ SimpleDatabaseOidListCell *cell = list->head;
+ SimpleDatabaseOidListCell *nextcell = NULL;
+
+ while (cell)
+ {
+ nextcell = cell->next;
+ pfree (cell);
+ cell = nextcell;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_string_full_list_delete
+ *
+ * delete all cell from string list.
+ */
+static void
+simple_string_full_list_delete(SimpleStringList *list)
+{
+ SimpleStringListCell *cell = list->head;
+ SimpleStringListCell *cellnext = NULL;
+
+ while (cell)
+ {
+ cellnext = cell->next;
+ pfree(cell);
+ cell = cellnext;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_db_oid_list_delete
+ *
+ * delete cell from database and oid list.
+ */
+static void
+simple_db_oid_list_delete(SimpleDatabaseOidList *list, SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev)
+{
+ if (prev == NULL)
+ {
+ list->head = cell->next;
+ pfree(cell);
+ }
+ else
+ {
+ prev->next = cell->next;
+ pfree(cell);
+ }
+}
+
+/*
+ * is_full_pattern
+ *
+ * This uses substring function to make 1st string from pattern.
+ * Outstring of substring function is compared with 1st string to
+ * validate this pattern.
+ *
+ * Returns true if 1st string can be constructed from given pattern.
+ *
+ */
+static bool
+is_full_pattern(PGconn *conn, const char *str, const char *ptrn)
+{
+ PQExpBuffer query;
+ PGresult *result;
+
+ query = createPQExpBuffer();
+
+ printfPQExpBuffer(query,
+ "SELECT substring ( "
+ " '%s' , "
+ " '%s' ) ", str, ptrn);
+
+ result = executeQuery(conn, query->data);
+
+ if (PQresultStatus(result) == PGRES_TUPLES_OK)
+ {
+ if (PQntuples(result) == 1)
+ {
+ const char *outstr = NULL;
+
+ /*
+ * If output string of substring function is matches with str, then
+ * we can construct str from pattern.
+ */
+ if (!PQgetisnull(result, 0, 0))
+ outstr = PQgetvalue(result, 0, 0);
+
+ if (outstr && pg_strcasecmp(outstr, str) == 0)
+ {
+ PQclear(result);
+ destroyPQExpBuffer(query);
+ return true;
+ }
+ }
+ }
+ else
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), query->data);
+
+ PQclear(result);
+ destroyPQExpBuffer(query);
+
+ return false;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 214240f1ae5..96a728adfab
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -219,6 +219,13 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '-p', $port, '-f', $plainfile,
+ "--exclude-database=grabadge",
+ '--globals-only' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -226,4 +233,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d5aa5c295ae..5608512aced 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2672,6 +2672,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.39.3
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-01-23 09:28 ` jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: jian he @ 2025-01-23 09:28 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Guillaume Lelarge <[email protected]>; Nathan Bossart <[email protected]>; Magnus Hagander <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>
hi.
The four patches attached are to solve the
TODO1: We need to think for --exclude-database=PATTERN for pg_restore.
it is based on your v11_pg_dumpall-with-directory-tar-custom-format-21-jan.patch
0001. pg_dumpall --exclude-database=PATTERN already works,
main function resolve pattern matching is expand_dbname_patterns.
make it an extern function, so pg_restore --exclude-database can also use it.
0002 cosmetic code changes not in pg_restore.c
0003 cosmetic code changes in pg_restore.c
0004 fully implement pg_restore --exclude-database=PATTERN
similar to pg_dumpall.c
declare two file static variables:
static SimpleStringList database_exclude_names = {NULL, NULL};
static SimpleStringList db_exclude_patterns = {NULL, NULL};
I also deleted the function is_full_pattern.
I use
$BIN10/pg_restore --exclude-database=*x* --exclude-database=*s*
--exclude-database=*t* --verbose --file=test.sql x1.dump
the verbose message to verify my changes.
Attachments:
[application/octet-stream] v11-0002-minor-coesmetic-change-not-in-pg_restore.c.no-cfbot (2.1K, 2-v11-0002-minor-coesmetic-change-not-in-pg_restore.c.no-cfbot)
download
[application/octet-stream] v11-0004-preliminary-work-for-pg_restore-exclude-datab.no-cfbot (8.7K, 3-v11-0004-preliminary-work-for-pg_restore-exclude-datab.no-cfbot)
download
[application/octet-stream] v11-0001-move-expand_dbname_patterns-to-common_dumpall.no-cfbot (5.4K, 4-v11-0001-move-expand_dbname_patterns-to-common_dumpall.no-cfbot)
download
[application/octet-stream] v11-0003-minor-coesmetic-change-in-pg_restore.c.no-cfbot (5.2K, 5-v11-0003-minor-coesmetic-change-in-pg_restore.c.no-cfbot)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-01-23 10:35 ` Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Mahendra Singh Thalor @ 2025-01-23 10:35 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Guillaume Lelarge <[email protected]>; Nathan Bossart <[email protected]>; Magnus Hagander <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>
On Thu, 23 Jan 2025 at 14:59, jian he <[email protected]> wrote:
>
> hi.
> The four patches attached are to solve the
> TODO1: We need to think for --exclude-database=PATTERN for pg_restore.
> it is based on your v11_pg_dumpall-with-directory-tar-custom-format-21-jan.patch
>
>
> 0001. pg_dumpall --exclude-database=PATTERN already works,
> main function resolve pattern matching is expand_dbname_patterns.
> make it an extern function, so pg_restore --exclude-database can also use it.
Hi Jian,
We can't use the same expand_dbname_patterns function pg_restore.
In the 1st patch, by mistake I also used this function but then I
realised that we should not use this function due to some limitation
for pg_restore.
While doing pg_dumpall, we have all the existence database names in
the pg_database catalog but while restoring, we don't have all
databases in the catalog.
Actually, we will read dbnames from map.dat file to skip matching
patterns for restore.
Ex: let say we have a fresh server with postgres and template1
databases. Now we want to restore one backup
and inside the map.dat file, we have dbname=db_123 and dbname=db_234.
If we want to use --exclude-database=db_123, then
your patch will not work as this db hasn't been created.
Please cross verify again and let me know your feedback.
I think, as of now, mine v11 patch is working as per expectation.
>
> 0002 cosmetic code changes not in pg_restore.c
> 0003 cosmetic code changes in pg_restore.c
>
>
> 0004 fully implement pg_restore --exclude-database=PATTERN
> similar to pg_dumpall.c
> declare two file static variables:
> static SimpleStringList database_exclude_names = {NULL, NULL};
> static SimpleStringList db_exclude_patterns = {NULL, NULL};
> I also deleted the function is_full_pattern.
>
>
> I use
> $BIN10/pg_restore --exclude-database=*x* --exclude-database=*s*
> --exclude-database=*t* --verbose --file=test.sql x1.dump
> the verbose message to verify my changes.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-01-24 15:20 ` jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: jian he @ 2025-01-24 15:20 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Guillaume Lelarge <[email protected]>; Nathan Bossart <[email protected]>; Magnus Hagander <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>
On Thu, Jan 23, 2025 at 6:35 PM Mahendra Singh Thalor
<[email protected]> wrote:
>
> On Thu, 23 Jan 2025 at 14:59, jian he <[email protected]> wrote:
> >
> > hi.
> > The four patches attached are to solve the
> > TODO1: We need to think for --exclude-database=PATTERN for pg_restore.
> > it is based on your v11_pg_dumpall-with-directory-tar-custom-format-21-jan.patch
> >
> >
> > 0001. pg_dumpall --exclude-database=PATTERN already works,
> > main function resolve pattern matching is expand_dbname_patterns.
> > make it an extern function, so pg_restore --exclude-database can also use it.
>
> Hi Jian,
> We can't use the same expand_dbname_patterns function pg_restore.
>
> In the 1st patch, by mistake I also used this function but then I
> realised that we should not use this function due to some limitation
> for pg_restore.
>
> While doing pg_dumpall, we have all the existence database names in
> the pg_database catalog but while restoring, we don't have all
> databases in the catalog.
> Actually, we will read dbnames from map.dat file to skip matching
> patterns for restore.
>
hi.
After some tests and thinking about your reply, I admit that using
expand_dbname_patterns
in pg_restore will not work.
We need to do pattern matching against the map.dat file.
Please check the attached v12 series based on your
v11_pg_dumpall-with-directory-tar-custom-format-21-jan.patch
v12-0001 cosmetic change.
v12-0002 implement pg_resore --exclude-database=PATTERN.
main gist of implementation:
for each database name in map.dat file,
check if this database name pattern matches with PATTERN or not.
pattern matching is using processSQLNamePattern.
your substring will not work.
some of the test cases.
$BIN10/pg_restore --exclude-database=* -Cd template1 --verbose dir10 >
dir_format 2>&1
$BIN10/pg_restore --exclude-database=*x* -Cd template1 --verbose dir10
> dir_format 2>&1
$BIN10/pg_restore --exclude-database=?* -Cd template1 --verbose dir10
> dir_format 2>&1
Attachments:
[application/octet-stream] v12-0002-pg_restore-exclude-database-PATTERN.no-cfbot (10.6K, 2-v12-0002-pg_restore-exclude-database-PATTERN.no-cfbot)
download
[application/octet-stream] v12-0001-coesmetic-change.no-cfbot (7.0K, 3-v12-0001-coesmetic-change.no-cfbot)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-01-26 14:46 ` jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: jian he @ 2025-01-26 14:46 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Guillaume Lelarge <[email protected]>; Nathan Bossart <[email protected]>; Magnus Hagander <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>
hi.
attached patching trying to refactor ReadOneStatement
for properly handling the single and double quotes.
the commit message also has some tests on it.
it is based on your
v11_pg_dumpall-with-directory-tar-custom-format-21-jan.patch.
Attachments:
[application/octet-stream] v11-0001-refactoring-ReadOneStatement.no-cfbot (2.5K, 2-v11-0001-refactoring-ReadOneStatement.no-cfbot)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-01-26 15:48 ` Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Mahendra Singh Thalor @ 2025-01-26 15:48 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Guillaume Lelarge <[email protected]>; Nathan Bossart <[email protected]>; Magnus Hagander <[email protected]>; Tom Lane <[email protected]>; Andrew Dunstan <[email protected]>; pgsql-hackers; Dilip Kumar <[email protected]>
> hi.
> After some tests and thinking about your reply, I admit that using
> expand_dbname_patterns
> in pg_restore will not work.
> We need to do pattern matching against the map.dat file.
> Please check the attached v12 series based on your
> v11_pg_dumpall-with-directory-tar-custom-format-21-jan.patch
>
> v12-0001 cosmetic change.
> v12-0002 implement pg_resore --exclude-database=PATTERN.
> main gist of implementation:
> for each database name in map.dat file,
> check if this database name pattern matches with PATTERN or not.
> pattern matching is using processSQLNamePattern.
>
> your substring will not work.
> some of the test cases.
> $BIN10/pg_restore --exclude-database=* -Cd template1 --verbose dir10 >
> dir_format 2>&1
Hi,
As per discussion with Robert Haas and Dilip Kumar, we thought that we
can't assume that
there will be a db connection every time while doing pg_restore but in
attached patch, we are
assuming that we have a db connection.
In my previous updates, I already mentioned this problem. I think, we
should not use connection
for --exclude-database, rather we should use direct functions to
validate patterns or we should
restrict as NAME only.
On Sun, 26 Jan 2025 at 20:17, jian he <[email protected]> wrote:
>
> hi.
> attached patching trying to refactor ReadOneStatement
> for properly handling the single and double quotes.
> the commit message also has some tests on it.
>
> it is based on your
> v11_pg_dumpall-with-directory-tar-custom-format-21-jan.patch.
I think, instead of char, if we read line by line, then we don't need
that much code and need not to worry about double quotes.
In the next version, I will merge some patches and will change it to
read line by line.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-01-28 04:49 ` Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Srinath Reddy @ 2025-01-28 04:49 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; [email protected]
Hi mahendra,
I have reviewed the code in the v11 patch and it looks good to me.
But in common_dumpall_restore.c there's parseDumpFormat which is common
between pg_dumpall and pg_restore ,as per the discussion in [1] thread i
don't think we should create a common api ,as discussed in the thread there
might chances in the future we might decide that some format is obsolete
and desupport it in pg_dumpall ,while support in pg_restore for
compatibility reasons.
[1]
https://www.postgresql.org/message-id/flat/CAFC%2Bb6pfK-BGcWW1kQmtxVrCh-JGjB2X02rLPQs_ZFaDGjZDsQ%40m...
Regards,
Srinath Reddy Sadipiralla,
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
@ 2025-01-28 06:21 ` Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Srinath Reddy @ 2025-01-28 06:21 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; [email protected]
make check-world fails,i think we don't need $port and $filename instead we
can use something like 'xxx'.so fixed it in the below patch.
Regards,
Srinath Reddy Sadipiralla,
EDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v1-fix-make-check-world.patch (68.6K, 3-v1-fix-make-check-world.patch)
download | inline diff:
From 7666c6d2321b696c5b4a7dd4c4c80f7b12b22813 Mon Sep 17 00:00:00 2001
From: Srinath Reddy Sadipiralla <[email protected]>
Date: Tue, 28 Jan 2025 11:41:14 +0530
Subject: [PATCH 1/1] Refactor command_fails_like in pg_dump/t/001_basic.pl
to pass make check-world.
---
doc/src/sgml/ref/pg_dumpall.sgml | 78 ++-
doc/src/sgml/ref/pg_restore.sgml | 29 +
src/bin/pg_dump/Makefile | 8 +-
src/bin/pg_dump/common_dumpall_restore.c | 314 ++++++++++
src/bin/pg_dump/common_dumpall_restore.h | 26 +
src/bin/pg_dump/meson.build | 2 +
src/bin/pg_dump/pg_backup.h | 4 +-
src/bin/pg_dump/pg_backup_archiver.c | 15 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 516 +++++++----------
src/bin/pg_dump/pg_restore.c | 701 ++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 11 +
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 1373 insertions(+), 340 deletions(-)
create mode 100644 src/bin/pg_dump/common_dumpall_restore.c
create mode 100644 src/bin/pg_dump/common_dumpall_restore.h
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 014f279258..8ca49a6597 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -121,7 +121,83 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- </para>
+ Note: This option can be omitted only when <option>--format</option> is plain
+ </para>
+ </listitem>
+ </varlistentry>
+
+<varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index b8b27e1719..ba2913b335 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -166,6 +166,25 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +334,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca7..a4e557d62c 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -47,11 +47,11 @@ all: pg_dump pg_restore pg_dumpall
pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dump.o common.o pg_dump_sort.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_restore: pg_restore.o common_dumpall_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_restore.o common_dumpall_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/common_dumpall_restore.c b/src/bin/pg_dump/common_dumpall_restore.c
new file mode 100644
index 0000000000..ace5077085
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.c
@@ -0,0 +1,314 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.c
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * this is a common file for pg_dumpall and pg_restore.
+ * src/bin/pg_dump/common_dumpall_restore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+#define exit_nicely(code) exit(code)
+
+/*
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the global variable 'connstr' is set to a connection string
+ * containing the options used.
+ */
+PGconn *
+connectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ static char *password = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 6;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. Remember the options used, in the form of a
+ * connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If needed, then copy server version to outer function. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/* ----------
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ * ----------
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/common_dumpall_restore.h b/src/bin/pg_dump/common_dumpall_restore.h
new file mode 100644
index 0000000000..a27c3e9fb8
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.h
+ * Common header file for pg_dumpall and pg_restore
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/common_dumpall_restore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef COMMON_DUMPALL_RESTORE_H
+#define COMMON_DUMPALL_RESTORE_H
+
+#include "pg_backup.h"
+
+extern PGconn *connectDatabase(const char *dbname,
+ const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+extern ArchiveFormat parseDumpFormat(const char *format);
+#endif /* COMMON_DUMPALL_RESTORE_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf..ddecac5cf0 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -49,6 +49,7 @@ bin_targets += pg_dump
pg_dumpall_sources = files(
'pg_dumpall.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
@@ -68,6 +69,7 @@ bin_targets += pg_dumpall
pg_restore_sources = files(
'pg_restore.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index f0f19bb0b2..65000e5a08 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -306,7 +306,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
@@ -319,7 +319,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
DataDirSyncMethod sync_method);
/* The --list option */
-extern void PrintTOCSummary(Archive *AHX);
+extern void PrintTOCSummary(Archive *AHX, bool append_data);
extern RestoreOptions *NewRestoreOptions(void);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 707a3fc844..7153d4a40b 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -82,7 +82,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -333,7 +333,7 @@ ProcessArchiveRestoreOptions(Archive *AHX)
/* Public */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -450,7 +450,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1263,7 +1263,7 @@ ArchiveEntry(Archive *AHX, CatalogId catalogId, DumpId dumpId,
/* Public */
void
-PrintTOCSummary(Archive *AHX)
+PrintTOCSummary(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -1279,7 +1279,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, append_data);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1658,7 +1658,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1678,7 +1679,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd..d94d0de2a5 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f515..f70ea9233f 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -21,7 +21,8 @@
/* Globals exported by this file */
const char *progname = NULL;
-#define MAX_ON_EXIT_NICELY 20
+/* TODO: increasing this to keep 100 db restoring by single pg_restore command. */
+#define MAX_ON_EXIT_NICELY 100
static struct
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index af857f00c7..2de2621c8f 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1148,7 +1148,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 396f79781c..5915b1b051 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -24,14 +25,17 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
+#define exit_nicely(code) exit(code)
typedef struct
{
@@ -64,28 +68,24 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -107,7 +107,7 @@ static int no_subscriptions = 0;
static int no_toast_compression = 0;
static int no_unlogged_table_data = 0;
static int no_role_passwords = 0;
-static int server_version;
+static int server_version;
static int load_via_partition_root = 0;
static int on_conflict_do_nothing = 0;
@@ -121,8 +121,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -147,6 +145,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -188,6 +187,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -237,7 +238,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -265,7 +266,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -414,6 +417,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If non-plain format is specified then we must provide the
+ * file name to create one main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -460,6 +478,33 @@ main(int argc, char *argv[])
if (on_conflict_do_nothing)
appendPQExpBufferStr(pgdumpopts, " --on-conflict-do-nothing");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char toc_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -468,7 +513,8 @@ main(int argc, char *argv[])
if (pgdb)
{
conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
@@ -476,10 +522,12 @@ main(int argc, char *argv[])
else
{
conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ prompt_password, true,
+ progname, &connstr, &server_version);
if (!conn)
{
@@ -496,19 +544,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -607,7 +642,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -620,7 +655,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -637,6 +672,8 @@ help(void)
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1487,10 +1524,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1504,7 +1544,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1512,9 +1552,33 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_log_error("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1529,6 +1593,18 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is non-plain dump format, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1547,9 +1623,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1558,19 +1642,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char toc_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(toc_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ toc_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1580,7 +1675,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1589,17 +1685,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is non-plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1649,256 +1764,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
@@ -1994,3 +1859,58 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if(!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 88ae39d938..79f61395ae 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,27 +41,67 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/connect.h"
+#include "compress_io.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
+#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
+typedef struct SimpleDatabaseOidListCell
+{
+ struct SimpleDatabaseOidListCell *next;
+ Oid db_oid;
+ const char *db_name;
+} SimpleDatabaseOidListCell;
+
+typedef struct SimpleDatabaseOidList
+{
+ SimpleDatabaseOidListCell *head;
+ SimpleDatabaseOidListCell *tail;
+} SimpleDatabaseOidList;
+
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid, const char *dbname);
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static bool restoreOneDatabase(const char *inputFileSpec,
+ RestoreOptions *opts, int numWorkers, bool append_data);
+static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static void execute_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static int filter_dbnames_for_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list, SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleDatabaseOidList *dbname_oid_list);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname);
+static bool is_full_pattern(PGconn *conn, const char *str, const char *ptrn);
+static void simple_string_full_list_delete(SimpleStringList *list);
+static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
+static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell, SimpleDatabaseOidListCell *prev);
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
static int disable_triggers = 0;
static int enable_row_security = 0;
@@ -77,11 +117,14 @@ main(int argc, char **argv)
static int strict_names = 0;
bool data_only = false;
bool schema_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
+ bool globals_only = false;
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -128,6 +171,7 @@ main(int argc, char **argv)
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -156,7 +200,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "aAcCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -183,11 +227,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -302,6 +349,10 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6:
+ /* list of databases patterns those needs to skip while restoring */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -329,6 +380,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -383,28 +441,80 @@ main(int argc, char **argv)
if (opts->formatName)
{
- switch (opts->formatName[0])
+ opts->format = parseDumpFormat(opts->formatName);
+
+ /* Plain format is not supported for pg_restore. */
+ if (opts->format == archNull)
{
- case 'c':
- case 'C':
- opts->format = archCustom;
- break;
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", or \"t\"",
+ opts->formatName);
+ }
+ }
- case 'd':
- case 'D':
- opts->format = archDirectory;
- break;
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ !IsFileExistsInDirectory(inputFileSpec, "toc.dat"))
+ {
+ /* If global.dat is exist, then process it. */
+ if (IsFileExistsInDirectory(pg_strdup(inputFileSpec), "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql commands. */
+ int exit_code = 0;
- case 't':
- case 'T':
- opts->format = archTar;
- break;
+ /* Connect to database to execute global sql commands from global.dat file. */
+ if (opts->cparams.dbname)
+ {
+ conn = connectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT, false,
+ progname, NULL, NULL);
- default:
- pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", or \"t\"",
- opts->formatName);
- }
- }
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ execute_global_sql_commands(conn, inputFileSpec, opts->filename);
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ if (conn)
+ PQfinish(conn);
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ exit_code = restoreAllDatabases(conn, inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_full_list_delete(&db_exclude_patterns);
+
+ return exit_code;
+ }/* end if */
+ }/* end if */
+
+ return restoreOneDatabase(inputFileSpec, opts, numWorkers, false);
+}
+
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ */
+static bool
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data)
+{
+ Archive *AH;
+ bool exit_code;
AH = OpenArchive(inputFileSpec, opts->format);
@@ -431,11 +541,11 @@ main(int argc, char **argv)
AH->numWorkers = numWorkers;
if (opts->tocSummary)
- PrintTOCSummary(AH);
+ PrintTOCSummary(AH, append_data);
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
/* done, print a summary of ignored errors */
@@ -471,6 +581,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -483,6 +594,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -621,3 +733,542 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * IsFileExistsInDirectory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+IsFileExistsInDirectory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.sql file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+ReadOneStatement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(inBuf, (char) c);
+
+ if (c == '\n')
+ {
+ if(inBuf->len > 1 &&
+ inBuf->data[inBuf->len - 2] == ';')
+ break;
+ else
+ continue;
+ }
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * filter_dbnames_for_restore
+ *
+ * This will remove names from all dblist that are given with exclude-database
+ * option.
+ *
+ * returns number of dbnames those will be restored.
+ */
+static int
+filter_dbnames_for_restore(PGconn *conn, SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
+ SimpleDatabaseOidListCell *dboidprecell = NULL;
+ int count_db = 0;
+
+ /* Return 0 if there is no db to restore. */
+ if (dboid_cell == NULL)
+ return 0;
+
+ /* Process one by one all dbnames and if needs to skip restoring. */
+ while (dboid_cell != NULL)
+ {
+ bool skip_db_restore = false;
+ SimpleDatabaseOidListCell *next = dboid_cell->next;
+
+ /* Now match this dbname with exclude-database list. */
+ for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ {
+ if ((conn && is_full_pattern(conn, dboid_cell->db_name, celldb->val)) ||
+ (!conn && pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0))
+ {
+ /*
+ * As we need to skip this dbname so set flag to remove it from
+ * list.
+ *
+ * Note: we can't remove this pattern from skip list as we
+ * might have multiple database name with this same pattern.
+ */
+ skip_db_restore = true;
+ break;
+ }
+ }
+
+ /* Increment count if db needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
+ simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ }
+ else
+ {
+ count_db++; /* Increment db couter. */
+ dboidprecell = dboid_cell;
+ }
+
+ /* Process next dbname from dbname list. */
+ dboid_cell = next;
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and correspoding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ {
+ pg_log_info("map.dat file is not present in dump of pg_dumpall, so nothing to restore.");
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid;
+ char db_oid_str[MAXPGPATH + 1];
+ char dbname[MAXPGPATH + 1];
+
+ /* Extract dboid. */
+ sscanf(line, "%u" , &db_oid);
+ sscanf(line, "%s" , db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found dbname as : \"%s\" and db_oid:%u in map.dat file while restoring", dbname, db_oid);
+
+ /* Report error if file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
+ SimpleDatabaseOidListCell *dboid_cell;
+ int exit_code = 0;
+ int num_db_restore;
+ int num_total_db;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /* If map.dat has no entry, return from here. */
+ if (dbname_oid_list.head == NULL)
+ return 0;
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect postgres database to dump into out file");
+
+ conn = connectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT, false,
+ progname, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect template1 database as failed to connect to postgres to dump into out file");
+
+ conn = connectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT, false,
+ progname, NULL, NULL);
+
+ if (!conn)
+ pg_log_info("there is no database connection so consider pattern as simple name for exclude-database");
+ }
+ }
+
+ /*
+ * TODO: To skip databases, we need to make a design.
+ *
+ * Skip any explicitly excluded database. If there is no database
+ * connection, then just consider pattern as simple name to compare.
+ */
+ num_db_restore = filter_dbnames_for_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL)
+ return 0;
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * To restore multiple databases, -C (create database) option should be specified
+ * or all databases should be created before pg_restore.
+ */
+ if (opts->createDB != 1)
+ pg_log_info("restoring dump of pg_dumpall without -C option, there might be multiple databases in directory.");
+
+ /* TODO: MAX_ON_EXIT_NICELY is 100 now... max AH handle register on exit .*/
+ if (num_db_restore > 100)
+ {
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+ pg_fatal("cound not restore more than 100 databases by single pg_restore, here total db:%d", num_db_restore);
+ }
+
+ /*
+ * XXX: TODO till now, we made a list of databases, those needs to be restored
+ * after skipping names of exclude-database. Now we can launch parallel
+ * workers to restore these databases.
+ */
+ dboid_cell = dbname_oid_list.head;
+
+ while(dboid_cell != NULL)
+ {
+ char subdirpath[MAXPGPATH];
+ int dbexit_code;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored into
+ * already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+
+ pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+
+ dbexit_code = restoreOneDatabase(subdirpath, opts, numWorkers, true);
+
+ /* Store exit_code to report it back. */
+ if (exit_code == 0 && dbexit_code != 0)
+ exit_code = dbexit_code;
+
+ dboid_cell = dboid_cell->next;
+ } /* end while */
+
+ /* Log number of processed databases.*/
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+
+ return exit_code;
+}
+
+/*
+ * execute_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ */
+static void
+execute_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Now open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ char out_file_path[MAXPGPATH];
+ FILE *ofile;
+ int c;
+
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+
+ ofile = fopen(out_file_path, PG_BINARY_W);
+
+ if (ofile == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", out_file_path);
+ }
+
+ /* Now append global.dat inot outfile. */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ fputc(c, ofile);
+ }
+
+ fclose(pfile);
+ fclose(ofile);
+ return;
+ }
+
+ /* Init sqlstatement to append commands */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements */
+ while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ {
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ fclose(pfile);
+}
+
+/*
+ * simple_db_oid_list_append
+ *
+ * appends a node to the list in the end.
+ */
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid, const char *dbname)
+{
+ SimpleDatabaseOidListCell *cell;
+
+ cell = pg_malloc_object(SimpleDatabaseOidListCell);
+
+ cell->next = NULL;
+ cell->db_oid = db_oid;
+ cell->db_name = pg_strdup(dbname);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * simple_db_oid_full_list_delete
+ *
+ * delete all cell from dbname and dboid list.
+ */
+static void
+simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
+{
+ SimpleDatabaseOidListCell *cell = list->head;
+ SimpleDatabaseOidListCell *nextcell = NULL;
+
+ while (cell)
+ {
+ nextcell = cell->next;
+ pfree (cell);
+ cell = nextcell;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_string_full_list_delete
+ *
+ * delete all cell from string list.
+ */
+static void
+simple_string_full_list_delete(SimpleStringList *list)
+{
+ SimpleStringListCell *cell = list->head;
+ SimpleStringListCell *cellnext = NULL;
+
+ while (cell)
+ {
+ cellnext = cell->next;
+ pfree(cell);
+ cell = cellnext;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_db_oid_list_delete
+ *
+ * delete cell from database and oid list.
+ */
+static void
+simple_db_oid_list_delete(SimpleDatabaseOidList *list, SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev)
+{
+ if (prev == NULL)
+ {
+ list->head = cell->next;
+ pfree(cell);
+ }
+ else
+ {
+ prev->next = cell->next;
+ pfree(cell);
+ }
+}
+
+/*
+ * is_full_pattern
+ *
+ * This uses substring function to make 1st string from pattern.
+ * Outstring of substring function is compared with 1st string to
+ * validate this pattern.
+ *
+ * Returns true if 1st string can be constructed from given pattern.
+ *
+ */
+static bool
+is_full_pattern(PGconn *conn, const char *str, const char *ptrn)
+{
+ PQExpBuffer query;
+ PGresult *result;
+
+ query = createPQExpBuffer();
+
+ printfPQExpBuffer(query,
+ "SELECT substring ( "
+ " '%s' , "
+ " '%s' ) ", str, ptrn);
+
+ result = executeQuery(conn, query->data);
+
+ if (PQresultStatus(result) == PGRES_TUPLES_OK)
+ {
+ if (PQntuples(result) == 1)
+ {
+ const char *outstr = NULL;
+
+ /*
+ * If output string of substring function is matches with str, then
+ * we can construct str from pattern.
+ */
+ if (!PQgetisnull(result, 0, 0))
+ outstr = PQgetvalue(result, 0, 0);
+
+ if (outstr && pg_strcasecmp(outstr, str) == 0)
+ {
+ PQclear(result);
+ destroyPQExpBuffer(query);
+ return true;
+ }
+ }
+ }
+ else
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), query->data);
+
+ PQclear(result);
+ destroyPQExpBuffer(query);
+
+ return false;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 214240f1ae..20ddf3646a
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -219,6 +219,13 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '-p', 'xxx', '-f', 'xxx',
+ "--exclude-database=grabadge",
+ '--globals-only' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -226,4 +233,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index a2644a2e65..96f460d2e3 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2673,6 +2673,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.43.0
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
@ 2025-01-28 06:27 ` Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Mahendra Singh Thalor @ 2025-01-28 06:27 UTC (permalink / raw)
To: Srinath Reddy <[email protected]>; +Cc: [email protected]
On Tue, 28 Jan 2025 at 10:19, Srinath Reddy <[email protected]> wrote:
>
>
> Hi mahendra,
>
> I have reviewed the code in the v11 patch and it looks good to me.
>
> But in common_dumpall_restore.c there's parseDumpFormat which is common between pg_dumpall and pg_restore ,as per the discussion in [1] thread i don't think we should create a common api ,as discussed in the thread there might chances in the future we might decide that some format is obsolete and desupport it in pg_dumpall ,while support in pg_restore for compatibility reasons.
>
Oaky. Thanks for review. I will make changes as per discussion in
another thread.
On Tue, 28 Jan 2025 at 11:52, Srinath Reddy <[email protected]> wrote:
>
> make check-world fails,i think we don't need $port and $filename instead we can use something like 'xxx'.so fixed it in the below patch.
In offline discussion, Andew already reported this test case. I will
fix this in the next version.
>
> Regards,
> Srinath Reddy Sadipiralla,
> EDB: http://www.enterprisedb.com
>
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-01-28 12:02 ` Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Mahendra Singh Thalor @ 2025-01-28 12:02 UTC (permalink / raw)
To: Srinath Reddy <[email protected]>; +Cc: [email protected]
On Fri, 24 Jan 2025 at 20:50, jian he <[email protected]> wrote:
>
> On Thu, Jan 23, 2025 at 6:35 PM Mahendra Singh Thalor
> <[email protected]> wrote:
>
> hi.
> After some tests and thinking about your reply, I admit that using
> expand_dbname_patterns
> in pg_restore will not work.
> We need to do pattern matching against the map.dat file.
> Please check the attached v12 series based on your
> v11_pg_dumpall-with-directory-tar-custom-format-21-jan.patch
>
> v12-0001 cosmetic change.
> v12-0002 implement pg_resore --exclude-database=PATTERN.
> main gist of implementation:
> for each database name in map.dat file,
> check if this database name pattern matches with PATTERN or not.
> pattern matching is using processSQLNamePattern.
>
> your substring will not work.
> some of the test cases.
> $BIN10/pg_restore --exclude-database=* -Cd template1 --verbose dir10 >
> dir_format 2>&1
> $BIN10/pg_restore --exclude-database=*x* -Cd template1 --verbose dir10
> > dir_format 2>&1
> $BIN10/pg_restore --exclude-database=?* -Cd template1 --verbose dir10
> > dir_format 2>&1
I merged v12_0001 into the latest patch. There was one bug in v12_001*
which was fixed in v12_0002*.
-extern PGconn **connectDatabase*(const char *dbname,
> - const char *connection_string, const char *pghost,
> - const char *pgport, const char *pguser,
> - trivalue prompt_password, bool fail_on_error,
> - const char *progname, const char **connstr, int *server_version);
> +extern PGconn *v(const char *dbname, const char *connection_string, const
> char *pghost,
As per v12_0002*, I made some changes into the current patch to avoid using
the substring function.
On Tue, 28 Jan 2025 at 11:57, Mahendra Singh Thalor <[email protected]>
wrote:
>
> On Tue, 28 Jan 2025 at 10:19, Srinath Reddy <[email protected]> wrote:
> >
> >
> > Hi mahendra,
> >
> > I have reviewed the code in the v11 patch and it looks good to me.
> >
> > But in common_dumpall_restore.c there's parseDumpFormat which is
common between pg_dumpall and pg_restore ,as per the discussion in [1]
thread i don't think we should create a common api ,as discussed in the
thread there might chances in the future we might decide that some format
is obsolete and desupport it in pg_dumpall ,while support in pg_restore for
compatibility reasons.
Fixed. In the latest patch, I removed the parseDumpFormat function.
In older versions, I was using the same function for pg_dumpall and
pg_restore but now some common code is already committed from this patch
and as per discussion, we will keep separate handling for parsing so adding
parseDumpFormat function only in pg_dumpall.c file.
On Sun, 26 Jan 2025 at 20:17, jian he <[email protected]> wrote:
>
> hi.
> attached patching trying to refactor ReadOneStatement
> for properly handling the single and double quotes.
> the commit message also has some tests on it.
>
> it is based on your
> v11_pg_dumpall-with-directory-tar-custom-format-21-jan.patch.
Okay. I am doing some more testing and code review for this type of test
cases. I will merge this delta into the next version.
> >
>
> Oaky. Thanks for review. I will make changes as per discussion in
> another thread.
>
>
> On Tue, 28 Jan 2025 at 11:52, Srinath Reddy <[email protected]> wrote:
> >
> > make check-world fails,i think we don't need $port and $filename
instead we can use something like 'xxx'.so fixed it in the below patch.
>
> In offline discussion, Andew already reported this test case. I will
> fix this in the next version.
>
Fixed.
Thanks Jian and Srinath for the testing and review.
Here, I am attaching an updated patch for review and testing.
I merged some of the delta patches that are shared by Jian and did some
fixes also.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v13_pg_dumpall-with-directory-tar-custom-format-28-jan.patch (70.9K, 3-v13_pg_dumpall-with-directory-tar-custom-format-28-jan.patch)
download | inline diff:
From e422aa60c99c1fc44578741594c52c30359b112f Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Tue, 28 Jan 2025 03:33:51 -0800
Subject: [PATCH] pg_dumpall with directory|tar|custom format and restore it
by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat and map.dat to restore all databases. If both files are exists in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list.
TODO1: We need to think for --exclude-database=PATTERN for pg_restore.
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
TODO2: We need to make changes for exit_nicely as we are one entry for each database while
restoring. MAX_ON_EXIT_NICELY
TODO3: some more test cases for new added options.
TODO4: We can dump and restore databases in parallel mode.
This needs more study
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
:q
---
doc/src/sgml/ref/pg_dumpall.sgml | 78 +++-
doc/src/sgml/ref/pg_restore.sgml | 29 ++
src/bin/pg_dump/Makefile | 8 +-
src/bin/pg_dump/common_dumpall_restore.c | 281 ++++++++++++
src/bin/pg_dump/common_dumpall_restore.h | 24 +
src/bin/pg_dump/meson.build | 2 +
src/bin/pg_dump/pg_backup.h | 4 +-
src/bin/pg_dump/pg_backup_archiver.c | 15 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 550 +++++++++++------------
src/bin/pg_dump/pg_restore.c | 728 ++++++++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 11 +-
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 1416 insertions(+), 323 deletions(-)
create mode 100644 src/bin/pg_dump/common_dumpall_restore.c
create mode 100644 src/bin/pg_dump/common_dumpall_restore.h
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 39d93c2..4298083 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -121,7 +121,83 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- </para>
+ Note: This option can be omitted only when <option>--format</option> is plain
+ </para>
+ </listitem>
+ </varlistentry>
+
+<varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index b8b27e1..ba2913b 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -167,6 +167,25 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
<listitem>
@@ -316,6 +335,16 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15..a4e557d 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -47,11 +47,11 @@ all: pg_dump pg_restore pg_dumpall
pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dump.o common.o pg_dump_sort.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_restore: pg_restore.o common_dumpall_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_restore.o common_dumpall_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/common_dumpall_restore.c b/src/bin/pg_dump/common_dumpall_restore.c
new file mode 100644
index 0000000..446f325
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.c
@@ -0,0 +1,281 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.c
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * this is a common file for pg_dumpall and pg_restore.
+ * src/bin/pg_dump/common_dumpall_restore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+#define exit_nicely(code) exit(code)
+
+/*
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the global variable 'connstr' is set to a connection string
+ * containing the options used.
+ */
+PGconn *
+connectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ static char *password = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 6;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. Remember the options used, in the form of a
+ * connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If needed, then copy server version to outer function. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/* ----------
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ * ----------
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
diff --git a/src/bin/pg_dump/common_dumpall_restore.h b/src/bin/pg_dump/common_dumpall_restore.h
new file mode 100644
index 0000000..7fe1c00
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.h
@@ -0,0 +1,24 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.h
+ * Common header file for pg_dumpall and pg_restore
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/common_dumpall_restore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef COMMON_DUMPALL_RESTORE_H
+#define COMMON_DUMPALL_RESTORE_H
+
+#include "pg_backup.h"
+
+extern PGconn *connectDatabase(const char *dbname, const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+#endif /* COMMON_DUMPALL_RESTORE_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6c..ddecac5 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -49,6 +49,7 @@ bin_targets += pg_dump
pg_dumpall_sources = files(
'pg_dumpall.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
@@ -68,6 +69,7 @@ bin_targets += pg_dumpall
pg_restore_sources = files(
'pg_restore.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index f0f19bb..65000e5 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -306,7 +306,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
@@ -319,7 +319,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
DataDirSyncMethod sync_method);
/* The --list option */
-extern void PrintTOCSummary(Archive *AHX);
+extern void PrintTOCSummary(Archive *AHX, bool append_data);
extern RestoreOptions *NewRestoreOptions(void);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 707a3fc..7153d4a 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -82,7 +82,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -333,7 +333,7 @@ ProcessArchiveRestoreOptions(Archive *AHX)
/* Public */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -450,7 +450,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1263,7 +1263,7 @@ ArchiveEntry(Archive *AHX, CatalogId catalogId, DumpId dumpId,
/* Public */
void
-PrintTOCSummary(Archive *AHX)
+PrintTOCSummary(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -1279,7 +1279,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, append_data);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1658,7 +1658,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1678,7 +1679,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b4..d94d0de 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f..f70ea92 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -21,7 +21,8 @@
/* Globals exported by this file */
const char *progname = NULL;
-#define MAX_ON_EXIT_NICELY 20
+/* TODO: increasing this to keep 100 db restoring by single pg_restore command. */
+#define MAX_ON_EXIT_NICELY 100
static struct
{
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 02e1fdf..61067e1 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1148,7 +1148,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 396f797..3e022ec 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -24,14 +25,17 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
+#define exit_nicely(code) exit(code)
typedef struct
{
@@ -64,28 +68,25 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -107,7 +108,7 @@ static int no_subscriptions = 0;
static int no_toast_compression = 0;
static int no_unlogged_table_data = 0;
static int no_role_passwords = 0;
-static int server_version;
+static int server_version;
static int load_via_partition_root = 0;
static int on_conflict_do_nothing = 0;
@@ -121,8 +122,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -147,6 +146,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -188,6 +188,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -237,7 +239,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -265,7 +267,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -414,6 +418,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If non-plain format is specified then we must provide the
+ * file name to create one main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -461,6 +480,33 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " --on-conflict-do-nothing");
/*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char toc_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
+ /*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
* "template1".
@@ -468,7 +514,8 @@ main(int argc, char *argv[])
if (pgdb)
{
conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
@@ -476,10 +523,12 @@ main(int argc, char *argv[])
else
{
conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ prompt_password, true,
+ progname, &connstr, &server_version);
if (!conn)
{
@@ -497,19 +546,6 @@ main(int argc, char *argv[])
&database_exclude_names);
/*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
- /*
* Set the client encoding if requested.
*/
if (dumpencoding)
@@ -607,7 +643,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -620,7 +656,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -637,6 +673,8 @@ help(void)
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1487,10 +1525,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1504,7 +1545,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1512,9 +1553,33 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_log_error("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1529,6 +1594,18 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is non-plain dump format, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1547,9 +1624,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1558,19 +1643,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char toc_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(toc_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ toc_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1580,7 +1676,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1589,17 +1686,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is non-plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1650,256 +1766,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
}
/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
-/*
* As above for a SQL command (which returns nothing).
*/
static void
@@ -1994,3 +1860,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if(!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c602272..fc248a4 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,27 +41,69 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/connect.h"
+#include "compress_io.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
+#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
+typedef struct SimpleDatabaseOidListCell
+{
+ struct SimpleDatabaseOidListCell *next;
+ Oid db_oid;
+ const char *db_name;
+} SimpleDatabaseOidListCell;
+
+typedef struct SimpleDatabaseOidList
+{
+ SimpleDatabaseOidListCell *head;
+ SimpleDatabaseOidListCell *tail;
+} SimpleDatabaseOidList;
+
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid, const char *dbname);
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static int restoreOneDatabase(const char *inputFileSpec,
+ RestoreOptions *opts, int numWorkers, bool append_data);
+static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static void execute_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static int filter_dbnames_for_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list, SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleDatabaseOidList *dbname_oid_list);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname);
+static void simple_string_full_list_delete(SimpleStringList *list);
+static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
+static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
static int disable_triggers = 0;
static int enable_row_security = 0;
@@ -77,11 +119,14 @@ main(int argc, char **argv)
static int strict_names = 0;
bool data_only = false;
bool schema_only = false;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -128,6 +173,7 @@ main(int argc, char **argv)
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -156,7 +202,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "aAcCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -183,11 +229,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -302,6 +351,10 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6:
+ /* list of databases patterns those needs to skip while restoring */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -329,6 +382,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -404,6 +464,76 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ !IsFileExistsInDirectory(inputFileSpec, "toc.dat"))
+ {
+ /* If global.dat is exist, then process it. */
+ if (IsFileExistsInDirectory(pg_strdup(inputFileSpec), "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql commands. */
+ int exit_code = 0;
+
+ /*
+ * Connect to database to execute global sql commands from
+ * global.dat file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = connectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /*
+ * Open global.dat file and execute/append all the global sql
+ * commands.
+ */
+ execute_global_sql_commands(conn, inputFileSpec, opts->filename);
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ if (conn)
+ PQfinish(conn);
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ exit_code = restoreAllDatabases(conn, inputFileSpec,
+ db_exclude_patterns, opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_full_list_delete(&db_exclude_patterns);
+
+ return exit_code;
+ }
+ }
+
+ return restoreOneDatabase(inputFileSpec, opts, numWorkers, false);
+}
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ */
+static int
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data)
+{
+ Archive *AH;
+ int exit_code;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -429,11 +559,11 @@ main(int argc, char **argv)
AH->numWorkers = numWorkers;
if (opts->tocSummary)
- PrintTOCSummary(AH);
+ PrintTOCSummary(AH, append_data);
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
/* done, print a summary of ignored errors */
@@ -469,6 +599,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -481,6 +612,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -619,3 +751,585 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * IsFileExistsInDirectory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+IsFileExistsInDirectory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.sql file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+ReadOneStatement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(inBuf, (char) c);
+
+ if (c == '\n')
+ {
+ if(inBuf->len > 1 &&
+ inBuf->data[inBuf->len - 2] == ';')
+ break;
+ else
+ continue;
+ }
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * filter_dbnames_for_restore
+ *
+ * This will remove names from all dblist those can
+ * be constructed from database_exclude_pattern list.
+ *
+ * returns number of dbnames those will be restored.
+ */
+static int
+filter_dbnames_for_restore(PGconn *conn, SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
+ SimpleDatabaseOidListCell *dboidprecell = NULL;
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ /* Return 0 if there is no db to restore. */
+ if (dboid_cell == NULL)
+ return 0;
+
+ query = createPQExpBuffer();
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ while (dboid_cell != NULL)
+ {
+ bool skip_db_restore = false;
+ SimpleDatabaseOidListCell *next = dboid_cell->next;
+
+ for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ {
+ /*
+ * the construct pattern matching query:
+ * SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE
+ * pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from the
+ * dboid_list variable, which is initially extracted from the
+ * map.dat file located in the backup directory. that's why we
+ * need quote_literal_cstr.
+ *
+ * If we don't have db connection, then consider patterns as NAME
+ * only.
+ */
+ if (!conn && (pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0))
+ {
+ /*
+ * As we need to skip this dbname so set flag to remove it from
+ * list.
+ */
+ skip_db_restore = true;
+ break;
+ }
+ else
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, celldb->val, false,
+ false, NULL, quote_literal_cstr(dboid_cell->db_name),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ celldb->val);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("\"%s\" database is matching with exclude \"%s\" pattern", dboid_cell->db_name, celldb->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+
+ if (skip_db_restore)
+ break;
+ }
+ }
+
+ /* Increment count if db needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
+ simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ }
+ else
+ {
+ count_db++; /* Increment db couter. */
+ dboidprecell = dboid_cell;
+ }
+
+ /* Process next dbname from dbname list. */
+ dboid_cell = next;
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and correspoding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ {
+ pg_log_info("map.dat file is not present in dump of pg_dumpall, so nothing to restore.");
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid;
+ char db_oid_str[MAXPGPATH + 1];
+ char dbname[MAXPGPATH + 1];
+
+ /* Extract dboid. */
+ sscanf(line, "%u" , &db_oid);
+ sscanf(line, "%s" , db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found dbname as : \"%s\" and db_oid:%u in map.dat file while restoring", dbname, db_oid);
+
+ /* Report error if file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
+ SimpleDatabaseOidListCell *dboid_cell;
+ int exit_code = 0;
+ int num_db_restore;
+ int num_total_db;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /* If map.dat has no entry, return from here. */
+ if (dbname_oid_list.head == NULL)
+ return 0;
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ if (!conn)
+ pg_log_info("there is no database connection so consider pattern as simple name for exclude-database");
+ }
+ }
+
+ /*
+ * processing pg_retsore --exclude-database=PATTERN.
+ */
+ num_db_restore = filter_dbnames_for_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL)
+ return 0;
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * To restore multiple databases, -C (create database) option should be specified
+ * or all databases should be created before pg_restore.
+ */
+ if (opts->createDB != 1)
+ pg_log_info("restoring dump of pg_dumpall without -C option, there might be multiple databases in directory.");
+
+ /* TODO: MAX_ON_EXIT_NICELY is 100 now... max AH handle register on exit .*/
+ if (num_db_restore > 100)
+ {
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+ pg_fatal("cound not restore more than 100 databases by single pg_restore, here total db:%d", num_db_restore);
+ }
+
+ /*
+ * XXX: TODO till now, we made a list of databases, those needs to be restored
+ * after skipping names of exclude-database. Now we can launch parallel
+ * workers to restore these databases.
+ */
+ dboid_cell = dbname_oid_list.head;
+
+ while(dboid_cell != NULL)
+ {
+ char subdirpath[MAXPGPATH];
+ int dbexit_code;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored into
+ * already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+
+ pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+
+ dbexit_code = restoreOneDatabase(subdirpath, opts, numWorkers, true);
+
+ /* Store exit_code to report it back. */
+ if (exit_code == 0 && dbexit_code != 0)
+ exit_code = dbexit_code;
+
+ dboid_cell = dboid_cell->next;
+ } /* end while */
+
+ /* Log number of processed databases.*/
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+
+ return exit_code;
+}
+
+/*
+ * execute_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ */
+static void
+execute_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Now open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ char out_file_path[MAXPGPATH];
+ FILE *ofile;
+ int c;
+
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+
+ ofile = fopen(out_file_path, PG_BINARY_W);
+
+ if (ofile == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", out_file_path);
+ }
+
+ /* Now append global.dat inot outfile. */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ fputc(c, ofile);
+ }
+
+ fclose(pfile);
+ fclose(ofile);
+ return;
+ }
+
+ /* Init sqlstatement to append commands */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements */
+ while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ {
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ fclose(pfile);
+}
+
+/*
+ * simple_db_oid_list_append
+ *
+ * appends a node to the list in the end.
+ */
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname)
+{
+ SimpleDatabaseOidListCell *cell;
+
+ cell = pg_malloc_object(SimpleDatabaseOidListCell);
+
+ cell->next = NULL;
+ cell->db_oid = db_oid;
+ cell->db_name = pg_strdup(dbname);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * simple_db_oid_full_list_delete
+ *
+ * delete all cell from dbname and dboid list.
+ */
+static void
+simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
+{
+ SimpleDatabaseOidListCell *cell = list->head;
+ SimpleDatabaseOidListCell *nextcell = NULL;
+
+ while (cell)
+ {
+ nextcell = cell->next;
+ pfree (cell);
+ cell = nextcell;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_string_full_list_delete
+ *
+ * delete all cell from string list.
+ */
+static void
+simple_string_full_list_delete(SimpleStringList *list)
+{
+ SimpleStringListCell *cell = list->head;
+ SimpleStringListCell *cellnext = NULL;
+
+ while (cell)
+ {
+ cellnext = cell->next;
+ pfree(cell);
+ cell = cellnext;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_db_oid_list_delete
+ *
+ * delete cell from database and oid list.
+ */
+static void
+simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev)
+{
+ if (prev == NULL)
+ {
+ list->head = cell->next;
+ pfree(cell);
+ }
+ else
+ {
+ prev->next = cell->next;
+ pfree(cell);
+ }
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr -
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 214240f..ebfd07b
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -219,11 +219,20 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
- [ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
+ [ 'pg_dumpall', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
qr/\Qpg_dumpall: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index a2644a2..96f460d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2673,6 +2673,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
1.8.3.1
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-01-29 09:38 ` jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: jian he @ 2025-01-29 09:38 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
hi.
we need to escape the semicolon within the single quotes or double quotes.
I think my patch in [1] is correct.
we can have "ERROR: role "z" already exists
but
error message like
pg_restore: error: could not execute query: "ERROR: unterminated
quoted string at or near "';
should not be accepted in execute_global_sql_commands, ReadOneStatement, PQexec
attached is the all the corner test case i come up with against
ReadOneStatement.
your v13 will generate errors like "ERROR: unterminated quoted string
at or near ..."',
which is not good, i think.
[1] https://www.postgresql.org/message-id/CACJufxEQUcjBocKJQ0Amf3AfiS9wFB7zYSHrj1qqD_oWeaJoGQ%40mail.gma...
Attachments:
[application/sql] global_object_quotes_escape_tests.sql (689B, 2-global_object_quotes_escape_tests.sql)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-01-31 03:52 ` jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: jian he @ 2025-01-31 03:52 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
hi.
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
Can we spare some words to explain the purpose of append_data.
in get_dbname_oid_list_from_mfile
pg_log_info("map.dat file is not present in dump of
pg_dumpall, so nothing to restore.");
maybe we can change it to
pg_log_info("databases restoring is skipped as map.dat file is
not present in \"%s\"", dumpdirpath);
we can aslo add Assert(dumpdirpath != NULL)
pg_log_info("found dbname as : \"%s\" and db_oid:%u in map.dat file
while restoring", dbname, db_oid);
also need to change. maybe
pg_log_info("found database \"%s\" (OID: %u) in map.dat file while
restoring.", dbname, db_oid);
I also did some minor refactoring, please check attached.
doc/src/sgml/ref/pg_restore.sgml
<refnamediv>
<refname>pg_restore</refname>
<refpurpose>
restore a <productname>PostgreSQL</productname> database from an
archive file created by <application>pg_dump</application>
</refpurpose>
</refnamediv>
need to change, since now we can restore multiple databases.
doc/src/sgml/ref/pg_dumpall.sgml
<refnamediv>
<refname>pg_dumpall</refname>
<refpurpose>extract a <productname>PostgreSQL</productname> database
cluster into a script file</refpurpose>
</refnamediv>
also need change.
Attachments:
[application/octet-stream] v13-0001-minor-coesmetic-change-based-on-v13.no-cfbot (9.0K, 2-v13-0001-minor-coesmetic-change-based-on-v13.no-cfbot)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-01-31 08:51 ` jian he <[email protected]>
2025-02-01 16:06 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 2 replies; 36+ messages in thread
From: jian he @ 2025-01-31 08:51 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
hi.
more small issues.
+ count_db++; /* Increment db couter. */
+ dboidprecell = dboid_cell;
+ }
+
typo, "couter" should be "counter".
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and correspoding db_oid.
+ *
typo, "correspoding" should be "corresponding".
execute_global_sql_commands comments didn't mention ``IF (outfile) ``
branch related code.
We can add some comments saying that
""IF opts->filename is not specified, then copy the content of
global.dat to opts->filename""".
or split it into two functions.
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid;
+ char db_oid_str[MAXPGPATH + 1];
+ char dbname[MAXPGPATH + 1];
+
+ /* Extract dboid. */
+ sscanf(line, "%u" , &db_oid);
+ sscanf(line, "%s" , db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found dbname as : \"%s\" and db_oid:%u in map.dat file
while restoring", dbname, db_oid);
+
+ /* Report error if file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
db_oid first should be set to 0, dbname first character first should be set to 0
(char dbname[0] = '\0') before sscanf call.
so if sscanf fail, the db_oid and dbname value is not undermined)
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-02-01 16:06 ` Srinath Reddy <[email protected]>
1 sibling, 0 replies; 36+ messages in thread
From: Srinath Reddy @ 2025-02-01 16:06 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; [email protected]; +Cc: [email protected]
Hi,
i think we have to change the pg_dumpall "--help" message similar to
pg_dump's specifying that now pg_dumpall dumps cluster into to other
non-text formats.
Need similar "--help" message change in pg_restore to specify that now
pg_restore supports restoring whole cluster from archive created from
pg_dumpall.
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 3e022ecdeb..728abe841c 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -667,7 +667,7 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script
file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster into an SQL script
file or to other formats.\n\n"), progname);
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index fc248a441e..c4e58c1f3b 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -582,6 +582,8 @@ static void
usage(const char *progname)
{
printf(_("%s restores a PostgreSQL database from an archive created by
pg_dump.\n\n"), progname);
+ printf(_("[or]\n"));
+ printf(_("%s restores a PostgreSQL entire cluster from an archive created
by pg_dumpall.\n\n"), progname); Regards, Srinath Reddy Sadipiralla, EDB:
https://www.enterprisedb.com <http://www.enterprisedb.com/;
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-02-02 20:19 ` Mahendra Singh Thalor <[email protected]>
2025-02-03 08:53 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
1 sibling, 2 replies; 36+ messages in thread
From: Mahendra Singh Thalor @ 2025-02-02 20:19 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
Thanks Jian for review, testing and delta patches.
On Wed, 29 Jan 2025 at 15:09, jian he <[email protected]> wrote:
>
> hi.
>
> we need to escape the semicolon within the single quotes or double quotes.
> I think my patch in [1] is correct.
>
> we can have "ERROR: role "z" already exists
> but
> error message like
> pg_restore: error: could not execute query: "ERROR: unterminated
> quoted string at or near "';
> should not be accepted in execute_global_sql_commands, ReadOneStatement,
PQexec
>
> attached is the all the corner test case i come up with against
> ReadOneStatement.
> your v13 will generate errors like "ERROR: unterminated quoted string
> at or near ..."',
> which is not good, i think.
>
> [1]
https://www.postgresql.org/message-id/CACJufxEQUcjBocKJQ0Amf3AfiS9wFB7zYSHrj1qqD_oWeaJoGQ%40mail.gma...
Yes, you are right. We can't read line by line. We should read char by char
and we need some extra handling for double quote names.
I have merged your delta patch into this and now I am doing some more
testing for corner cases of this type of names.
*Ex*: add some comments in names etc or multiple semicolons or other
special characters in name.
On Fri, 31 Jan 2025 at 09:23, jian he <[email protected]> wrote:
>
> hi.
>
> -extern void RestoreArchive(Archive *AHX);
> +extern void RestoreArchive(Archive *AHX, bool append_data);
> Can we spare some words to explain the purpose of append_data.
Fixed. I added some comments on the top of the RestoreArchive function.
>
> in get_dbname_oid_list_from_mfile
> pg_log_info("map.dat file is not present in dump of
> pg_dumpall, so nothing to restore.");
> maybe we can change it to
> pg_log_info("databases restoring is skipped as map.dat file is
> not present in \"%s\"", dumpdirpath);
Fixed.
> we can aslo add Assert(dumpdirpath != NULL)
No, we don't need it as we are already checking inputfileSpec!= NULL.
>
> pg_log_info("found dbname as : \"%s\" and db_oid:%u in map.dat file
> while restoring", dbname, db_oid);
> also need to change. maybe
> pg_log_info("found database \"%s\" (OID: %u) in map.dat file while
> restoring.", dbname, db_oid);
Fixed.
>
> I also did some minor refactoring, please check attached.
Thanks. I merged it.
>
>
> doc/src/sgml/ref/pg_restore.sgml
> <refnamediv>
> <refname>pg_restore</refname>
>
> <refpurpose>
> restore a <productname>PostgreSQL</productname> database from an
> archive file created by <application>pg_dump</application>
> </refpurpose>
> </refnamediv>
> need to change, since now we can restore multiple databases.
Agreed. I added some comments.
>
> doc/src/sgml/ref/pg_dumpall.sgml
> <refnamediv>
> <refname>pg_dumpall</refname>
> <refpurpose>extract a <productname>PostgreSQL</productname> database
> cluster into a script file</refpurpose>
> </refnamediv>
> also need change.
On Sat, 1 Feb 2025 at 21:36, Srinath Reddy <[email protected]> wrote:
>
> Hi,
> i think we have to change the pg_dumpall "--help" message similar to
pg_dump's specifying that now pg_dumpall dumps cluster into to other
non-text formats.
> Need similar "--help" message change in pg_restore to specify that now
pg_restore supports restoring whole cluster from archive created from
pg_dumpall.
As Jian suggested, we need to change docs so I did the same changes into
doc and --help also.
On Fri, 31 Jan 2025 at 14:22, jian he <[email protected]> wrote:
>
> hi.
> more small issues.
>
> + count_db++; /* Increment db couter. */
> + dboidprecell = dboid_cell;
> + }
> +
> typo, "couter" should be "counter".
Fixed.
>
> +
> +/*
> + * get_dbname_oid_list_from_mfile
> + *
> + * Open map.dat file and read line by line and then prepare a list of
database
> + * names and correspoding db_oid.
> + *
> typo, "correspoding" should be "corresponding".
Fixed.
>
>
> execute_global_sql_commands comments didn't mention ``IF (outfile) ``
> branch related code.
> We can add some comments saying that
> ""IF opts->filename is not specified, then copy the content of
> global.dat to opts->filename""".
We already have some comments on the top of the execute_global_sql_commands
function.
>
> or split it into two functions.
Done. I added a new function for outfile.
>
>
> + while((fgets(line, MAXPGPATH, pfile)) != NULL)
> + {
> + Oid db_oid;
> + char db_oid_str[MAXPGPATH + 1];
> + char dbname[MAXPGPATH + 1];
> +
> + /* Extract dboid. */
> + sscanf(line, "%u" , &db_oid);
> + sscanf(line, "%s" , db_oid_str);
> +
> + /* Now copy dbname. */
> + strcpy(dbname, line + strlen(db_oid_str) + 1);
> +
> + /* Remove \n from dbanme. */
> + dbname[strlen(dbname) - 1] = '\0';
> +
> + pg_log_info("found dbname as : \"%s\" and db_oid:%u in map.dat file
> while restoring", dbname, db_oid);
> +
> + /* Report error if file has any corrupted data. */
> + if (!OidIsValid(db_oid) || strlen(dbname) == 0)
> + pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
> +
> + /*
> + * XXX : before adding dbname into list, we can verify that this db
> + * needs to skipped for restore or not but as of now, we are making
> + * a list of all the databases.
> + */
> + simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
> + count++;
> + }
>
>
> db_oid first should be set to 0, dbname first character first should be
set to 0
> (char dbname[0] = '\0') before sscanf call.
> so if sscanf fail, the db_oid and dbname value is not undermined)
Okay. Fixed.
Here, I am attaching an updated patch for review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v14_pg_dumpall-with-non-text_format-3rd_feb.patch (73.6K, 3-v14_pg_dumpall-with-non-text_format-3rd_feb.patch)
download | inline diff:
From af34451db26a11543196f52464849774e5a3fe59 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Mon, 3 Feb 2025 01:27:29 +0530
Subject: [PATCH] pg_dumpall with directory|tar|custom format and restore it
by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat to restore all databases. If global.dat file is exist in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list (if exist)
TODO1: We need to think for --exclude-database=PATTERN for pg_restore.
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
if db connection,
if no db connection, then PATTERN=NAME matching only
TODO2: We need to make changes for exit_nicely as we are one entry for each database while
restoring. MAX_ON_EXIT_NICELY
TODO3: some more test cases for new added options.
TODO4: We can dump and restore databases in parallel mode.
This needs more study
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
---
doc/src/sgml/ref/pg_dumpall.sgml | 80 ++-
doc/src/sgml/ref/pg_restore.sgml | 31 +
src/bin/pg_dump/Makefile | 8 +-
src/bin/pg_dump/common_dumpall_restore.c | 286 +++++++++
src/bin/pg_dump/common_dumpall_restore.h | 26 +
src/bin/pg_dump/meson.build | 2 +
src/bin/pg_dump/pg_backup.h | 4 +-
src/bin/pg_dump/pg_backup_archiver.c | 22 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 552 ++++++++--------
src/bin/pg_dump/pg_restore.c | 773 ++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 9 +
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 1476 insertions(+), 326 deletions(-)
create mode 100644 src/bin/pg_dump/common_dumpall_restore.c
create mode 100644 src/bin/pg_dump/common_dumpall_restore.h
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 39d93c2c0e3..6e1975f5ff0 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster based on specified dump format </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -121,7 +121,83 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- </para>
+ Note: This option can be omitted only when <option>--format</option> is plain
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index b8b27e1719e..0609b7eb534 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -20,6 +20,8 @@ PostgreSQL documentation
<refpurpose>
restore a <productname>PostgreSQL</productname> database from an
archive file created by <application>pg_dump</application>
+ or restore multiple <productname>PostgreSQL</productname> database from an
+ archive directory is created by <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -166,6 +168,25 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +336,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..a4e557d62c7 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -47,11 +47,11 @@ all: pg_dump pg_restore pg_dumpall
pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dump.o common.o pg_dump_sort.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_restore: pg_restore.o common_dumpall_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_restore.o common_dumpall_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/common_dumpall_restore.c b/src/bin/pg_dump/common_dumpall_restore.c
new file mode 100644
index 00000000000..b162cf69412
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.c
@@ -0,0 +1,286 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.c
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * This is a common file for pg_dumpall and pg_restore.
+ * src/bin/pg_dump/common_dumpall_restore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+#define exit_nicely(code) exit(code)
+
+/*
+ * connectDatabase
+ *
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the global variable 'connstr' is set to a connection string
+ * containing the options used.
+ */
+PGconn *
+connectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ static char *password = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 6;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. Remember the options used, in the form of a
+ * connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If needed, then copy server version to outer function. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/*
+ * constructConnStr
+ *
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * executeQuery
+ *
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
diff --git a/src/bin/pg_dump/common_dumpall_restore.h b/src/bin/pg_dump/common_dumpall_restore.h
new file mode 100644
index 00000000000..a0dcdbe0807
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.h
+ * Common header file for pg_dumpall and pg_restore
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/common_dumpall_restore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef COMMON_DUMPALL_RESTORE_H
+#define COMMON_DUMPALL_RESTORE_H
+
+#include "pg_backup.h"
+
+/* TODO: increasing this to keep 100 db restoring by single pg_restore command. */
+#define MAX_ON_EXIT_NICELY 100
+extern PGconn *connectDatabase(const char *dbname, const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+#endif /* COMMON_DUMPALL_RESTORE_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..ddecac5cf09 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -49,6 +49,7 @@ bin_targets += pg_dump
pg_dumpall_sources = files(
'pg_dumpall.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
@@ -68,6 +69,7 @@ bin_targets += pg_dumpall
pg_restore_sources = files(
'pg_restore.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index f0f19bb0b29..65000e5a083 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -306,7 +306,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
@@ -319,7 +319,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
DataDirSyncMethod sync_method);
/* The --list option */
-extern void PrintTOCSummary(Archive *AHX);
+extern void PrintTOCSummary(Archive *AHX, bool append_data);
extern RestoreOptions *NewRestoreOptions(void);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 707a3fc844c..e91f4b836f6 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -82,7 +82,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -331,9 +331,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -450,7 +455,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1263,7 +1268,7 @@ ArchiveEntry(Archive *AHX, CatalogId catalogId, DumpId dumpId,
/* Public */
void
-PrintTOCSummary(Archive *AHX)
+PrintTOCSummary(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -1279,7 +1284,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, append_data);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1658,7 +1663,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1678,7 +1684,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..47589cca90f 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -13,6 +13,7 @@
*/
#include "postgres_fe.h"
+#include "common_dumpall_restore.h"
#ifdef WIN32
#include "parallel.h"
#endif
@@ -21,8 +22,6 @@
/* Globals exported by this file */
const char *progname = NULL;
-#define MAX_ON_EXIT_NICELY 20
-
static struct
{
on_exit_nicely_callback function;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 02e1fdf8f78..61067e1542e 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1148,7 +1148,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 396f79781c5..80341db324d 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -24,14 +25,17 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
+#define exit_nicely(code) exit(code)
typedef struct
{
@@ -64,28 +68,25 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -107,7 +108,7 @@ static int no_subscriptions = 0;
static int no_toast_compression = 0;
static int no_unlogged_table_data = 0;
static int no_role_passwords = 0;
-static int server_version;
+static int server_version;
static int load_via_partition_root = 0;
static int on_conflict_do_nothing = 0;
@@ -121,8 +122,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -147,6 +146,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -188,6 +188,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -237,7 +239,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -265,7 +267,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -414,6 +418,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If non-plain format is specified then we must provide the
+ * file name to create one main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -460,6 +479,33 @@ main(int argc, char *argv[])
if (on_conflict_do_nothing)
appendPQExpBufferStr(pgdumpopts, " --on-conflict-do-nothing");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char toc_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -468,7 +514,8 @@ main(int argc, char *argv[])
if (pgdb)
{
conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
@@ -476,10 +523,12 @@ main(int argc, char *argv[])
else
{
conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ prompt_password, true,
+ progname, &connstr, &server_version);
if (!conn)
{
@@ -496,19 +545,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -607,7 +643,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -620,7 +656,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -631,12 +667,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster based on specified dump format.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1487,10 +1525,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1504,7 +1545,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1512,9 +1553,33 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_log_error("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1529,6 +1594,18 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is non-plain dump format, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1547,9 +1624,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1558,19 +1643,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char toc_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(toc_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ toc_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1580,7 +1676,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1589,17 +1686,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is non-plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1649,256 +1765,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
@@ -1994,3 +1860,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if(!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c602272d7db..42c4fe3ce2e 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,27 +41,71 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/connect.h"
+#include "compress_io.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
+#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
+typedef struct SimpleDatabaseOidListCell
+{
+ struct SimpleDatabaseOidListCell *next;
+ Oid db_oid;
+ const char *db_name;
+} SimpleDatabaseOidListCell;
+
+typedef struct SimpleDatabaseOidList
+{
+ SimpleDatabaseOidListCell *head;
+ SimpleDatabaseOidListCell *tail;
+} SimpleDatabaseOidList;
+
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid, const char *dbname);
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data);
+static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static void execute_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static void copy_global_file_to_out_file(const char *outfile, FILE *pfile);
+static int filter_dbnames_for_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleDatabaseOidList *dbname_oid_list);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname);
+static void simple_string_full_list_delete(SimpleStringList *list);
+static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
+static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
static int disable_triggers = 0;
static int enable_row_security = 0;
@@ -77,11 +121,14 @@ main(int argc, char **argv)
static int strict_names = 0;
bool data_only = false;
bool schema_only = false;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -128,6 +175,7 @@ main(int argc, char **argv)
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -156,7 +204,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "aAcCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -183,11 +231,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -302,6 +353,10 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6:
+ /* list of databases patterns those needs to skip while restoring */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -329,6 +384,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -404,6 +466,78 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ !IsFileExistsInDirectory(inputFileSpec, "toc.dat"))
+ {
+ /* If global.dat is exist, then process it. */
+ if (IsFileExistsInDirectory(pg_strdup(inputFileSpec), "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql commands. */
+ int exit_code = 0;
+
+ /*
+ * Connect to database to execute global sql commands from
+ * global.dat file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = connectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /*
+ * Open global.dat file and execute/append all the global sql
+ * commands.
+ */
+ execute_global_sql_commands(conn, inputFileSpec, opts->filename);
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ if (conn)
+ PQfinish(conn);
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ exit_code = restoreAllDatabases(conn, inputFileSpec,
+ db_exclude_patterns,
+ opts,
+ numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_full_list_delete(&db_exclude_patterns);
+
+ return exit_code;
+ }
+ }
+
+ return restoreOneDatabase(inputFileSpec, opts, numWorkers, false);
+}
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ */
+static int
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data)
+{
+ Archive *AH;
+ int exit_code;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -429,11 +563,11 @@ main(int argc, char **argv)
AH->numWorkers = numWorkers;
if (opts->tocSummary)
- PrintTOCSummary(AH);
+ PrintTOCSummary(AH, append_data);
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
/* done, print a summary of ignored errors */
@@ -469,6 +603,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -481,6 +616,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -619,3 +755,626 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * IsFileExistsInDirectory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+IsFileExistsInDirectory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.sql file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+ReadOneStatement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+ int m;
+
+ StringInfoData q;
+ initStringInfo(&q);
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != '\n' && c != ';')
+ {
+ appendStringInfoChar(inBuf, (char) c);
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != ';' && c != '\n')
+ appendStringInfoChar(inBuf, (char) c);
+ else
+ break;
+ }
+ }
+
+ if (c == '\'' || c == '"')
+ {
+ appendStringInfoChar(&q, (char) c);
+ m = c;
+
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(&q, (char) c);
+
+ if(c == m)
+ {
+ appendStringInfoString(inBuf, q.data);
+ resetStringInfo(&q);
+ break;
+ }
+ }
+ }
+
+ if (c == ';')
+ {
+ appendStringInfoChar(inBuf, (char) ';');
+ break;
+ }
+
+ if (c == '\n')
+ appendStringInfoChar(inBuf, (char) '\n');
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * filter_dbnames_for_restore
+ *
+ * This will remove names from all dblist those can
+ * be constructed from database_exclude_pattern list.
+ *
+ * returns number of dbnames those will be restored.
+ */
+static int
+filter_dbnames_for_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
+ SimpleDatabaseOidListCell *dboidprecell = NULL;
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ /* Return 0 if there is no db to restore. */
+ if (dboid_cell == NULL)
+ return 0;
+
+ query = createPQExpBuffer();
+
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ while (dboid_cell != NULL)
+ {
+ bool skip_db_restore = false;
+ SimpleDatabaseOidListCell *next = dboid_cell->next;
+
+ for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ {
+ /*
+ * the construct pattern matching query:
+ * SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE
+ * pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from the
+ * dboid_list variable, which is initially extracted from the
+ * map.dat file located in the backup directory. that's why we
+ * need quote_literal_cstr.
+ *
+ * If we don't have db connection, then consider patterns as NAME
+ * only.
+ */
+ if (pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0)
+ skip_db_restore = true;
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, celldb->val, false,
+ false, NULL, quote_literal_cstr(dboid_cell->db_name),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ celldb->val);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database \"%s\" is matching with exclude pattern: \"%s\"", dboid_cell->db_name, celldb->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /* Increment count if db needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
+ simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ }
+ else
+ {
+ count_db++; /* Increment db counter. */
+ dboidprecell = dboid_cell;
+ }
+
+ /* Process next dbname from dbname list. */
+ dboid_cell = next;
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ {
+ pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid = InvalidOid;
+ char db_oid_str[MAXPGPATH + 1];
+ char dbname[MAXPGPATH + 1] = {'\0'};
+
+ /* Extract dboid. */
+ sscanf(line, "%u" , &db_oid);
+ sscanf(line, "%s" , db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found database \"%s\" (OID: %u) in map.dat file while restoring.", dbname, db_oid);
+
+ /* Report error if file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
+ SimpleDatabaseOidListCell *dboid_cell;
+ int exit_code = 0;
+ int num_db_restore = 0;
+ int num_total_db;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /* If map.dat has no entry, return from here. */
+ if (dbname_oid_list.head == NULL)
+ return 0;
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ if (!conn)
+ pg_log_info("there is no database connection so consider pattern as simple name for exclude-database");
+ }
+ }
+
+ /*
+ * processing pg_retsore --exclude-database=PATTERN/NAME if no connection.
+ */
+ num_db_restore = filter_dbnames_for_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL)
+ return 0;
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * To restore multiple databases, -C (create database) option should be specified
+ * or all databases should be created before pg_restore.
+ */
+ if (opts->createDB != 1)
+ pg_log_info("restoring dump of pg_dumpall without -C option, there might be multiple databases in directory.");
+
+ /* TODO: MAX_ON_EXIT_NICELY is 100 now... max AH handle register on exit .*/
+ if (num_db_restore > MAX_ON_EXIT_NICELY)
+ {
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+ pg_fatal("cound not restore more than %d databases by single pg_restore, here total db:%d",
+ MAX_ON_EXIT_NICELY,
+ num_db_restore);
+ }
+
+ /*
+ * XXX: TODO till now, we made a list of databases, those needs to be restored
+ * after skipping names of exclude-database. Now we can launch parallel
+ * workers to restore these databases.
+ */
+ dboid_cell = dbname_oid_list.head;
+
+ while(dboid_cell != NULL)
+ {
+ char subdirpath[MAXPGPATH];
+ int dbexit_code;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored into
+ * already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+
+ pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+
+ dbexit_code = restoreOneDatabase(subdirpath, opts, numWorkers, true);
+
+ /* Store exit_code to report it back. */
+ if (exit_code == 0 && dbexit_code != 0)
+ exit_code = dbexit_code;
+
+ dboid_cell = dboid_cell->next;
+ }
+
+ /* Log number of processed databases.*/
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+
+ return exit_code;
+}
+
+/*
+ * execute_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ */
+static void
+execute_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Now open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ copy_global_file_to_out_file(outfile, pfile);
+ return;
+ }
+
+ /* Init sqlstatement to append commands */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements */
+ while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ {
+ pg_log_info("executing %s", sqlstatement.data);
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ fclose(pfile);
+}
+
+/*
+ * copy_global_file_to_out_file
+ *
+ * This will copy global.dat file into out file.
+ */
+static void
+copy_global_file_to_out_file(const char *outfile, FILE *pfile)
+{
+ char out_file_path[MAXPGPATH];
+ FILE *ofile;
+ int c;
+
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ ofile = fopen(out_file_path, PG_BINARY_W);
+
+ if (ofile == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", out_file_path);
+ }
+
+ /* Now append global.dat into out file. */
+ while ((c = fgetc(pfile)) != EOF)
+ fputc(c, ofile);
+
+ fclose(pfile);
+ fclose(ofile);
+}
+
+/*
+ * simple_db_oid_list_append
+ *
+ * appends a node to the list in the end.
+ */
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname)
+{
+ SimpleDatabaseOidListCell *cell;
+
+ cell = pg_malloc_object(SimpleDatabaseOidListCell);
+
+ cell->next = NULL;
+ cell->db_oid = db_oid;
+ cell->db_name = pg_strdup(dbname);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * simple_db_oid_full_list_delete
+ *
+ * delete all cell from dbname and dboid list.
+ */
+static void
+simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
+{
+ SimpleDatabaseOidListCell *cell = list->head;
+ SimpleDatabaseOidListCell *nextcell = NULL;
+
+ while (cell)
+ {
+ nextcell = cell->next;
+ pfree (cell);
+ cell = nextcell;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_string_full_list_delete
+ *
+ * delete all cell from string list.
+ */
+static void
+simple_string_full_list_delete(SimpleStringList *list)
+{
+ SimpleStringListCell *cell = list->head;
+ SimpleStringListCell *cellnext = NULL;
+
+ while (cell)
+ {
+ cellnext = cell->next;
+ pfree(cell);
+ cell = cellnext;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_db_oid_list_delete
+ *
+ * delete cell from database and oid list.
+ */
+static void
+simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev)
+{
+ if (prev == NULL)
+ {
+ list->head = cell->next;
+ pfree(cell);
+ }
+ else
+ {
+ prev->next = cell->next;
+ pfree(cell);
+ }
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr
+ *
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 214240f1ae5..de41ec06d86
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -219,6 +219,11 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -226,4 +231,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9a3bee93dec..cdaf1ad343c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2673,6 +2673,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.39.3
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-02-03 08:53 ` Srinath Reddy <[email protected]>
2025-02-03 09:02 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
1 sibling, 1 reply; 36+ messages in thread
From: Srinath Reddy @ 2025-02-03 08:53 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; [email protected]; +Cc: [email protected]
Hi,
I found a bug ,while using "./pg_restore pdd -f -" actually it has to copy
everything(global sql commands + remaining dump ) into stdout as per the
"-f, --file=FILENAME output file name (- for stdout)" but it is
copying global sql commands to a file literally naming it as "-" and
remaining dump is written to stdout without those global sql commands."-"
is not a output file it signifies stdout in terminal cmds.so we have to
handle this case.
because of above reason "./pg_restore pdd -g -f -" also does the same
creates a file "-" and writes globals to that file instead of stdout.
This is the delta patch to handle this case.please have a look and give
some feedback.
@@ -84,7 +84,7 @@ static int restoreAllDatabases(PGconn *conn, const char
*dumpdirpath,
SimpleStringList
db_exclude_patterns, RestoreOptions *opts, int numWorkers);
static void execute_global_sql_commands(PGconn *conn, const char
*dumpdirpath,
const char *outfile);
-static void copy_global_file_to_out_file(const char *outfile, FILE *pfile);
+static void copy_global_file(const char *outfile, FILE *pfile);
static int filter_dbnames_for_restore(PGconn *conn,
SimpleDatabaseOidList *dbname_oid_list,
- ofile = fopen(out_file_path, PG_BINARY_W);
+ if (strcmp(outfile, "-") == 0){
+ int fn = fileno(stdout);
+ ofile = fdopen(dup(fn), PG_BINARY_W);
+ }
+ else{
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ ofile = fopen(out_file_path, PG_BINARY_W);
+ }
+
if (ofile == NULL)
{
Regards,
Srinath Reddy Sadipiralla,
EDB: https://www.enterprisedb.com <http://www.enterprisedb.com/;
>
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 08:53 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
@ 2025-02-03 09:02 ` Srinath Reddy <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Srinath Reddy @ 2025-02-03 09:02 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; [email protected]; +Cc: [email protected]
here's the whole version of delta patch
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 42c4fe3ce2..90e6b71a50 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -84,7 +84,7 @@ static int restoreAllDatabases(PGconn *conn, const char
*dumpdirpath,
SimpleStringList db_exclude_patterns, RestoreOptions *opts, int
numWorkers);
static void execute_global_sql_commands(PGconn *conn, const char
*dumpdirpath,
const char *outfile);
-static void copy_global_file_to_out_file(const char *outfile, FILE *pfile);
+static void copy_global_file(const char *outfile, FILE *pfile);
static int filter_dbnames_for_restore(PGconn *conn,
SimpleDatabaseOidList *dbname_oid_list,
SimpleStringList db_exclude_patterns);
@@ -1178,7 +1178,7 @@ execute_global_sql_commands(PGconn *conn, const char
*dumpdirpath, const char *o
*/
if (outfile)
{
- copy_global_file_to_out_file(outfile, pfile);
+ copy_global_file(outfile, pfile);
return;
}
@@ -1207,24 +1207,35 @@ execute_global_sql_commands(PGconn *conn, const
char *dumpdirpath, const char *o
}
/*
- * copy_global_file_to_out_file
+ * copy_global_file
*
- * This will copy global.dat file into out file.
+ * This will copy global.dat file into out file, if file is given
+ * else copies to stdout.
+ *
*/
static void
-copy_global_file_to_out_file(const char *outfile, FILE *pfile)
+copy_global_file(const char *outfile, FILE *pfile)
{
char out_file_path[MAXPGPATH];
FILE *ofile;
int c;
- snprintf(out_file_path, MAXPGPATH, "%s", outfile);
- ofile = fopen(out_file_path, PG_BINARY_W);
+ if (strcmp(outfile, "-") == 0)
+ {
+ int fn = fileno(stdout);
+ ofile = fdopen(dup(fn), PG_BINARY_W);
+ }
+ else
+ {
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ ofile = fopen(out_file_path, PG_BINARY_W);
+ }
+
if (ofile == NULL)
{
fclose(pfile);
- pg_fatal("could not open file: \"%s\"", out_file_path);
+ pg_fatal("could not open file: \"%s\"", outfile);
}
/* Now append global.dat into out file. */
> Regards,
> Srinath Reddy Sadipiralla,
> EDB: https://www.enterprisedb.com <http://www.enterprisedb.com/;
>
>>
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-02-03 09:14 ` jian he <[email protected]>
2025-02-03 12:49 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
1 sibling, 2 replies; 36+ messages in thread
From: jian he @ 2025-02-03 09:14 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
hi.
git clean -fdx && $BIN10/pg_dumpall --format=directory --file=dir10
$BIN10/pg_restore --format=directory --file=1.sql --verbose dir10 >
dir_format 2>&1
there is no "\connect dbname" command.
pipe 1.sql to psql will execute all the database dump into a single
database, which is not good.
we need "\connect dbname" in file 1.sql
--------<<<<<<<>>>>>>>>>>>>>>>------------------
$BIN10/pg_dumpall --format=directory --exclude-database=src10 --file=dir12_temp
drop table t from database x
$BIN10/pg_restore --format=directory --dbname=x --verbose dir12_temp >
dir_format 2>&1
--------log info------------------
pg_restore: found database "template1" (OID: 1) in map.dat file while restoring.
pg_restore: found database "x" (OID: 19554) in map.dat file while restoring.
pg_restore: found total 2 database names in map.dat file
pg_restore: needs to restore 2 databases out of 2 databases
pg_restore: restoring dump of pg_dumpall without -C option, there
might be multiple databases in directory.
pg_restore: restoring database "template1"
pg_restore: connecting to database for restore
pg_restore: implied data-only restore
pg_restore: restoring database "x"
pg_restore: connecting to database for restore
pg_restore: processing data for table "public.t"
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 3374; 0 19555 TABLE DATA t jian
pg_restore: error: could not execute query: ERROR: relation
"public.t" does not exist
Command was: COPY public.t (a) FROM stdin;
pg_restore: warning: errors ignored on restore: 1
pg_restore: number of restored databases are 2
________________________
$BIN10/pg_restore --format=directory --list dir12_temp
selected output:
; Selected TOC Entries:
;
217; 1259 19555 TABLE public t jian
3374; 0 19555 TABLE DATA public t jian
3228; 2606 19560 CONSTRAINT public t t_pkey jian
As you can see, dir12_temp has TABLE and TABLE DATA.
so the above log message: "pg_restore: implied data-only restore" is
not what we expected.
BTW, add --create option, it works as i expected.
like
$BIN10/pg_restore --format=directory --create --dbname=x --verbose
dir12_temp > dir_format 2>&1
output is what i expected.
--------<<<<<<<>>>>>>>>>>>>>>>------------------
with the changes in filter_dbnames_for_restore.
so <option>--exclude-database=<replaceable
class="parameter">pattern</replaceable></option>
will behave differently when you specify the --file option or not.
* --file option specified
-exclude-database=pattern not allow any special wildcard character.
it does not behave the same as the doc mentioned.
* --file option not specified, it behaves the same as the doc mentioned.
That's kind of tricky, either more words in the doc explain the
scarenio where --file option is specified
or disallow --file option when --exclude-database is specified.
we need to update pg_restore.sgml about MAX_ON_EXIT_NICELY 100?
there is some corner like num_db_restore == 0, num_db_restore >= 100
in that scarenio, the execute_global_sql_commands already executed,
which is not ideal, since you have pg_fatal and some sql commands
already executed.
maybe we can be if 0 < num_db_restore < 100 then
call execute_global_sql_commands and restoreAllDatabases.
the attached patch trying to do that.
attached patch also doing some cosmetic changes.
Attachments:
[application/octet-stream] v14-0001-pg_restore-dump-global-objects-at-least-one-d.no-cfbot (5.1K, 2-v14-0001-pg_restore-dump-global-objects-at-least-one-d.no-cfbot)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-02-03 12:49 ` jian he <[email protected]>
1 sibling, 0 replies; 36+ messages in thread
From: jian he @ 2025-02-03 12:49 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
On Mon, Feb 3, 2025 at 5:14 PM jian he <[email protected]> wrote:
>
> there is some corner like num_db_restore == 0, num_db_restore >= 100
> in that scarenio, the execute_global_sql_commands already executed,
> which is not ideal, since you have pg_fatal and some sql commands
> already executed.
> maybe we can be if 0 < num_db_restore < 100 then
> call execute_global_sql_commands and restoreAllDatabases.
>
>
> the attached patch trying to do that.
> attached patch also doing some cosmetic changes.
hi.
please ignore the previous patch. see this email attached patch.
previously I complained that the ``pg_restore --list`` needed a db
connection and also called execute_global_sql_commands in [1]
this email attached patch fixes the problem, now pg_restore --list no
need db connection.
now the logic is:
if num_db_restore value is ok (0 < num_db_restore < MAX_ON_EXIT_NICELY)
*AND* we didn't specify --list option
then call execute_global_sql_commands.
[1] https://postgr.es/m/CACJufxHUDGWe=2ZukvMfuwEcSK8CsVYm=9+rtPnrW7CRCfoCsw@mail.gmail.com
Attachments:
[application/octet-stream] v14-0001-fix-pg_restore-list-option-and-handle-invoke-.no-cfbot (5.5K, 2-v14-0001-fix-pg_restore-list-option-and-handle-invoke-.no-cfbot)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-02-03 18:04 ` Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
1 sibling, 1 reply; 36+ messages in thread
From: Mahendra Singh Thalor @ 2025-02-03 18:04 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
On Mon, 3 Feb 2025 at 14:23, Srinath Reddy <[email protected]> wrote:
>
> Hi,
> I found a bug ,while using "./pg_restore pdd -f -" actually it has to copy everything(global sql commands + remaining dump ) into stdout as per the "-f, --file=FILENAME output file name (- for stdout)" but it is copying global sql commands to a file literally naming it as "-" and remaining dump is written to stdout without those global sql commands."-" is not a output file it signifies stdout in terminal cmds.so we have to handle this case.
> because of above reason "./pg_restore pdd -g -f -" also does the same creates a file "-" and writes globals to that file instead of stdout.
I also tested this but in my testing, I can see that all globals are
printed into the console also. Patch was creating a "-" file that was
wrong.
Yes, we should consider "-" as a stdout. In the latest patch, I have
fixed this issue.
On Mon, 3 Feb 2025 at 14:44, jian he <[email protected]> wrote:
>
> hi.
>
> git clean -fdx && $BIN10/pg_dumpall --format=directory --file=dir10
> $BIN10/pg_restore --format=directory --file=1.sql --verbose dir10 >
> dir_format 2>&1
>
> there is no "\connect dbname" command.
> pipe 1.sql to psql will execute all the database dump into a single
> database, which is not good.
> we need "\connect dbname" in file 1.sql
We can't add this command directly to the dump file. We need to add
some TOC entry for this command. I will try to make a TOC entry for
this command.
>
>
> --------<<<<<<<>>>>>>>>>>>>>>>------------------
> $BIN10/pg_dumpall --format=directory --exclude-database=src10 --file=dir12_temp
> drop table t from database x
> $BIN10/pg_restore --format=directory --dbname=x --verbose dir12_temp >
> dir_format 2>&1
> --------log info------------------
> pg_restore: found database "template1" (OID: 1) in map.dat file while restoring.
> pg_restore: found database "x" (OID: 19554) in map.dat file while restoring.
> pg_restore: found total 2 database names in map.dat file
> pg_restore: needs to restore 2 databases out of 2 databases
> pg_restore: restoring dump of pg_dumpall without -C option, there
> might be multiple databases in directory.
> pg_restore: restoring database "template1"
> pg_restore: connecting to database for restore
> pg_restore: implied data-only restore
> pg_restore: restoring database "x"
> pg_restore: connecting to database for restore
> pg_restore: processing data for table "public.t"
> pg_restore: while PROCESSING TOC:
> pg_restore: from TOC entry 3374; 0 19555 TABLE DATA t jian
> pg_restore: error: could not execute query: ERROR: relation
> "public.t" does not exist
> Command was: COPY public.t (a) FROM stdin;
> pg_restore: warning: errors ignored on restore: 1
> pg_restore: number of restored databases are 2
> ________________________
> $BIN10/pg_restore --format=directory --list dir12_temp
> selected output:
>
> ; Selected TOC Entries:
> ;
> 217; 1259 19555 TABLE public t jian
> 3374; 0 19555 TABLE DATA public t jian
> 3228; 2606 19560 CONSTRAINT public t t_pkey jian
>
> As you can see, dir12_temp has TABLE and TABLE DATA.
> so the above log message: "pg_restore: implied data-only restore" is
> not what we expected.
I will do some tests with pg_dump and -t option.
>
> BTW, add --create option, it works as i expected.
> like
> $BIN10/pg_restore --format=directory --create --dbname=x --verbose
> dir12_temp > dir_format 2>&1
> output is what i expected.
>
> --------<<<<<<<>>>>>>>>>>>>>>>------------------
> with the changes in filter_dbnames_for_restore.
> so <option>--exclude-database=<replaceable
> class="parameter">pattern</replaceable></option>
> will behave differently when you specify the --file option or not.
>
> * --file option specified
> -exclude-database=pattern not allow any special wildcard character.
> it does not behave the same as the doc mentioned.
> * --file option not specified, it behaves the same as the doc mentioned.
>
> That's kind of tricky, either more words in the doc explain the
> scarenio where --file option is specified
> or disallow --file option when --exclude-database is specified.
We will do some more doc changes for this in next versions.
>
> we need to update pg_restore.sgml about MAX_ON_EXIT_NICELY 100?
Temporary, we increased this size. Based on other opinions, we will do
more changes for this.
>
> there is some corner like num_db_restore == 0, num_db_restore >= 100
> in that scarenio, the execute_global_sql_commands already executed,
> which is not ideal, since you have pg_fatal and some sql commands
> already executed.
> maybe we can be if 0 < num_db_restore < 100 then
> call execute_global_sql_commands and restoreAllDatabases.
Got it. Fixed it as per delta patch and added some extra condition to
the IF clause.
Here, I am attaching an updated patch for review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v15_pg_dumpall-with-non-text_format-4th_feb.patch (74.1K, 2-v15_pg_dumpall-with-non-text_format-4th_feb.patch)
download | inline diff:
From cfa2902fd835f9afe8c7ae571615b5d24900fa3a Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Mon, 3 Feb 2025 23:18:47 +0530
Subject: [PATCH] pg_dumpall with directory|tar|custom format and restore it
by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat to restore all databases. If global.dat file is exist in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list (if exist)
TODO1: We need to think for --exclude-database=PATTERN for pg_restore.
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
if db connection,
if no db connection, then PATTERN=NAME matching only
TODO2: We need to make changes for exit_nicely as we are one entry for each database while
restoring. MAX_ON_EXIT_NICELY
TODO3: some more test cases for new added options.
TODO4: We can dump and restore databases in parallel mode.
This needs more study
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
---
doc/src/sgml/ref/pg_dumpall.sgml | 80 ++-
doc/src/sgml/ref/pg_restore.sgml | 31 +
src/bin/pg_dump/Makefile | 8 +-
src/bin/pg_dump/common_dumpall_restore.c | 286 ++++++++
src/bin/pg_dump/common_dumpall_restore.h | 26 +
src/bin/pg_dump/meson.build | 2 +
src/bin/pg_dump/pg_backup.h | 4 +-
src/bin/pg_dump/pg_backup_archiver.c | 22 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 552 ++++++++--------
src/bin/pg_dump/pg_restore.c | 791 ++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 9 +
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 1494 insertions(+), 326 deletions(-)
create mode 100644 src/bin/pg_dump/common_dumpall_restore.c
create mode 100644 src/bin/pg_dump/common_dumpall_restore.h
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 39d93c2c0e3..6e1975f5ff0 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster based on specified dump format </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -121,7 +121,83 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- </para>
+ Note: This option can be omitted only when <option>--format</option> is plain
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index b8b27e1719e..0609b7eb534 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -20,6 +20,8 @@ PostgreSQL documentation
<refpurpose>
restore a <productname>PostgreSQL</productname> database from an
archive file created by <application>pg_dump</application>
+ or restore multiple <productname>PostgreSQL</productname> database from an
+ archive directory is created by <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -166,6 +168,25 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +336,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..a4e557d62c7 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -47,11 +47,11 @@ all: pg_dump pg_restore pg_dumpall
pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dump.o common.o pg_dump_sort.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_restore: pg_restore.o common_dumpall_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_restore.o common_dumpall_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/common_dumpall_restore.c b/src/bin/pg_dump/common_dumpall_restore.c
new file mode 100644
index 00000000000..b162cf69412
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.c
@@ -0,0 +1,286 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.c
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * This is a common file for pg_dumpall and pg_restore.
+ * src/bin/pg_dump/common_dumpall_restore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+#define exit_nicely(code) exit(code)
+
+/*
+ * connectDatabase
+ *
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the global variable 'connstr' is set to a connection string
+ * containing the options used.
+ */
+PGconn *
+connectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ static char *password = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 6;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. Remember the options used, in the form of a
+ * connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If needed, then copy server version to outer function. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/*
+ * constructConnStr
+ *
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * executeQuery
+ *
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
diff --git a/src/bin/pg_dump/common_dumpall_restore.h b/src/bin/pg_dump/common_dumpall_restore.h
new file mode 100644
index 00000000000..a0dcdbe0807
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.h
+ * Common header file for pg_dumpall and pg_restore
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/common_dumpall_restore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef COMMON_DUMPALL_RESTORE_H
+#define COMMON_DUMPALL_RESTORE_H
+
+#include "pg_backup.h"
+
+/* TODO: increasing this to keep 100 db restoring by single pg_restore command. */
+#define MAX_ON_EXIT_NICELY 100
+extern PGconn *connectDatabase(const char *dbname, const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+#endif /* COMMON_DUMPALL_RESTORE_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..ddecac5cf09 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -49,6 +49,7 @@ bin_targets += pg_dump
pg_dumpall_sources = files(
'pg_dumpall.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
@@ -68,6 +69,7 @@ bin_targets += pg_dumpall
pg_restore_sources = files(
'pg_restore.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index f0f19bb0b29..65000e5a083 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -306,7 +306,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
@@ -319,7 +319,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
DataDirSyncMethod sync_method);
/* The --list option */
-extern void PrintTOCSummary(Archive *AHX);
+extern void PrintTOCSummary(Archive *AHX, bool append_data);
extern RestoreOptions *NewRestoreOptions(void);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 707a3fc844c..e91f4b836f6 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -82,7 +82,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -331,9 +331,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -450,7 +455,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1263,7 +1268,7 @@ ArchiveEntry(Archive *AHX, CatalogId catalogId, DumpId dumpId,
/* Public */
void
-PrintTOCSummary(Archive *AHX)
+PrintTOCSummary(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -1279,7 +1284,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, append_data);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1658,7 +1663,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1678,7 +1684,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..47589cca90f 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -13,6 +13,7 @@
*/
#include "postgres_fe.h"
+#include "common_dumpall_restore.h"
#ifdef WIN32
#include "parallel.h"
#endif
@@ -21,8 +22,6 @@
/* Globals exported by this file */
const char *progname = NULL;
-#define MAX_ON_EXIT_NICELY 20
-
static struct
{
on_exit_nicely_callback function;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 02e1fdf8f78..61067e1542e 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1148,7 +1148,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 396f79781c5..80341db324d 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -24,14 +25,17 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
+#define exit_nicely(code) exit(code)
typedef struct
{
@@ -64,28 +68,25 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -107,7 +108,7 @@ static int no_subscriptions = 0;
static int no_toast_compression = 0;
static int no_unlogged_table_data = 0;
static int no_role_passwords = 0;
-static int server_version;
+static int server_version;
static int load_via_partition_root = 0;
static int on_conflict_do_nothing = 0;
@@ -121,8 +122,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -147,6 +146,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -188,6 +188,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -237,7 +239,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -265,7 +267,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -414,6 +418,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If non-plain format is specified then we must provide the
+ * file name to create one main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -460,6 +479,33 @@ main(int argc, char *argv[])
if (on_conflict_do_nothing)
appendPQExpBufferStr(pgdumpopts, " --on-conflict-do-nothing");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char toc_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -468,7 +514,8 @@ main(int argc, char *argv[])
if (pgdb)
{
conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
@@ -476,10 +523,12 @@ main(int argc, char *argv[])
else
{
conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ prompt_password, true,
+ progname, &connstr, &server_version);
if (!conn)
{
@@ -496,19 +545,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -607,7 +643,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -620,7 +656,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -631,12 +667,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster based on specified dump format.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1487,10 +1525,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1504,7 +1545,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1512,9 +1553,33 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_log_error("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1529,6 +1594,18 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is non-plain dump format, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1547,9 +1624,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1558,19 +1643,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char toc_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(toc_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ toc_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1580,7 +1676,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1589,17 +1686,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is non-plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1649,256 +1765,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
@@ -1994,3 +1860,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if(!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c602272d7db..444415d2ee0 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,27 +41,70 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/connect.h"
+#include "compress_io.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
+#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
+typedef struct SimpleDatabaseOidListCell
+{
+ struct SimpleDatabaseOidListCell *next;
+ Oid db_oid;
+ const char *db_name;
+} SimpleDatabaseOidListCell;
+
+typedef struct SimpleDatabaseOidList
+{
+ SimpleDatabaseOidListCell *head;
+ SimpleDatabaseOidListCell *tail;
+} SimpleDatabaseOidList;
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data);
+static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static void process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static void copy_global_file_to_out_file(const char *outfile, FILE *pfile);
+static int filter_dbnames_for_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleDatabaseOidList *dbname_oid_list);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname);
+static void simple_string_full_list_delete(SimpleStringList *list);
+static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
+static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list,
+ Oid db_oid, const char *dbname);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
static int disable_triggers = 0;
static int enable_row_security = 0;
@@ -77,11 +120,14 @@ main(int argc, char **argv)
static int strict_names = 0;
bool data_only = false;
bool schema_only = false;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -128,6 +174,7 @@ main(int argc, char **argv)
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -156,7 +203,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "aAcCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -183,11 +230,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -302,6 +352,10 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6:
+ /* list of databases patterns those needs to skip while restoring */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -329,6 +383,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -404,6 +465,80 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ !IsFileExistsInDirectory(inputFileSpec, "toc.dat"))
+ {
+ /* If global.dat is exist, then process it. */
+ if (IsFileExistsInDirectory(pg_strdup(inputFileSpec), "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql commands. */
+ int exit_code = 0;
+
+ /*
+ * Connect to database to execute global sql commands from
+ * global.dat file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = connectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ /*
+ * Open global.dat file and execute/append all the global sql
+ * commands.
+ */
+ if (!opts->tocSummary || opts->filename)
+ process_global_sql_commands(conn, inputFileSpec, opts->filename);
+
+ if (conn)
+ PQfinish(conn);
+
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ exit_code = restoreAllDatabases(conn, inputFileSpec,
+ db_exclude_patterns,
+ opts,
+ numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_full_list_delete(&db_exclude_patterns);
+
+ return exit_code;
+ }
+ }
+
+ return restoreOneDatabase(inputFileSpec, opts, numWorkers, false);
+}
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ */
+static int
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data)
+{
+ Archive *AH;
+ int exit_code;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -429,11 +564,11 @@ main(int argc, char **argv)
AH->numWorkers = numWorkers;
if (opts->tocSummary)
- PrintTOCSummary(AH);
+ PrintTOCSummary(AH, append_data);
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
/* done, print a summary of ignored errors */
@@ -469,6 +604,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -481,6 +617,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -619,3 +756,643 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * IsFileExistsInDirectory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+IsFileExistsInDirectory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.sql file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+ReadOneStatement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+ int m;
+
+ StringInfoData q;
+ initStringInfo(&q);
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != '\n' && c != ';')
+ {
+ appendStringInfoChar(inBuf, (char) c);
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != ';' && c != '\n')
+ appendStringInfoChar(inBuf, (char) c);
+ else
+ break;
+ }
+ }
+
+ if (c == '\'' || c == '"')
+ {
+ appendStringInfoChar(&q, (char) c);
+ m = c;
+
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(&q, (char) c);
+
+ if(c == m)
+ {
+ appendStringInfoString(inBuf, q.data);
+ resetStringInfo(&q);
+ break;
+ }
+ }
+ }
+
+ if (c == ';')
+ {
+ appendStringInfoChar(inBuf, (char) ';');
+ break;
+ }
+
+ if (c == '\n')
+ appendStringInfoChar(inBuf, (char) '\n');
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * filter_dbnames_for_restore
+ *
+ * This will remove names from all dblist those can
+ * be constructed from database_exclude_pattern list.
+ *
+ * returns number of dbnames those will be restored.
+ */
+static int
+filter_dbnames_for_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
+ SimpleDatabaseOidListCell *dboidprecell = NULL;
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ /* Return 0 if there is no db to restore. */
+ if (dboid_cell == NULL)
+ return 0;
+
+ query = createPQExpBuffer();
+
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ while (dboid_cell != NULL)
+ {
+ bool skip_db_restore = false;
+ SimpleDatabaseOidListCell *next = dboid_cell->next;
+
+ for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ {
+ /*
+ * the construct pattern matching query:
+ * SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE
+ * pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from the
+ * dboid_list variable, which is initially extracted from the
+ * map.dat file located in the backup directory. that's why we
+ * need quote_literal_cstr.
+ *
+ * If we don't have db connection, then consider patterns as NAME
+ * only.
+ */
+ if (pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0)
+ skip_db_restore = true;
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, celldb->val, false,
+ false, NULL, quote_literal_cstr(dboid_cell->db_name),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ celldb->val);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database \"%s\" is matching with exclude pattern: \"%s\"", dboid_cell->db_name, celldb->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /* Increment count if db needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
+ simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ }
+ else
+ {
+ count_db++; /* Increment db counter. */
+ dboidprecell = dboid_cell;
+ }
+
+ /* Process next dbname from dbname list. */
+ dboid_cell = next;
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ {
+ pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid = InvalidOid;
+ char db_oid_str[MAXPGPATH + 1] = {'\0'};
+ char dbname[MAXPGPATH + 1] = {'\0'};
+
+ /* Extract dboid. */
+ sscanf(line, "%u" , &db_oid);
+ sscanf(line, "%s" , db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found database \"%s\" (OID: %u) in map.dat file while restoring.", dbname, db_oid);
+
+ /* Report error if file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
+ SimpleDatabaseOidListCell *dboid_cell;
+ int exit_code = 0;
+ int num_db_restore = 0;
+ int num_total_db;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /* If map.dat has no entry, return from here. */
+ if (dbname_oid_list.head == NULL)
+ return 0;
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ if (!conn)
+ pg_log_info("there is no database connection so consider pattern as simple name for --exclude-database");
+ }
+ }
+
+ /*
+ * processing pg_retsore --exclude-database=PATTERN/NAME if no connection.
+ */
+ num_db_restore = filter_dbnames_for_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /*
+ * To restore multiple databases, -C (create database) option should be specified
+ * or all databases should be created before pg_restore.
+ */
+ if (opts->createDB != 1)
+ pg_log_info("restoring dump of pg_dumpall without -C option, there might be multiple databases in directory.");
+
+ /* TODO: MAX_ON_EXIT_NICELY is 100 now... max AH handle register on exit .*/
+ if (num_db_restore > MAX_ON_EXIT_NICELY)
+ {
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+ pg_fatal("cound not restore more than %d databases by single pg_restore, here total database:%d",
+ MAX_ON_EXIT_NICELY,
+ num_db_restore);
+ }
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ if (!opts->tocSummary || opts->filename)
+ process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL)
+ {
+ pg_log_info("no database needs to restore out of %d databases", num_total_db);
+ return 0;
+ }
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * XXX: TODO till now, we made a list of databases, those needs to be restored
+ * after skipping names of exclude-database. Now we can launch parallel
+ * workers to restore these databases.
+ */
+ dboid_cell = dbname_oid_list.head;
+
+ while(dboid_cell != NULL)
+ {
+ char subdirpath[MAXPGPATH];
+ int dbexit_code;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored into
+ * already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+
+ pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+
+ dbexit_code = restoreOneDatabase(subdirpath, opts, numWorkers, true);
+
+ /* Store exit_code to report it back. */
+ if (exit_code == 0 && dbexit_code != 0)
+ exit_code = dbexit_code;
+
+ dboid_cell = dboid_cell->next;
+ }
+
+ /* Log number of processed databases.*/
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+
+ return exit_code;
+}
+
+/*
+ * process_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ */
+static void
+process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Now open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ copy_global_file_to_out_file(outfile, pfile);
+ return;
+ }
+
+ /* Init sqlstatement to append commands. */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements. */
+ while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ {
+ pg_log_info("executing %s", sqlstatement.data);
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ fclose(pfile);
+}
+
+/*
+ * copy_global_file_to_out_file
+ *
+ * This will copy global.dat file into out file. If "-" is used as outfile,
+ * then it will use into stdout.
+ */
+static void
+copy_global_file_to_out_file(const char *outfile, FILE *pfile)
+{
+ char out_file_path[MAXPGPATH];
+ FILE *OPF;
+ int c;
+
+ /* "-" is used for stdout. */
+ if (strcmp(outfile, "-") == 0)
+ OPF = stdout;
+ else
+ {
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ OPF = fopen(out_file_path, PG_BINARY_W);
+
+ if (OPF == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", outfile);
+ }
+ }
+
+ /* Now append global.dat into out file. */
+ while ((c = fgetc(pfile)) != EOF)
+ fputc(c, OPF);
+
+ fclose(pfile);
+
+ /* Close out file. */
+ if (strcmp(outfile, "-") != 0)
+ fclose(OPF);
+}
+
+/*
+ * simple_db_oid_list_append
+ *
+ * appends a node to the list in the end.
+ */
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname)
+{
+ SimpleDatabaseOidListCell *cell;
+
+ cell = pg_malloc_object(SimpleDatabaseOidListCell);
+
+ cell->next = NULL;
+ cell->db_oid = db_oid;
+ cell->db_name = pg_strdup(dbname);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * simple_db_oid_full_list_delete
+ *
+ * delete all cell from dbname and dboid list.
+ */
+static void
+simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
+{
+ SimpleDatabaseOidListCell *cell = list->head;
+ SimpleDatabaseOidListCell *nextcell = NULL;
+
+ while (cell)
+ {
+ nextcell = cell->next;
+ pfree (cell);
+ cell = nextcell;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_string_full_list_delete
+ *
+ * delete all cell from string list.
+ */
+static void
+simple_string_full_list_delete(SimpleStringList *list)
+{
+ SimpleStringListCell *cell = list->head;
+ SimpleStringListCell *cellnext = NULL;
+
+ while (cell)
+ {
+ cellnext = cell->next;
+ pfree(cell);
+ cell = cellnext;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_db_oid_list_delete
+ *
+ * delete cell from database and oid list.
+ */
+static void
+simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev)
+{
+ if (prev == NULL)
+ {
+ list->head = cell->next;
+ pfree(cell);
+ }
+ else
+ {
+ prev->next = cell->next;
+ pfree(cell);
+ }
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr
+ *
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 214240f1ae5..de41ec06d86
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -219,6 +219,11 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -226,4 +231,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9a3bee93dec..cdaf1ad343c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2673,6 +2673,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.39.3
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-02-04 02:04 ` jian he <[email protected]>
2025-02-04 07:42 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 05:40 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 2 replies; 36+ messages in thread
From: jian he @ 2025-02-04 02:04 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
hi.
just a quick response for v15.
the pg_restore man page says option --list as "List the table of
contents of the archive".
but
$BIN10/pg_restore --format=directory --list --file=1.sql dir10
also output the contents of "global.dat", we should not output it.
in restoreAllDatabases, we can do the following change:
```
/* Open global.dat file and execute/append all the global sql commands. */
if (!opts->tocSummary)
process_global_sql_commands(conn, dumpdirpath, opts->filename);
```
what should happen with
$BIN10/pg_restore --format=directory --globals-only --verbose dir10 --list
Should we error out saying "--globals-only" and "--list" are conflict options?
if so then in main function we can do the following change:
```
if (globals_only)
{
process_global_sql_commands(conn, inputFileSpec, opts->filename);
if (conn)
PQfinish(conn);
pg_log_info("databases restoring is skipped as -g/--globals-only
option is specified");
}
```
in restoreAllDatabases, if num_db_restore == 0, we will still call
process_global_sql_commands.
I am not sure this is what we expected.
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-02-04 07:42 ` jian he <[email protected]>
1 sibling, 0 replies; 36+ messages in thread
From: jian he @ 2025-02-04 07:42 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
hi.
This attached patch solves problems mentioned in [1].
so pg_restore --file restoring multiple databases will have the
```\connect dbname``` command in it.
the output plain text file can be used in psql.
pg_restore --file output will be:
--
-- Database "template1" dump
--
-- Dumped from database version 18devel_debug_build_622f678c10
-- Dumped by pg_dump version 18devel_debug_build_622f678c10
-- Started on 2025-02-04 14:34:44 CST
\connect template1
.....
-- Completed on 2025-02-04 14:34:53 CST
--
-- Database "template1" dump complete
--
[1] https://postgr.es/m/CACJufxFrzYJ0oZNm=v9hg10UpPQNe+p0+2ydNirHxyhUT_JtXw@mail.gmail.com
Attachments:
[application/octet-stream] v15-0001-make-pg_restore-file-option-using-connect-for.no-cfbot (7.1K, 2-v15-0001-make-pg_restore-file-option-using-connect-for.no-cfbot)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-02-11 05:40 ` Mahendra Singh Thalor <[email protected]>
2025-02-11 15:09 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
1 sibling, 1 reply; 36+ messages in thread
From: Mahendra Singh Thalor @ 2025-02-11 05:40 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
Thanks Jian.
On Tue, 4 Feb 2025 at 07:35, jian he <[email protected]> wrote:
>
> hi.
>
> just a quick response for v15.
>
> the pg_restore man page says option --list as "List the table of
> contents of the archive".
> but
> $BIN10/pg_restore --format=directory --list --file=1.sql dir10
> also output the contents of "global.dat", we should not output it.
I think we can add an error for --list option if used with the dump of
pg_dumpall. If a user wants to use --list option, then they can use a
single dump file.
>
> in restoreAllDatabases, we can do the following change:
> ```
> /* Open global.dat file and execute/append all the global sql commands. */
> if (!opts->tocSummary)
> process_global_sql_commands(conn, dumpdirpath, opts->filename);
> ```
>
>
> what should happen with
> $BIN10/pg_restore --format=directory --globals-only --verbose dir10 --list
>
> Should we error out saying "--globals-only" and "--list" are conflict options?
> if so then in main function we can do the following change:
Fixed.
>
> ```
> if (globals_only)
> {
> process_global_sql_commands(conn, inputFileSpec, opts->filename);
> if (conn)
> PQfinish(conn);
> pg_log_info("databases restoring is skipped as -g/--globals-only
> option is specified");
> }
> ```
>
>
> in restoreAllDatabases, if num_db_restore == 0, we will still call
> process_global_sql_commands.
> I am not sure this is what we expected.
This is correct. We should run global commands as we are dumping those
even if we don't dump any database.
Apart from these, I merged v15 delta to print db names. Either we can
print the db name or we can remove also but as of now, I merged delta
patch.
Here, I am attaching an updated patch for review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v16_pg_dumpall-with-non-text_format-11th_feb.patch (75.8K, 2-v16_pg_dumpall-with-non-text_format-11th_feb.patch)
download | inline diff:
From 36aea11e07ee22838d9698faa0fa5518e329abc4 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Tue, 11 Feb 2025 10:58:04 +0530
Subject: [PATCH] pg_dumpall with directory|tar|custom format and restore it
by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat to restore all databases. If global.dat file is exist in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list (if exist)
TODO1: We need to think for --exclude-database=PATTERN for pg_restore.
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
if db connection,
if no db connection, then PATTERN=NAME matching only
TODO2: We need to make changes for exit_nicely as we are one entry for each database while
restoring. MAX_ON_EXIT_NICELY
TODO3: some more test cases for new added options.
TODO4: We can dump and restore databases in parallel mode.
This needs more study
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
---
---
doc/src/sgml/ref/pg_dumpall.sgml | 80 ++-
doc/src/sgml/ref/pg_restore.sgml | 31 +
src/bin/pg_dump/Makefile | 8 +-
src/bin/pg_dump/common_dumpall_restore.c | 286 ++++++++
src/bin/pg_dump/common_dumpall_restore.h | 26 +
src/bin/pg_dump/meson.build | 2 +
src/bin/pg_dump/pg_backup.h | 4 +-
src/bin/pg_dump/pg_backup_archiver.c | 41 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 552 +++++++---------
src/bin/pg_dump/pg_restore.c | 797 ++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 9 +
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 1513 insertions(+), 332 deletions(-)
create mode 100644 src/bin/pg_dump/common_dumpall_restore.c
create mode 100644 src/bin/pg_dump/common_dumpall_restore.h
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 39d93c2c0e3..6e1975f5ff0 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster based on specified dump format </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -121,7 +121,83 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- </para>
+ Note: This option can be omitted only when <option>--format</option> is plain
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index b8b27e1719e..0609b7eb534 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -20,6 +20,8 @@ PostgreSQL documentation
<refpurpose>
restore a <productname>PostgreSQL</productname> database from an
archive file created by <application>pg_dump</application>
+ or restore multiple <productname>PostgreSQL</productname> database from an
+ archive directory is created by <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -166,6 +168,25 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +336,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..a4e557d62c7 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -47,11 +47,11 @@ all: pg_dump pg_restore pg_dumpall
pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dump.o common.o pg_dump_sort.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_restore: pg_restore.o common_dumpall_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_restore.o common_dumpall_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/common_dumpall_restore.c b/src/bin/pg_dump/common_dumpall_restore.c
new file mode 100644
index 00000000000..b162cf69412
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.c
@@ -0,0 +1,286 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.c
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * This is a common file for pg_dumpall and pg_restore.
+ * src/bin/pg_dump/common_dumpall_restore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+#define exit_nicely(code) exit(code)
+
+/*
+ * connectDatabase
+ *
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the global variable 'connstr' is set to a connection string
+ * containing the options used.
+ */
+PGconn *
+connectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ static char *password = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 6;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. Remember the options used, in the form of a
+ * connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If needed, then copy server version to outer function. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/*
+ * constructConnStr
+ *
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * executeQuery
+ *
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
diff --git a/src/bin/pg_dump/common_dumpall_restore.h b/src/bin/pg_dump/common_dumpall_restore.h
new file mode 100644
index 00000000000..a0dcdbe0807
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.h
+ * Common header file for pg_dumpall and pg_restore
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/common_dumpall_restore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef COMMON_DUMPALL_RESTORE_H
+#define COMMON_DUMPALL_RESTORE_H
+
+#include "pg_backup.h"
+
+/* TODO: increasing this to keep 100 db restoring by single pg_restore command. */
+#define MAX_ON_EXIT_NICELY 100
+extern PGconn *connectDatabase(const char *dbname, const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+#endif /* COMMON_DUMPALL_RESTORE_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..ddecac5cf09 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -49,6 +49,7 @@ bin_targets += pg_dump
pg_dumpall_sources = files(
'pg_dumpall.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
@@ -68,6 +69,7 @@ bin_targets += pg_dumpall
pg_restore_sources = files(
'pg_restore.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index f0f19bb0b29..729ffc9e124 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -306,7 +306,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, const char *dbname);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
@@ -319,7 +319,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
DataDirSyncMethod sync_method);
/* The --list option */
-extern void PrintTOCSummary(Archive *AHX);
+extern void PrintTOCSummary(Archive *AHX, bool append_data);
extern RestoreOptions *NewRestoreOptions(void);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 707a3fc844c..fd6fd16642e 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -82,7 +82,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -331,9 +331,16 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ * If dbname is not NULL, then pg_restore restore archive to file will have
+ * comments about which database currently is being dumped.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, const char *dbname)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -450,9 +457,12 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
- ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
+ if (append_data && dbname != NULL)
+ ahprintf(AH, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
+ else
+ ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
if (AH->archiveRemoteVersion)
ahprintf(AH, "-- Dumped from database version %s\n",
@@ -792,8 +802,10 @@ RestoreArchive(Archive *AHX)
if (AH->public.verbose)
dumpTimestamp(AH, "Completed on", time(NULL));
- ahprintf(AH, "--\n-- PostgreSQL database dump complete\n--\n\n");
-
+ if (append_data && dbname != NULL)
+ ahprintf(AH, "--\n-- Database \"%s\" dump complete\n--\n\n", dbname);
+ else
+ ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
/*
* Clean up & we're done.
*/
@@ -1263,7 +1275,7 @@ ArchiveEntry(Archive *AHX, CatalogId catalogId, DumpId dumpId,
/* Public */
void
-PrintTOCSummary(Archive *AHX)
+PrintTOCSummary(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -1279,7 +1291,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, append_data);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1658,7 +1670,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1678,7 +1691,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
@@ -2920,13 +2933,13 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
/*
* DATABASE and DATABASE PROPERTIES also have a special rule: they are
- * restored in createDB mode, and not restored otherwise, independently of
- * all else.
+ * restored in createDB mode or restored format is not plain file, and not
+ * restored otherwise, independently of all else.
*/
if (strcmp(te->desc, "DATABASE") == 0 ||
strcmp(te->desc, "DATABASE PROPERTIES") == 0)
{
- if (ropt->createDB)
+ if (ropt->createDB || AH->format != archNull)
return REQ_SCHEMA;
else
return 0;
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..45f0fb46e0f 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, NULL);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..47589cca90f 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -13,6 +13,7 @@
*/
#include "postgres_fe.h"
+#include "common_dumpall_restore.h"
#ifdef WIN32
#include "parallel.h"
#endif
@@ -21,8 +22,6 @@
/* Globals exported by this file */
const char *progname = NULL;
-#define MAX_ON_EXIT_NICELY 20
-
static struct
{
on_exit_nicely_callback function;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 02e1fdf8f78..51c595a7a5f 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1148,7 +1148,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, NULL);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 396f79781c5..80341db324d 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -24,14 +25,17 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
+#define exit_nicely(code) exit(code)
typedef struct
{
@@ -64,28 +68,25 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -107,7 +108,7 @@ static int no_subscriptions = 0;
static int no_toast_compression = 0;
static int no_unlogged_table_data = 0;
static int no_role_passwords = 0;
-static int server_version;
+static int server_version;
static int load_via_partition_root = 0;
static int on_conflict_do_nothing = 0;
@@ -121,8 +122,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -147,6 +146,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -188,6 +188,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -237,7 +239,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -265,7 +267,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -414,6 +418,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If non-plain format is specified then we must provide the
+ * file name to create one main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -460,6 +479,33 @@ main(int argc, char *argv[])
if (on_conflict_do_nothing)
appendPQExpBufferStr(pgdumpopts, " --on-conflict-do-nothing");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char toc_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -468,7 +514,8 @@ main(int argc, char *argv[])
if (pgdb)
{
conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
@@ -476,10 +523,12 @@ main(int argc, char *argv[])
else
{
conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ prompt_password, true,
+ progname, &connstr, &server_version);
if (!conn)
{
@@ -496,19 +545,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -607,7 +643,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -620,7 +656,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -631,12 +667,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster based on specified dump format.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1487,10 +1525,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1504,7 +1545,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1512,9 +1553,33 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_log_error("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1529,6 +1594,18 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is non-plain dump format, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1547,9 +1624,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1558,19 +1643,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char toc_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(toc_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ toc_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1580,7 +1676,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1589,17 +1686,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is non-plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1649,256 +1765,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
@@ -1994,3 +1860,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if(!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c602272d7db..af7d815a770 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,27 +41,71 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/connect.h"
+#include "compress_io.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
+#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
+typedef struct SimpleDatabaseOidListCell
+{
+ struct SimpleDatabaseOidListCell *next;
+ Oid db_oid;
+ const char *db_name;
+} SimpleDatabaseOidListCell;
+
+typedef struct SimpleDatabaseOidList
+{
+ SimpleDatabaseOidListCell *head;
+ SimpleDatabaseOidListCell *tail;
+} SimpleDatabaseOidList;
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data,
+ const char *dbname);
+static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static void process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static void copy_global_file_to_out_file(const char *outfile, FILE *pfile);
+static int filter_dbnames_for_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleDatabaseOidList *dbname_oid_list);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname);
+static void simple_string_full_list_delete(SimpleStringList *list);
+static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
+static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list,
+ Oid db_oid, const char *dbname);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
static int disable_triggers = 0;
static int enable_row_security = 0;
@@ -77,11 +121,14 @@ main(int argc, char **argv)
static int strict_names = 0;
bool data_only = false;
bool schema_only = false;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -128,6 +175,7 @@ main(int argc, char **argv)
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -156,7 +204,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "aAcCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -183,11 +231,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -302,6 +353,10 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6:
+ /* list of databases patterns those needs to skip while restoring */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -329,6 +384,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -404,6 +466,86 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ !IsFileExistsInDirectory(inputFileSpec, "toc.dat"))
+ {
+ /* If global.dat is exist, then process it. */
+ if (IsFileExistsInDirectory(pg_strdup(inputFileSpec), "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql commands. */
+ int exit_code = 0;
+
+ /*
+ * User is suggested to use single database dump for --list option.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when using dump of pg_dumpall");
+
+ /*
+ * Connect to database to execute global sql commands from
+ * global.dat file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = connectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ /*
+ * Open global.dat file and execute/append all the global sql
+ * commands.
+ */
+ process_global_sql_commands(conn, inputFileSpec, opts->filename);
+
+ if (conn)
+ PQfinish(conn);
+
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ exit_code = restoreAllDatabases(conn, inputFileSpec,
+ db_exclude_patterns,
+ opts,
+ numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_full_list_delete(&db_exclude_patterns);
+
+ return exit_code;
+ }
+ }
+
+ return restoreOneDatabase(inputFileSpec, opts, numWorkers, false, NULL);
+}
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ * dbname is the current to be restored database name.
+ */
+static int
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, const char *dbname)
+{
+ Archive *AH;
+ int exit_code;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -429,11 +571,11 @@ main(int argc, char **argv)
AH->numWorkers = numWorkers;
if (opts->tocSummary)
- PrintTOCSummary(AH);
+ PrintTOCSummary(AH, append_data);
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, dbname);
}
/* done, print a summary of ignored errors */
@@ -469,6 +611,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -481,6 +624,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -619,3 +763,642 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * IsFileExistsInDirectory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+IsFileExistsInDirectory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.sql file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+ReadOneStatement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+ int m;
+
+ StringInfoData q;
+ initStringInfo(&q);
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != '\n' && c != ';')
+ {
+ appendStringInfoChar(inBuf, (char) c);
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != ';' && c != '\n')
+ appendStringInfoChar(inBuf, (char) c);
+ else
+ break;
+ }
+ }
+
+ if (c == '\'' || c == '"')
+ {
+ appendStringInfoChar(&q, (char) c);
+ m = c;
+
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(&q, (char) c);
+
+ if(c == m)
+ {
+ appendStringInfoString(inBuf, q.data);
+ resetStringInfo(&q);
+ break;
+ }
+ }
+ }
+
+ if (c == ';')
+ {
+ appendStringInfoChar(inBuf, (char) ';');
+ break;
+ }
+
+ if (c == '\n')
+ appendStringInfoChar(inBuf, (char) '\n');
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * filter_dbnames_for_restore
+ *
+ * This will remove names from all dblist those can
+ * be constructed from database_exclude_pattern list.
+ *
+ * returns number of dbnames those will be restored.
+ */
+static int
+filter_dbnames_for_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
+ SimpleDatabaseOidListCell *dboidprecell = NULL;
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ /* Return 0 if there is no db to restore. */
+ if (dboid_cell == NULL)
+ return 0;
+
+ query = createPQExpBuffer();
+
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ while (dboid_cell != NULL)
+ {
+ bool skip_db_restore = false;
+ SimpleDatabaseOidListCell *next = dboid_cell->next;
+
+ for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ {
+ /*
+ * the construct pattern matching query:
+ * SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE
+ * pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from the
+ * dboid_list variable, which is initially extracted from the
+ * map.dat file located in the backup directory. that's why we
+ * need quote_literal_cstr.
+ *
+ * If we don't have db connection, then consider patterns as NAME
+ * only.
+ */
+ if (pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0)
+ skip_db_restore = true;
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, celldb->val, false,
+ false, NULL, quote_literal_cstr(dboid_cell->db_name),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ celldb->val);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database \"%s\" is matching with exclude pattern: \"%s\"", dboid_cell->db_name, celldb->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /* Increment count if db needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
+ simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ }
+ else
+ {
+ count_db++; /* Increment db counter. */
+ dboidprecell = dboid_cell;
+ }
+
+ /* Process next dbname from dbname list. */
+ dboid_cell = next;
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ {
+ pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid = InvalidOid;
+ char db_oid_str[MAXPGPATH + 1] = {'\0'};
+ char dbname[MAXPGPATH + 1] = {'\0'};
+
+ /* Extract dboid. */
+ sscanf(line, "%u" , &db_oid);
+ sscanf(line, "%s" , db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found database \"%s\" (OID: %u) in map.dat file while restoring.", dbname, db_oid);
+
+ /* Report error if file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
+ SimpleDatabaseOidListCell *dboid_cell;
+ int exit_code = 0;
+ int num_db_restore = 0;
+ int num_total_db;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /* If map.dat has no entry, return from here. */
+ if (dbname_oid_list.head == NULL)
+ return 0;
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ if (!conn)
+ pg_log_info("there is no database connection so consider pattern as simple name for --exclude-database");
+ }
+ }
+
+ /*
+ * processing pg_retsore --exclude-database=PATTERN/NAME if no connection.
+ */
+ num_db_restore = filter_dbnames_for_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /*
+ * To restore multiple databases, -C (create database) option should be specified
+ * or all databases should be created before pg_restore.
+ */
+ if (opts->createDB != 1)
+ pg_log_info("restoring dump of pg_dumpall without -C option, there might be multiple databases in directory.");
+
+ /* TODO: MAX_ON_EXIT_NICELY is 100 now... max AH handle register on exit .*/
+ if (num_db_restore > MAX_ON_EXIT_NICELY)
+ {
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+ pg_fatal("cound not restore more than %d databases by single pg_restore, here total database:%d",
+ MAX_ON_EXIT_NICELY,
+ num_db_restore);
+ }
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL)
+ {
+ pg_log_info("no database needs to restore out of %d databases", num_total_db);
+ return 0;
+ }
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * XXX: TODO till now, we made a list of databases, those needs to be restored
+ * after skipping names of exclude-database. Now we can launch parallel
+ * workers to restore these databases.
+ */
+ dboid_cell = dbname_oid_list.head;
+
+ while(dboid_cell != NULL)
+ {
+ char subdirpath[MAXPGPATH];
+ int dbexit_code;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored into
+ * already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+
+ pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+
+ dbexit_code = restoreOneDatabase(subdirpath, opts, numWorkers, true, dboid_cell->db_name);
+
+ /* Store exit_code to report it back. */
+ if (exit_code == 0 && dbexit_code != 0)
+ exit_code = dbexit_code;
+
+ dboid_cell = dboid_cell->next;
+ }
+
+ /* Log number of processed databases.*/
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+
+ return exit_code;
+}
+
+/*
+ * process_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ */
+static void
+process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Now open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ copy_global_file_to_out_file(outfile, pfile);
+ return;
+ }
+
+ /* Init sqlstatement to append commands. */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements. */
+ while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ {
+ pg_log_info("executing %s", sqlstatement.data);
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ fclose(pfile);
+}
+
+/*
+ * copy_global_file_to_out_file
+ *
+ * This will copy global.dat file into out file. If "-" is used as outfile,
+ * then print commands to the stdout.
+ */
+static void
+copy_global_file_to_out_file(const char *outfile, FILE *pfile)
+{
+ char out_file_path[MAXPGPATH];
+ FILE *OPF;
+ int c;
+
+ /* "-" is used for stdout. */
+ if (strcmp(outfile, "-") == 0)
+ OPF = stdout;
+ else
+ {
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ OPF = fopen(out_file_path, PG_BINARY_W);
+
+ if (OPF == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", outfile);
+ }
+ }
+
+ /* Now append global.dat into out file. */
+ while ((c = fgetc(pfile)) != EOF)
+ fputc(c, OPF);
+
+ fclose(pfile);
+
+ /* Close out file. */
+ if (strcmp(outfile, "-") != 0)
+ fclose(OPF);
+}
+
+/*
+ * simple_db_oid_list_append
+ *
+ * appends a node to the list in the end.
+ */
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname)
+{
+ SimpleDatabaseOidListCell *cell;
+
+ cell = pg_malloc_object(SimpleDatabaseOidListCell);
+
+ cell->next = NULL;
+ cell->db_oid = db_oid;
+ cell->db_name = pg_strdup(dbname);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * simple_db_oid_full_list_delete
+ *
+ * delete all cell from dbname and dboid list.
+ */
+static void
+simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
+{
+ SimpleDatabaseOidListCell *cell = list->head;
+ SimpleDatabaseOidListCell *nextcell = NULL;
+
+ while (cell)
+ {
+ nextcell = cell->next;
+ pfree (cell);
+ cell = nextcell;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_string_full_list_delete
+ *
+ * delete all cell from string list.
+ */
+static void
+simple_string_full_list_delete(SimpleStringList *list)
+{
+ SimpleStringListCell *cell = list->head;
+ SimpleStringListCell *cellnext = NULL;
+
+ while (cell)
+ {
+ cellnext = cell->next;
+ pfree(cell);
+ cell = cellnext;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_db_oid_list_delete
+ *
+ * delete cell from database and oid list.
+ */
+static void
+simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev)
+{
+ if (prev == NULL)
+ {
+ list->head = cell->next;
+ pfree(cell);
+ }
+ else
+ {
+ prev->next = cell->next;
+ pfree(cell);
+ }
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr
+ *
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 214240f1ae5..de41ec06d86
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -219,6 +219,11 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -226,4 +231,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9a3bee93dec..cdaf1ad343c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2673,6 +2673,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.39.3
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 05:40 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-02-11 15:09 ` jian he <[email protected]>
2025-02-11 17:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: jian he @ 2025-02-11 15:09 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
hi.
review based on v16.
because of
https://postgr.es/m/CAFC+b6pWQiSL+3rvLxN9vhC8aONp4OV9c6u+BVD6kmWmDbd1WQ@mail.gmail.com
in copy_global_file_to_out_file, now it is:
if (strcmp(outfile, "-") == 0)
OPF = stdout;
I am confused, why "-" means stdout.
``touch ./- `` command works fine.
i think dash is not special character, you may see
https://stackoverflow.com/a/40650391/15603477
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_log_error("could not create subdirectory \"%s\": %m", db_subdir);
here we should use pg_fatal?
pg_log_info("executing %s", sqlstatement.data);
change to
pg_log_info("executing query: %s", sqlstatement.data);
message would be more similar to the next pg_log_error(...) message.
+ /*
+ * User is suggested to use single database dump for --list option.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when using dump of pg_dumpall");
maybe change to
+ pg_fatal("option -l/--list cannot be used when restoring multiple databases");
$BIN10/pg_restore --format=directory --list dir10_x
if the directory only has one database, then we can actually print out
the tocSummary.
if the directory has more than one database then pg_fatal.
To tolerate this corner case (only one database) means that pg_restore
--list requires a DB connection,
but I am not sure that is fine.
anyway, the attached patch allows this corner case.
PrintTOCSummary can only print out summary for a single database.
so we don't need to change PrintTOCSummary.
+ /*
+ * To restore multiple databases, -C (create database) option should
be specified
+ * or all databases should be created before pg_restore.
+ */
+ if (opts->createDB != 1)
+ pg_log_info("restoring dump of pg_dumpall without -C option, there
might be multiple databases in directory.");
we can change it to
+ if (opts->createDB != 1 && num_db_restore > 0)
+ pg_log_info("restoring multiple databases without -C option.");
Bug.
when pg_restore --globals-only can be applied when we are restoring a
single database (can be an output of pg_dump).
There are some tests per https://commitfest.postgresql.org/52/5495, I
will check it later.
The attached patch is the change for the above reviews.
Attachments:
[application/octet-stream] v16_misc_changes.nocfbot (6.9K, 2-v16_misc_changes.nocfbot)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 05:40 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-11 15:09 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-02-11 17:17 ` Mahendra Singh Thalor <[email protected]>
2025-02-12 07:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Mahendra Singh Thalor @ 2025-02-11 17:17 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
On Tue, 11 Feb 2025 at 20:40, jian he <[email protected]> wrote:
>
> hi.
> review based on v16.
>
> because of
>
https://postgr.es/m/CAFC+b6pWQiSL+3rvLxN9vhC8aONp4OV9c6u+BVD6kmWmDbd1WQ@mail.gmail.com
>
> in copy_global_file_to_out_file, now it is:
> if (strcmp(outfile, "-") == 0)
> OPF = stdout;
> I am confused, why "-" means stdout.
> ``touch ./- `` command works fine.
> i think dash is not special character, you may see
> https://stackoverflow.com/a/40650391/15603477
"-" is used for stdout. This is mentioned in the doc.
pg_restore link <https://www.postgresql.org/docs/current/app-pgrestore.html;
> -f *filename*
> --file=*filename*
>
> Specify output file for generated script, or for the listing when used
> with -l. Use - for stdout.
>
>
>
> + /* Create a subdirectory with 'databases' name under main directory. */
> + if (mkdir(db_subdir, 0755) != 0)
> + pg_log_error("could not create subdirectory \"%s\": %m", db_subdir);
> here we should use pg_fatal?
Yes, we should use pg_fatal.
>
>
> pg_log_info("executing %s", sqlstatement.data);
> change to
> pg_log_info("executing query: %s", sqlstatement.data);
> message would be more similar to the next pg_log_error(...) message.
Okay.
>
>
> + /*
> + * User is suggested to use single database dump for --list option.
> + */
> + if (opts->tocSummary)
> + pg_fatal("option -l/--list cannot be used when using dump of
pg_dumpall");
> maybe change to
> + pg_fatal("option -l/--list cannot be used when restoring multiple
databases");
okay.
>
> $BIN10/pg_restore --format=directory --list dir10_x
> if the directory only has one database, then we can actually print out
> the tocSummary.
> if the directory has more than one database then pg_fatal.
> To tolerate this corner case (only one database) means that pg_restore
> --list requires a DB connection,
> but I am not sure that is fine.
> anyway, the attached patch allows this corner case.
No, we don't need this corner case. If a user wants to restore a
single database with --list option, then the user should give a particular
dump file with pg_restore.
>
>
> PrintTOCSummary can only print out summary for a single database.
> so we don't need to change PrintTOCSummary.
>
>
> + /*
> + * To restore multiple databases, -C (create database) option should
> be specified
> + * or all databases should be created before pg_restore.
> + */
> + if (opts->createDB != 1)
> + pg_log_info("restoring dump of pg_dumpall without -C option, there
> might be multiple databases in directory.");
>
> we can change it to
> + if (opts->createDB != 1 && num_db_restore > 0)
> + pg_log_info("restoring multiple databases without -C option.");
okay.
>
>
> Bug.
> when pg_restore --globals-only can be applied when we are restoring a
> single database (can be an output of pg_dump).
As of now, we are ignoring this option. We can add an error in the "else"
part of the global.dat file.
Ex: option --globals-only is only supported with dump of pg_dumpall.
Similarly --exclude-database also.
>
>
> There are some tests per https://commitfest.postgresql.org/52/5495, I
> will check it later.
> The attached patch is the change for the above reviews.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 05:40 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-11 15:09 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 17:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-02-12 07:14 ` jian he <[email protected]>
2025-02-13 11:25 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: jian he @ 2025-02-12 07:14 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
On Wed, Feb 12, 2025 at 1:17 AM Mahendra Singh Thalor
<[email protected]> wrote:
>
> >
> > There are some tests per https://commitfest.postgresql.org/52/5495, I
> > will check it later.
hi.
the cfbot failure is related to function _tocEntryRequired
if (strcmp(te->desc, "DATABASE") == 0 ||
strcmp(te->desc, "DATABASE PROPERTIES") == 0)
{
- if (ropt->createDB)
+ if (ropt->createDB || AH->format != archNull)
return REQ_SCHEMA;
else
return 0;
for restoring multiple databases:
in v16 implementation: pg_restore even if you do not specify --create,
it actually did what pg_restore --create option does.
if there are multiple databases in the archive:
to make the pg_restore --file output is usable, the output file need
have \connect and CREATE DATABASE
command. that is exactly what --create option would do.
pg_restore --file behavior need align with pg_restore --dbname.
therefore pg_restore restoring multiple databases will use --create option.
we can either error out (pg_fatal) saying
restoring multiple databases requires the pg_restore --create option.
Or we can add a pg_log_info saying
pg_restore --create option will be set to true while restoring
multiple databases.
for restoring one database, the master behavior is fine.
so we don't need to change _tocEntryRequired.
Attachments:
[application/octet-stream] v16_misc.nocfbot (2.2K, 2-v16_misc.nocfbot)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 05:40 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-11 15:09 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 17:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-12 07:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-02-13 11:25 ` Mahendra Singh Thalor <[email protected]>
2025-02-18 04:30 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-02-18 06:10 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
0 siblings, 2 replies; 36+ messages in thread
From: Mahendra Singh Thalor @ 2025-02-13 11:25 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
Thanks Jian.
On Wed, 12 Feb 2025 at 12:45, jian he <[email protected]> wrote:
>
> On Wed, Feb 12, 2025 at 1:17 AM Mahendra Singh Thalor
> <[email protected]> wrote:
> >
> > >
> > > There are some tests per https://commitfest.postgresql.org/52/5495, I
> > > will check it later.
>
> hi.
> the cfbot failure is related to function _tocEntryRequired
>
> if (strcmp(te->desc, "DATABASE") == 0 ||
> strcmp(te->desc, "DATABASE PROPERTIES") == 0)
> {
> - if (ropt->createDB)
> + if (ropt->createDB || AH->format != archNull)
> return REQ_SCHEMA;
> else
> return 0;
>
> for restoring multiple databases:
> in v16 implementation: pg_restore even if you do not specify --create,
> it actually did what pg_restore --create option does.
>
> if there are multiple databases in the archive:
> to make the pg_restore --file output is usable, the output file need
> have \connect and CREATE DATABASE
> command. that is exactly what --create option would do.
> pg_restore --file behavior need align with pg_restore --dbname.
> therefore pg_restore restoring multiple databases will use --create option.
>
>
> we can either error out (pg_fatal) saying
> restoring multiple databases requires the pg_restore --create option.
> Or we can add a pg_log_info saying
> pg_restore --create option will be set to true while restoring
> multiple databases.
In my earlier version, I was giving an error if --create option was
not specified.
I think it will be good and more preferable if we give an error
without the --create option if dump was taken from pg_dumpall. Even
though there is a single database in the dump of pg_dumpall, it is
possible that a particular database hasn't been created.
Ex: -d postgres and we have db1 dump in file. In this case, we have
only one database dump but this database has not been created.
If the user wants to restore a single database, then the user should
use a single database dump file. Forcefully adding --create option is
not a good idea, instead we will give an error to the user and let him
correct the inputs.
Apart from the above handling, I fixed all the pending review comments
in this patch and made some more changes.
Here, I am attaching an updated patch for review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v17_pg_dumpall-with-non-text_format-13th_feb.patch (77.0K, 2-v17_pg_dumpall-with-non-text_format-13th_feb.patch)
download | inline diff:
From 6c02c291516e812943dc5e1f3e7122c77579a5c6 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Thu, 13 Feb 2025 16:41:18 +0530
Subject: [PATCH] pg_dumpall with directory|tar|custom format and restore it
by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat to restore all databases. If global.dat file is exist in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list (if exist)
TODO1: We need to think for --exclude-database=PATTERN for pg_restore.
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
if db connection,
if no db connection, then PATTERN=NAME matching only
TODO2: We need to make changes for exit_nicely as we are one entry for each database while
restoring. MAX_ON_EXIT_NICELY
TODO3: some more test cases for new added options.
TODO4: We can dump and restore databases in parallel mode.
This needs more study
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
---
---
---
doc/src/sgml/ref/pg_dumpall.sgml | 80 ++-
doc/src/sgml/ref/pg_restore.sgml | 31 +
src/bin/pg_dump/Makefile | 8 +-
src/bin/pg_dump/common_dumpall_restore.c | 286 ++++++++
src/bin/pg_dump/common_dumpall_restore.h | 26 +
src/bin/pg_dump/meson.build | 2 +
src/bin/pg_dump/pg_backup.h | 4 +-
src/bin/pg_dump/pg_backup_archiver.c | 35 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 552 +++++++--------
src/bin/pg_dump/pg_restore.c | 819 ++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 9 +
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 1529 insertions(+), 332 deletions(-)
create mode 100644 src/bin/pg_dump/common_dumpall_restore.c
create mode 100644 src/bin/pg_dump/common_dumpall_restore.h
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 39d93c2c0e3..6e1975f5ff0 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster based on specified dump format </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -121,7 +121,83 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- </para>
+ Note: This option can be omitted only when <option>--format</option> is plain
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index b8b27e1719e..924d5bac817 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -20,6 +20,8 @@ PostgreSQL documentation
<refpurpose>
restore a <productname>PostgreSQL</productname> database from an
archive file created by <application>pg_dump</application>
+ or restore multiple <productname>PostgreSQL</productname> database from an
+ archive directory created by <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -166,6 +168,25 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +336,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..a4e557d62c7 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -47,11 +47,11 @@ all: pg_dump pg_restore pg_dumpall
pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dump.o common.o pg_dump_sort.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_restore: pg_restore.o common_dumpall_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_restore.o common_dumpall_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/common_dumpall_restore.c b/src/bin/pg_dump/common_dumpall_restore.c
new file mode 100644
index 00000000000..b162cf69412
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.c
@@ -0,0 +1,286 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.c
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * This is a common file for pg_dumpall and pg_restore.
+ * src/bin/pg_dump/common_dumpall_restore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+#define exit_nicely(code) exit(code)
+
+/*
+ * connectDatabase
+ *
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the global variable 'connstr' is set to a connection string
+ * containing the options used.
+ */
+PGconn *
+connectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ static char *password = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 6;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. Remember the options used, in the form of a
+ * connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If needed, then copy server version to outer function. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/*
+ * constructConnStr
+ *
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * executeQuery
+ *
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
diff --git a/src/bin/pg_dump/common_dumpall_restore.h b/src/bin/pg_dump/common_dumpall_restore.h
new file mode 100644
index 00000000000..a0dcdbe0807
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.h
+ * Common header file for pg_dumpall and pg_restore
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/common_dumpall_restore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef COMMON_DUMPALL_RESTORE_H
+#define COMMON_DUMPALL_RESTORE_H
+
+#include "pg_backup.h"
+
+/* TODO: increasing this to keep 100 db restoring by single pg_restore command. */
+#define MAX_ON_EXIT_NICELY 100
+extern PGconn *connectDatabase(const char *dbname, const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+#endif /* COMMON_DUMPALL_RESTORE_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..ddecac5cf09 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -49,6 +49,7 @@ bin_targets += pg_dump
pg_dumpall_sources = files(
'pg_dumpall.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
@@ -68,6 +69,7 @@ bin_targets += pg_dumpall
pg_restore_sources = files(
'pg_restore.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index f0f19bb0b29..729ffc9e124 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -306,7 +306,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, const char *dbname);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
@@ -319,7 +319,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
DataDirSyncMethod sync_method);
/* The --list option */
-extern void PrintTOCSummary(Archive *AHX);
+extern void PrintTOCSummary(Archive *AHX, bool append_data);
extern RestoreOptions *NewRestoreOptions(void);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index b9d7ab98c3e..1dfa0420ea2 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -82,7 +82,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -331,9 +331,16 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ * If dbname is not NULL, then pg_restore restore archive to file will have
+ * comments about which database currently is being dumped.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, const char *dbname)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -450,9 +457,12 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
- ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
+ if (append_data && dbname != NULL)
+ ahprintf(AH, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
+ else
+ ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
if (AH->archiveRemoteVersion)
ahprintf(AH, "-- Dumped from database version %s\n",
@@ -792,8 +802,10 @@ RestoreArchive(Archive *AHX)
if (AH->public.verbose)
dumpTimestamp(AH, "Completed on", time(NULL));
- ahprintf(AH, "--\n-- PostgreSQL database dump complete\n--\n\n");
-
+ if (append_data && dbname != NULL)
+ ahprintf(AH, "--\n-- Database \"%s\" dump complete\n--\n\n", dbname);
+ else
+ ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
/*
* Clean up & we're done.
*/
@@ -1263,7 +1275,7 @@ ArchiveEntry(Archive *AHX, CatalogId catalogId, DumpId dumpId,
/* Public */
void
-PrintTOCSummary(Archive *AHX)
+PrintTOCSummary(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -1279,7 +1291,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, append_data);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1658,7 +1670,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1678,7 +1691,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..45f0fb46e0f 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, NULL);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..47589cca90f 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -13,6 +13,7 @@
*/
#include "postgres_fe.h"
+#include "common_dumpall_restore.h"
#ifdef WIN32
#include "parallel.h"
#endif
@@ -21,8 +22,6 @@
/* Globals exported by this file */
const char *progname = NULL;
-#define MAX_ON_EXIT_NICELY 20
-
static struct
{
on_exit_nicely_callback function;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 30dfda8c3ff..7efd59999d3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1148,7 +1148,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, NULL);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 64a60a26092..b75e4f56f31 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -24,14 +25,17 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
+#define exit_nicely(code) exit(code)
typedef struct
{
@@ -64,28 +68,25 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -107,7 +108,7 @@ static int no_subscriptions = 0;
static int no_toast_compression = 0;
static int no_unlogged_table_data = 0;
static int no_role_passwords = 0;
-static int server_version;
+static int server_version;
static int load_via_partition_root = 0;
static int on_conflict_do_nothing = 0;
@@ -121,8 +122,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -147,6 +146,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -188,6 +188,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -237,7 +239,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -265,7 +267,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -414,6 +418,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If non-plain format is specified then we must provide the
+ * file name to create one main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -460,6 +479,33 @@ main(int argc, char *argv[])
if (on_conflict_do_nothing)
appendPQExpBufferStr(pgdumpopts, " --on-conflict-do-nothing");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char toc_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -468,7 +514,8 @@ main(int argc, char *argv[])
if (pgdb)
{
conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
@@ -476,10 +523,12 @@ main(int argc, char *argv[])
else
{
conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ prompt_password, true,
+ progname, &connstr, &server_version);
if (!conn)
{
@@ -496,19 +545,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -608,7 +644,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -621,7 +657,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -632,12 +668,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster based on specified dump format.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1488,10 +1526,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1505,7 +1546,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1513,9 +1554,33 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_fatal("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1530,6 +1595,18 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is non-plain dump format, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1548,9 +1625,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1559,19 +1644,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char toc_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(toc_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ toc_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1581,7 +1677,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1590,17 +1687,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is non-plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1650,256 +1766,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
@@ -1995,3 +1861,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if(!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c602272d7db..d5431297a1e 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,27 +41,71 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/connect.h"
+#include "compress_io.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
+#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
+typedef struct SimpleDatabaseOidListCell
+{
+ struct SimpleDatabaseOidListCell *next;
+ Oid db_oid;
+ const char *db_name;
+} SimpleDatabaseOidListCell;
+
+typedef struct SimpleDatabaseOidList
+{
+ SimpleDatabaseOidListCell *head;
+ SimpleDatabaseOidListCell *tail;
+} SimpleDatabaseOidList;
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data,
+ const char *dbname);
+static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static void process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static void copy_global_file_to_out_file(const char *outfile, FILE *pfile);
+static int filter_dbnames_for_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleDatabaseOidList *dbname_oid_list);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname);
+static void simple_string_full_list_delete(SimpleStringList *list);
+static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
+static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list,
+ Oid db_oid, const char *dbname);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
static int disable_triggers = 0;
static int enable_row_security = 0;
@@ -77,11 +121,14 @@ main(int argc, char **argv)
static int strict_names = 0;
bool data_only = false;
bool schema_only = false;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -128,6 +175,7 @@ main(int argc, char **argv)
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -156,7 +204,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "aAcCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -183,11 +231,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -302,6 +353,10 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6:
+ /* list of databases patterns those needs to skip while restoring */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -329,6 +384,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -404,6 +466,106 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL && !IsFileExistsInDirectory(inputFileSpec, "toc.dat"))
+ {
+ /* If global.dat is exist, then process it. */
+ if (IsFileExistsInDirectory(pg_strdup(inputFileSpec), "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql commands. */
+ int exit_code = 0;
+
+ /*
+ * User is suggested to use single database dump for --list option.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring multiple databases by archive of pg_dumpall");
+
+ /*
+ * To restore multiple databases, -C (create database) option
+ * should be specified.
+ * Even there is single database in dump, report error because it
+ * might be possible that database hasn't created so better we
+ * report error.
+ */
+ if (opts->createDB != 1)
+ {
+ pg_log_error("-C/--create option should be specified when restoring multiple databases by archive of pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("If db is already created and dump has single db dump, then use particular dump file.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Connect to database to execute global sql commands from
+ * global.dat file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = connectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ /*
+ * Open global.dat file and execute/append all the global sql
+ * commands.
+ */
+ process_global_sql_commands(conn, inputFileSpec, opts->filename);
+
+ if (conn)
+ PQfinish(conn);
+
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ exit_code = restoreAllDatabases(conn, inputFileSpec,
+ db_exclude_patterns,
+ opts,
+ numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_full_list_delete(&db_exclude_patterns);
+
+ return exit_code;
+ }
+ }
+
+ if (db_exclude_patterns.head != NULL)
+ pg_fatal("option --exclude-database can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ return restoreOneDatabase(inputFileSpec, opts, numWorkers, false, NULL);
+}
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ * dbname is the current to be restored database name.
+ */
+static int
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, const char *dbname)
+{
+ Archive *AH;
+ int exit_code;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -429,11 +591,11 @@ main(int argc, char **argv)
AH->numWorkers = numWorkers;
if (opts->tocSummary)
- PrintTOCSummary(AH);
+ PrintTOCSummary(AH, append_data);
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, dbname);
}
/* done, print a summary of ignored errors */
@@ -451,7 +613,8 @@ main(int argc, char **argv)
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n"
+ "If archive is created by pg_dumpall, then restores multiple databases also. \n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -469,6 +632,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -481,6 +645,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -513,8 +678,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be combined\n"
+ "and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -619,3 +784,637 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * IsFileExistsInDirectory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+IsFileExistsInDirectory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.sql file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+ReadOneStatement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+ int m;
+
+ StringInfoData q;
+ initStringInfo(&q);
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != '\n' && c != ';')
+ {
+ appendStringInfoChar(inBuf, (char) c);
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != ';' && c != '\n')
+ appendStringInfoChar(inBuf, (char) c);
+ else
+ break;
+ }
+ }
+
+ if (c == '\'' || c == '"')
+ {
+ appendStringInfoChar(&q, (char) c);
+ m = c;
+
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(&q, (char) c);
+
+ if(c == m)
+ {
+ appendStringInfoString(inBuf, q.data);
+ resetStringInfo(&q);
+ break;
+ }
+ }
+ }
+
+ if (c == ';')
+ {
+ appendStringInfoChar(inBuf, (char) ';');
+ break;
+ }
+
+ if (c == '\n')
+ appendStringInfoChar(inBuf, (char) '\n');
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * filter_dbnames_for_restore
+ *
+ * This will remove names from all dblist those can
+ * be constructed from database_exclude_pattern list.
+ *
+ * returns number of dbnames those will be restored.
+ */
+static int
+filter_dbnames_for_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
+ SimpleDatabaseOidListCell *dboidprecell = NULL;
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ /* Return 0 if there is no db to restore. */
+ if (dboid_cell == NULL)
+ return 0;
+
+ query = createPQExpBuffer();
+
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ while (dboid_cell != NULL)
+ {
+ bool skip_db_restore = false;
+ SimpleDatabaseOidListCell *next = dboid_cell->next;
+
+ for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ {
+ /*
+ * the construct pattern matching query:
+ * SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE
+ * pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from the
+ * dboid_list variable, which is initially extracted from the
+ * map.dat file located in the backup directory. that's why we
+ * need quote_literal_cstr.
+ *
+ * If we don't have db connection, then consider patterns as NAME
+ * only.
+ */
+ if (pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0)
+ skip_db_restore = true;
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, celldb->val, false,
+ false, NULL, quote_literal_cstr(dboid_cell->db_name),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ celldb->val);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database \"%s\" is matching with exclude pattern: \"%s\"", dboid_cell->db_name, celldb->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /* Increment count if db needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
+ simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ }
+ else
+ {
+ count_db++; /* Increment db counter. */
+ dboidprecell = dboid_cell;
+ }
+
+ /* Process next dbname from dbname list. */
+ dboid_cell = next;
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ {
+ pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid = InvalidOid;
+ char db_oid_str[MAXPGPATH + 1] = {'\0'};
+ char dbname[MAXPGPATH + 1] = {'\0'};
+
+ /* Extract dboid. */
+ sscanf(line, "%u" , &db_oid);
+ sscanf(line, "%s" , db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found database \"%s\" (OID: %u) in map.dat file while restoring.", dbname, db_oid);
+
+ /* Report error if file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
+ SimpleDatabaseOidListCell *dboid_cell;
+ int exit_code = 0;
+ int num_db_restore = 0;
+ int num_total_db;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /* If map.dat has no entry, return from here. */
+ if (dbname_oid_list.head == NULL)
+ return 0;
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ if (!conn)
+ pg_log_info("there is no database connection so consider pattern as simple name for --exclude-database");
+ }
+ }
+
+ /*
+ * processing pg_retsore --exclude-database=PATTERN/NAME if no connection.
+ */
+ num_db_restore = filter_dbnames_for_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* TODO: MAX_ON_EXIT_NICELY is 100 now... max AH handle register on exit .*/
+ if (num_db_restore > MAX_ON_EXIT_NICELY)
+ {
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+ pg_fatal("cound not restore more than %d databases by single pg_restore, here total database:%d",
+ MAX_ON_EXIT_NICELY,
+ num_db_restore);
+ }
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL)
+ {
+ pg_log_info("no database needs to restore out of %d databases", num_total_db);
+ return 0;
+ }
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * XXX: TODO till now, we made a list of databases, those needs to be restored
+ * after skipping names of exclude-database. Now we can launch parallel
+ * workers to restore these databases.
+ */
+ dboid_cell = dbname_oid_list.head;
+
+ while(dboid_cell != NULL)
+ {
+ char subdirpath[MAXPGPATH];
+ int dbexit_code;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored into
+ * already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+
+ pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+
+ /* Restore single database and save exit_code. */
+ dbexit_code = restoreOneDatabase(subdirpath, opts, numWorkers,
+ true, dboid_cell->db_name);
+
+ /* Store exit_code to report it back. */
+ if (exit_code == 0 && dbexit_code != 0)
+ exit_code = dbexit_code;
+
+ dboid_cell = dboid_cell->next;
+ }
+
+ /* Log number of processed databases.*/
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+
+ return exit_code;
+}
+
+/*
+ * process_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ */
+static void
+process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Now open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ copy_global_file_to_out_file(outfile, pfile);
+ return;
+ }
+
+ /* Init sqlstatement to append commands. */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements. */
+ while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ {
+ pg_log_info("executing query: %s", sqlstatement.data);
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ fclose(pfile);
+}
+
+/*
+ * copy_global_file_to_out_file
+ *
+ * This will copy global.dat file into out file. If "-" is used as outfile,
+ * then print commands to the stdout.
+ */
+static void
+copy_global_file_to_out_file(const char *outfile, FILE *pfile)
+{
+ char out_file_path[MAXPGPATH];
+ FILE *OPF;
+ int c;
+
+ /* "-" is used for stdout. */
+ if (strcmp(outfile, "-") == 0)
+ OPF = stdout;
+ else
+ {
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ OPF = fopen(out_file_path, PG_BINARY_W);
+
+ if (OPF == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", outfile);
+ }
+ }
+
+ /* Now append global.dat into out file. */
+ while ((c = fgetc(pfile)) != EOF)
+ fputc(c, OPF);
+
+ fclose(pfile);
+
+ /* Close out file. */
+ if (strcmp(outfile, "-") != 0)
+ fclose(OPF);
+}
+
+/*
+ * simple_db_oid_list_append
+ *
+ * appends a node to the list in the end.
+ */
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname)
+{
+ SimpleDatabaseOidListCell *cell;
+
+ cell = pg_malloc_object(SimpleDatabaseOidListCell);
+
+ cell->next = NULL;
+ cell->db_oid = db_oid;
+ cell->db_name = pg_strdup(dbname);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * simple_db_oid_full_list_delete
+ *
+ * delete all cell from dbname and dboid list.
+ */
+static void
+simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
+{
+ SimpleDatabaseOidListCell *cell = list->head;
+ SimpleDatabaseOidListCell *nextcell = NULL;
+
+ while (cell)
+ {
+ nextcell = cell->next;
+ pfree (cell);
+ cell = nextcell;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_string_full_list_delete
+ *
+ * delete all cell from string list.
+ */
+static void
+simple_string_full_list_delete(SimpleStringList *list)
+{
+ SimpleStringListCell *cell = list->head;
+ SimpleStringListCell *cellnext = NULL;
+
+ while (cell)
+ {
+ cellnext = cell->next;
+ pfree(cell);
+ cell = cellnext;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_db_oid_list_delete
+ *
+ * delete cell from database and oid list.
+ */
+static void
+simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev)
+{
+ if (prev == NULL)
+ {
+ list->head = cell->next;
+ pfree(cell);
+ }
+ else
+ {
+ prev->next = cell->next;
+ pfree(cell);
+ }
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr
+ *
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 214240f1ae5..de41ec06d86
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -219,6 +219,11 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -226,4 +231,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b6c170ac249..5851589b0b2 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2674,6 +2674,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.39.3
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 05:40 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-11 15:09 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 17:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-12 07:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-13 11:25 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-02-18 04:30 ` Srinath Reddy <[email protected]>
2025-02-19 11:32 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
1 sibling, 1 reply; 36+ messages in thread
From: Srinath Reddy @ 2025-02-18 04:30 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; [email protected]; +Cc: [email protected]
Hi,
i think during restore we should not force user to use -C during cases like
./pg_restore pdd -g -f -
./pg_restore pdd -a -f -
./pg_restore pdd -s -f -
because its not good to use -C to create database every time when we are
using these options individually.
latest patch throws following error for all the above cases
pg_restore: error: -C/--create option should be specified when restoring
multiple databases by archive of pg_dumpall
pg_restore: hint: Try "pg_restore --help" for more information.
pg_restore: hint: If db is already created and dump has single db dump,
then use particular dump file.
Thanks and Regards
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com <http://www.enterprisedb.com/;
Srinath Reddy Sadipiralla,
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 05:40 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-11 15:09 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 17:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-12 07:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-13 11:25 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-18 04:30 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
@ 2025-02-19 11:32 ` Mahendra Singh Thalor <[email protected]>
2025-02-19 11:37 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Mahendra Singh Thalor @ 2025-02-19 11:32 UTC (permalink / raw)
To: Srinath Reddy <[email protected]>; +Cc: [email protected]; [email protected]
On Tue, 18 Feb 2025 at 10:00, Srinath Reddy <[email protected]> wrote:
>
> Hi,
> i think during restore we should not force user to use -C during cases like
> ./pg_restore pdd -g -f -
> ./pg_restore pdd -a -f -
> ./pg_restore pdd -s -f -
> because its not good to use -C to create database every time when we are using these options individually.
> latest patch throws following error for all the above cases
-g => we can allow this case without the -C option.
-a and -s => user should use this option with a single database (i
mean user should use a particular dump file to restore, not full dump
directory of all the databases.)
As pg_dumpall dumps all the databases in create mode, we should either
use --create option in our code or we should give an error. I think,
error is a good option if the user is using a dump of pg_dumpall.
If the user wants to use all the options, then the user should use a
single database dump path.
If we allow users without the --create option, then pg_restore will
create all the tables under a single database even if those tables are
in different databases.
I will fix the -g option(1st test case) in the next patch.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 05:40 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-11 15:09 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 17:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-12 07:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-13 11:25 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-18 04:30 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-02-19 11:32 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-02-19 11:37 ` jian he <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: jian he @ 2025-02-19 11:37 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
hi.
Currently, pg_retore says
--exit-on-error
Exit if an error is encountered while sending SQL commands to the
database. The default is to continue and to display a count of errors
at the end of the restoration.
Do we need to apply this to restore executing global commands (create
role, create tablespace)?
If not then we need to put some words in pg_restoe --exit-on-error
option saying that while restoring global objects --exit-on-error
option is ignored.
IMHO, in pg_restore.sgml, we need words explicitly saying that
when restoring multiple databases, all the specified options will
apply to each individual database.
I tested the following options for restoring multiple databases. The
results look good to me.
--index=index
--table=table
--schema-only
--transaction-size
--no-comments
some part of (--filter=filename)
--exclude-schema=schema
attach is a minor cosmetic change.
Attachments:
[application/octet-stream] v17_pg_dumpall.minorchange (2.2K, 2-v17_pg_dumpall.minorchange)
download
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 05:40 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-11 15:09 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 17:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-12 07:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-13 11:25 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-02-18 06:10 ` jian he <[email protected]>
2025-02-18 08:31 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-19 12:37 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
1 sibling, 2 replies; 36+ messages in thread
From: jian he @ 2025-02-18 06:10 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
hi.
<refnamediv>
<refname>pg_restore</refname>
<refpurpose>
restore a <productname>PostgreSQL</productname> database from an
archive file created by <application>pg_dump</application>
or restore multiple <productname>PostgreSQL</productname> database from an
archive directory created by <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
i think it's way too verbose. we can change it to:
<refpurpose>
restore <productname>PostgreSQL</productname> database from an
archive file created by <application>pg_dump</application> or
<application>pg_dumpall</application>
</refpurpose>
<para>
<application>pg_restore</application> is a utility for restoring a
<productname>PostgreSQL</productname> database from an archive
created by <xref linkend="app-pgdump"/> in one of the non-plain-text
formats.
we can change it to
<para>
<application>pg_restore</application> is a utility for restoring
<productname>PostgreSQL</productname> databases from an archive
created by <xref linkend="app-pgdump"/> or <xref
linkend="app-pgdumpall"/> in one of the non-plain-text
formats.
similarly, pg_dumpall first 3 sentences in the description section
needs to change.
in pg_restore.sgml <option>--create</option section,
maybe we can explicitly mention that restoring multiple databases,
<option>--create</option> is required.
like: "This option is required when restoring multiple databases."
restoreAllDatabases
+ if (!conn)
+ pg_log_info("there is no database connection so consider pattern as
simple name for --exclude-database");
filter_dbnames_for_restore
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database
option as no db connection while doing pg_restore.");
these two log messages sent out the same information.
maybe we can remove the first one, and change the second to
if (!conn && db_exclude_patterns.head != NULL)
pg_log_info("considering PATTERN as NAME for
--exclude-database option as no db connection while doing
pg_restore.");
as mentioned in the previous thread, there is no need to change PrintTOCSummary.
another minor issue about comments.
I guess we can tolerate this minor issue.
$BIN10/pg_restore --format=tar --create --file=1.sql
--exclude-database=src10 --verbose tar10 > dir_format 2>&1
1.sql file will copy tar10/global.dat as is. but we already excluded
src10. but 1.sql will still have comments as
--
-- Database "src10" dump
--
$BIN10/pg_dumpall --format=custom --file=x1.dump --globals-only
$BIN10/pg_dumpall --format=custom --file=x2.dump
Currently x1.dump/global.dat is differ from x2.dump/global.dat
if we dump multiple databases using pg_dumpall we have
"
--
-- Databases
--
--
-- Database "template1" dump
--
--
-- Database "src10" dump
--
--
-- Database "x" dump
--
"
maybe there are not need, since we already have map.dat file
I am not sure if the following is as expected or not.
$BIN10/pg_dumpall --format=custom --file=x1.dump --globals-only
$BIN10/pg_restore --create --file=3.sql --globals-only x1.dump --verbose
$BIN10/pg_restore --create --file=3.sql x1.dump --verbose
the first pg_restore command will copy x1.dump/global.dat as is to 3.sql,
the second pg_restore will not copy anything to 3.sql.
but the second command implies copying global dumps to 3.sql?
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 05:40 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-11 15:09 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 17:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-12 07:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-13 11:25 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-18 06:10 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-02-18 08:31 ` jian he <[email protected]>
1 sibling, 0 replies; 36+ messages in thread
From: jian he @ 2025-02-18 08:31 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
On Tue, Feb 18, 2025 at 2:10 PM jian he <[email protected]> wrote:
>
> hi.
hi. more cosmetic minor issues.
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath,
SimpleDatabaseOidList *dbname_oid_list)
...
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
i think the above comment in get_dbname_oid_list_from_mfile is not necessary.
we already have comments in filter_dbnames_for_restore.
in get_dbname_oid_list_from_mfile:
```
pfile = fopen(map_file_path, PG_BINARY_R);
if (pfile == NULL)
pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
```
file does not exist, we use pg_fatal, so if the directory does not
exist, we should also use pg_fatal.
so
if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
{
pg_log_info("databases restoring is skipped as map.dat file is
not present in \"%s\"", dumpdirpath);
return 0;
}
can be
if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
pg_fatal("map.dat file: \"%s\"/map.dat does not exists", dumpdirpath);
+ /* Report error if file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
i think the comments should be
+ /* Report error and exit if the file has any corrupted data. */
+/*
+ * filter_dbnames_for_restore
+ *
+ * This will remove names from all dblist those can
+ * be constructed from database_exclude_pattern list.
+ *
+ * returns number of dbnames those will be restored.
+ */
+static int
+filter_dbnames_for_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
there is no "database_exclude_pattern" list, so the above comments are
slightly wrong.
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.sql file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
here, "global sql" should change to "gloal.dat".
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
does this mean pg_dumpall --no-sync option only works for plain format.
if so, we need to update the pg_dumpall --no-sync section.
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 09:29 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 18:56 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-23 10:35 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 14:46 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-26 15:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Re: Non-text mode for pg_dumpall Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 03:52 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-31 08:51 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-02 20:19 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-03 09:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-03 18:04 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 05:40 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-11 15:09 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-11 17:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-12 07:14 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-02-13 11:25 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-02-18 06:10 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-02-19 12:37 ` Mahendra Singh Thalor <[email protected]>
1 sibling, 0 replies; 36+ messages in thread
From: Mahendra Singh Thalor @ 2025-02-19 12:37 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Srinath Reddy <[email protected]>; [email protected]
On Tue, 18 Feb 2025 at 10:00, Srinath Reddy <[email protected]> wrote:
>
> Hi,
> i think during restore we should not force user to use -C during cases like
> ./pg_restore pdd -g -f -
> ./pg_restore pdd -a -f -
> ./pg_restore pdd -s -f -
> because its not good to use -C to create database every time when we are using these options individually.
> latest patch throws following error for all the above cases
Fixed. (./pg_restore pdd -g -f -)
Thanks Jian and Srinath for the review and testing.
On Tue, 18 Feb 2025 at 11:41, jian he <[email protected]> wrote:
>
> hi.
>
> <refnamediv>
> <refname>pg_restore</refname>
> <refpurpose>
> restore a <productname>PostgreSQL</productname> database from an
> archive file created by <application>pg_dump</application>
> or restore multiple <productname>PostgreSQL</productname> database from an
> archive directory created by <application>pg_dumpall</application>
> </refpurpose>
> </refnamediv>
>
> i think it's way too verbose. we can change it to:
> <refpurpose>
> restore <productname>PostgreSQL</productname> database from an
> archive file created by <application>pg_dump</application> or
> <application>pg_dumpall</application>
> </refpurpose>
Fixed.
>
>
> <para>
> <application>pg_restore</application> is a utility for restoring a
> <productname>PostgreSQL</productname> database from an archive
> created by <xref linkend="app-pgdump"/> in one of the non-plain-text
> formats.
> we can change it to
> <para>
> <application>pg_restore</application> is a utility for restoring
> <productname>PostgreSQL</productname> databases from an archive
> created by <xref linkend="app-pgdump"/> or <xref
> linkend="app-pgdumpall"/> in one of the non-plain-text
> formats.
Fixed.
>
>
> similarly, pg_dumpall first 3 sentences in the description section
> needs to change.
>
I think we can keep them for pg_dumpall.
>
> in pg_restore.sgml <option>--create</option section,
> maybe we can explicitly mention that restoring multiple databases,
> <option>--create</option> is required.
> like: "This option is required when restoring multiple databases."
Fixed.
>
>
> restoreAllDatabases
> + if (!conn)
> + pg_log_info("there is no database connection so consider pattern as
> simple name for --exclude-database");
> filter_dbnames_for_restore
> + if (!conn)
> + pg_log_info("considering PATTERN as NAME for --exclude-database
> option as no db connection while doing pg_restore.");
>
> these two log messages sent out the same information.
> maybe we can remove the first one, and change the second to
> if (!conn && db_exclude_patterns.head != NULL)
> pg_log_info("considering PATTERN as NAME for
> --exclude-database option as no db connection while doing
> pg_restore.");
Fixed.
>
>
> as mentioned in the previous thread, there is no need to change PrintTOCSummary.
Yes, I removed it.
>
>
> another minor issue about comments.
> I guess we can tolerate this minor issue.
> $BIN10/pg_restore --format=tar --create --file=1.sql
> --exclude-database=src10 --verbose tar10 > dir_format 2>&1
> 1.sql file will copy tar10/global.dat as is. but we already excluded
> src10. but 1.sql will still have comments as
> --
> -- Database "src10" dump
> --
Fixed.
>
> $BIN10/pg_dumpall --format=custom --file=x1.dump --globals-only
> $BIN10/pg_dumpall --format=custom --file=x2.dump
>
> Currently x1.dump/global.dat is differ from x2.dump/global.dat
> if we dump multiple databases using pg_dumpall we have
> "
> --
> -- Databases
> --
> --
> -- Database "template1" dump
> --
> --
> -- Database "src10" dump
> --
> --
> -- Database "x" dump
> --
> "
> maybe there are not need, since we already have map.dat file
Okay. Fixed.
>
>
> I am not sure if the following is as expected or not.
> $BIN10/pg_dumpall --format=custom --file=x1.dump --globals-only
> $BIN10/pg_restore --create --file=3.sql --globals-only x1.dump --verbose
> $BIN10/pg_restore --create --file=3.sql x1.dump --verbose
>
> the first pg_restore command will copy x1.dump/global.dat as is to 3.sql,
> the second pg_restore will not copy anything to 3.sql.
> but the second command implies copying global dumps to 3.sql?
We should copy global.dat. I fixed this in the v18 patch.
On Tue, 18 Feb 2025 at 14:02, jian he <[email protected]> wrote:
>
> On Tue, Feb 18, 2025 at 2:10 PM jian he <[email protected]> wrote:
> >
> > hi.
>
> hi. more cosmetic minor issues.
>
> +static int
> +get_dbname_oid_list_from_mfile(const char *dumpdirpath,
> SimpleDatabaseOidList *dbname_oid_list)
> ...
> + /*
> + * XXX : before adding dbname into list, we can verify that this db
> + * needs to skipped for restore or not but as of now, we are making
> + * a list of all the databases.
> + */
> i think the above comment in get_dbname_oid_list_from_mfile is not necessary.
> we already have comments in filter_dbnames_for_restore.
As of now, I am keeping this comment as this will be helpful while
implementing parallel pg_restore.
>
> in get_dbname_oid_list_from_mfile:
> ```
> pfile = fopen(map_file_path, PG_BINARY_R);
> if (pfile == NULL)
> pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
> ```
> file does not exist, we use pg_fatal, so if the directory does not
> exist, we should also use pg_fatal.
> so
> if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
> {
> pg_log_info("databases restoring is skipped as map.dat file is
> not present in \"%s\"", dumpdirpath);
> return 0;
> }
> can be
> if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
> pg_fatal("map.dat file: \"%s\"/map.dat does not exists", dumpdirpath);
No, we can't add FATAL here as in case of global-only dump, we will
not have a map.dat file.
>
>
> + /* Report error if file has any corrupted data. */
> + if (!OidIsValid(db_oid) || strlen(dbname) == 0)
> + pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
> i think the comments should be
> + /* Report error and exit if the file has any corrupted data. */
Fixed.
>
>
> +/*
> + * filter_dbnames_for_restore
> + *
> + * This will remove names from all dblist those can
> + * be constructed from database_exclude_pattern list.
> + *
> + * returns number of dbnames those will be restored.
> + */
> +static int
> +filter_dbnames_for_restore(PGconn *conn,
> + SimpleDatabaseOidList *dbname_oid_list,
> there is no "database_exclude_pattern" list, so the above comments are
> slightly wrong.
Fixed.
>
>
> +/*
> + * ReadOneStatement
> + *
> + * This will start reading from passed file pointer using fgetc and read till
> + * semicolon(sql statement terminator for global.sql file)
> + *
> + * EOF is returned if end-of-file input is seen; time to shut down.
> + */
> here, "global sql" should change to "gloal.dat".
Fixed.
>
>
> /* sync the resulting file, errors are not fatal */
> - if (dosync)
> + if (dosync && (archDumpFormat == archNull))
> (void) fsync_fname(filename, false);
> does this mean pg_dumpall --no-sync option only works for plain format.
> if so, we need to update the pg_dumpall --no-sync section.
As of now, we are using this option with plain format as we dump
server commands in different db file. We can test this more.
On Wed, 19 Feb 2025 at 17:08, jian he <[email protected]> wrote:
>
> hi.
>
> Currently, pg_retore says
> --exit-on-error
> Exit if an error is encountered while sending SQL commands to the
> database. The default is to continue and to display a count of errors
> at the end of the restoration.
> Do we need to apply this to restore executing global commands (create
> role, create tablespace)?
> If not then we need to put some words in pg_restoe --exit-on-error
> option saying that while restoring global objects --exit-on-error
> option is ignored.
I think this is the same for all pg_restore commands. Still if we want
to add some docs, we can put.
>
>
>
> IMHO, in pg_restore.sgml, we need words explicitly saying that
> when restoring multiple databases, all the specified options will
> apply to each individual database.
We can skip this extra info. I will try in the next version if we can
add something in doc.
>
> I tested the following options for restoring multiple databases. The
> results look good to me.
> --index=index
> --table=table
> --schema-only
> --transaction-size
> --no-comments
> some part of (--filter=filename)
> --exclude-schema=schema
Thank you for detailed testing.
>
> attach is a minor cosmetic change.
Okay.
Here, I am attaching an updated patch for review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v18_pg_dumpall-with-non-text_format-19th_feb.patch (77.4K, 2-v18_pg_dumpall-with-non-text_format-19th_feb.patch)
download | inline diff:
From 0b4864eedf20b0a323d88d5158bcbfa0d6ffda94 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Wed, 19 Feb 2025 17:47:27 +0530
Subject: [PATCH] pg_dumpall with directory|tar|custom format and restore it
by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat to restore all databases. If global.dat file is exist in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list (if exist)
TODO1: We need to think for --exclude-database=PATTERN for pg_restore.
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
if db connection,
if no db connection, then PATTERN=NAME matching only
TODO2: We need to make changes for exit_nicely as we are one entry for each database while
restoring. MAX_ON_EXIT_NICELY
TODO3: some more test cases for new added options.
TODO4: We can dump and restore databases in parallel mode.
This needs more study
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
---
doc/src/sgml/ref/pg_dumpall.sgml | 80 ++-
doc/src/sgml/ref/pg_restore.sgml | 41 +-
src/bin/pg_dump/Makefile | 8 +-
src/bin/pg_dump/common_dumpall_restore.c | 286 ++++++++
src/bin/pg_dump/common_dumpall_restore.h | 26 +
src/bin/pg_dump/meson.build | 2 +
src/bin/pg_dump/pg_backup.h | 4 +-
src/bin/pg_dump/pg_backup_archiver.c | 22 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 552 +++++++--------
src/bin/pg_dump/pg_restore.c | 827 ++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 9 +
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 1533 insertions(+), 333 deletions(-)
create mode 100644 src/bin/pg_dump/common_dumpall_restore.c
create mode 100644 src/bin/pg_dump/common_dumpall_restore.h
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 39d93c2c0e3..6e1975f5ff0 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster based on specified dump format </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -121,7 +121,83 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- </para>
+ Note: This option can be omitted only when <option>--format</option> is plain
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index b8b27e1719e..835b3315713 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> database from an
+ archive file created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -37,9 +38,10 @@ PostgreSQL documentation
<title>Description</title>
<para>
- <application>pg_restore</application> is a utility for restoring a
+ <application>pg_restore</application> is a utility for restoring
<productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
database to the state it was in at the time it was saved. The
archive files also allow <application>pg_restore</application> to
@@ -140,6 +142,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from dump of <application>pg_dumpall</application>.
</para>
<para>
@@ -166,6 +170,25 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +338,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..a4e557d62c7 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -47,11 +47,11 @@ all: pg_dump pg_restore pg_dumpall
pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dump.o common.o pg_dump_sort.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_restore: pg_restore.o common_dumpall_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_restore.o common_dumpall_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/common_dumpall_restore.c b/src/bin/pg_dump/common_dumpall_restore.c
new file mode 100644
index 00000000000..b162cf69412
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.c
@@ -0,0 +1,286 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.c
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * This is a common file for pg_dumpall and pg_restore.
+ * src/bin/pg_dump/common_dumpall_restore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+#define exit_nicely(code) exit(code)
+
+/*
+ * connectDatabase
+ *
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the global variable 'connstr' is set to a connection string
+ * containing the options used.
+ */
+PGconn *
+connectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ static char *password = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 6;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. Remember the options used, in the form of a
+ * connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If needed, then copy server version to outer function. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/*
+ * constructConnStr
+ *
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * executeQuery
+ *
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
diff --git a/src/bin/pg_dump/common_dumpall_restore.h b/src/bin/pg_dump/common_dumpall_restore.h
new file mode 100644
index 00000000000..a0dcdbe0807
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.h
+ * Common header file for pg_dumpall and pg_restore
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/common_dumpall_restore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef COMMON_DUMPALL_RESTORE_H
+#define COMMON_DUMPALL_RESTORE_H
+
+#include "pg_backup.h"
+
+/* TODO: increasing this to keep 100 db restoring by single pg_restore command. */
+#define MAX_ON_EXIT_NICELY 100
+extern PGconn *connectDatabase(const char *dbname, const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+#endif /* COMMON_DUMPALL_RESTORE_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..ddecac5cf09 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -49,6 +49,7 @@ bin_targets += pg_dump
pg_dumpall_sources = files(
'pg_dumpall.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
@@ -68,6 +69,7 @@ bin_targets += pg_dumpall
pg_restore_sources = files(
'pg_restore.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index f0f19bb0b29..65000e5a083 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -306,7 +306,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
@@ -319,7 +319,7 @@ extern Archive *CreateArchive(const char *FileSpec, const ArchiveFormat fmt,
DataDirSyncMethod sync_method);
/* The --list option */
-extern void PrintTOCSummary(Archive *AHX);
+extern void PrintTOCSummary(Archive *AHX, bool append_data);
extern RestoreOptions *NewRestoreOptions(void);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index b9d7ab98c3e..b8b07562069 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -82,7 +82,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -331,9 +331,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -450,7 +455,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1263,7 +1268,7 @@ ArchiveEntry(Archive *AHX, CatalogId catalogId, DumpId dumpId,
/* Public */
void
-PrintTOCSummary(Archive *AHX)
+PrintTOCSummary(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -1279,7 +1284,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, append_data);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1658,7 +1663,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1678,7 +1684,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..47589cca90f 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -13,6 +13,7 @@
*/
#include "postgres_fe.h"
+#include "common_dumpall_restore.h"
#ifdef WIN32
#include "parallel.h"
#endif
@@ -21,8 +22,6 @@
/* Globals exported by this file */
const char *progname = NULL;
-#define MAX_ON_EXIT_NICELY 20
-
static struct
{
on_exit_nicely_callback function;
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 30dfda8c3ff..2b28cb39905 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1148,7 +1148,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 64a60a26092..b75e4f56f31 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -24,14 +25,17 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "pg_backup.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
+#define exit_nicely(code) exit(code)
typedef struct
{
@@ -64,28 +68,25 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -107,7 +108,7 @@ static int no_subscriptions = 0;
static int no_toast_compression = 0;
static int no_unlogged_table_data = 0;
static int no_role_passwords = 0;
-static int server_version;
+static int server_version;
static int load_via_partition_root = 0;
static int on_conflict_do_nothing = 0;
@@ -121,8 +122,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -147,6 +146,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -188,6 +188,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -237,7 +239,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -265,7 +267,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -414,6 +418,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If non-plain format is specified then we must provide the
+ * file name to create one main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -460,6 +479,33 @@ main(int argc, char *argv[])
if (on_conflict_do_nothing)
appendPQExpBufferStr(pgdumpopts, " --on-conflict-do-nothing");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char toc_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -468,7 +514,8 @@ main(int argc, char *argv[])
if (pgdb)
{
conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
@@ -476,10 +523,12 @@ main(int argc, char *argv[])
else
{
conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ prompt_password, true,
+ progname, &connstr, &server_version);
if (!conn)
{
@@ -496,19 +545,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -608,7 +644,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -621,7 +657,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -632,12 +668,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster based on specified dump format.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1488,10 +1526,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1505,7 +1546,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1513,9 +1554,33 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_fatal("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1530,6 +1595,18 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is non-plain dump format, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1548,9 +1625,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1559,19 +1644,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char toc_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(toc_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ toc_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1581,7 +1677,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1590,17 +1687,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is non-plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1650,256 +1766,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
@@ -1995,3 +1861,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if(!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c602272d7db..8974ee886d5 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,27 +41,71 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/connect.h"
+#include "compress_io.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
+#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
+typedef struct SimpleDatabaseOidListCell
+{
+ struct SimpleDatabaseOidListCell *next;
+ Oid db_oid;
+ const char *db_name;
+} SimpleDatabaseOidListCell;
+
+typedef struct SimpleDatabaseOidList
+{
+ SimpleDatabaseOidListCell *head;
+ SimpleDatabaseOidListCell *tail;
+} SimpleDatabaseOidList;
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data,
+ const char *dbname);
+static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static void process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static void copy_global_file_to_out_file(const char *outfile, FILE *pfile);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleDatabaseOidList *dbname_oid_list);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname);
+static void simple_string_full_list_delete(SimpleStringList *list);
+static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
+static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list,
+ Oid db_oid, const char *dbname);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
static int disable_triggers = 0;
static int enable_row_security = 0;
@@ -77,11 +121,14 @@ main(int argc, char **argv)
static int strict_names = 0;
bool data_only = false;
bool schema_only = false;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -128,6 +175,7 @@ main(int argc, char **argv)
{"no-security-labels", no_argument, &no_security_labels, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -156,7 +204,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "aAcCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -183,11 +231,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -302,6 +353,10 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6:
+ /* list of databases patterns those needs to skip while restoring */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -329,6 +384,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -404,6 +466,106 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL && !IsFileExistsInDirectory(inputFileSpec, "toc.dat"))
+ {
+ /* If global.dat is exist, then process it. */
+ if (IsFileExistsInDirectory(pg_strdup(inputFileSpec), "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql commands. */
+ int exit_code = 0;
+
+ /*
+ * User is suggested to use single database dump for --list option.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring multiple databases by archive of pg_dumpall");
+
+ /*
+ * To restore multiple databases, -C (create database) option
+ * should be specified.
+ * Even there is single database in dump, report error because it
+ * might be possible that database hasn't created so better we
+ * report error.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("-C/--create option should be specified when restoring multiple databases by archive of pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("If db is already created and dump has single db dump, then use particular dump file.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Connect to database to execute global sql commands from
+ * global.dat file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = connectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ /*
+ * Open global.dat file and execute/append all the global sql
+ * commands.
+ */
+ process_global_sql_commands(conn, inputFileSpec, opts->filename);
+
+ if (conn)
+ PQfinish(conn);
+
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ exit_code = restoreAllDatabases(conn, inputFileSpec,
+ db_exclude_patterns,
+ opts,
+ numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_full_list_delete(&db_exclude_patterns);
+
+ return exit_code;
+ }
+ }
+
+ if (db_exclude_patterns.head != NULL)
+ pg_fatal("option --exclude-database can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ return restoreOneDatabase(inputFileSpec, opts, numWorkers, false, NULL);
+}
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ * dbname is the current to be restored database name.
+ */
+static int
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, const char *dbname)
+{
+ Archive *AH;
+ int exit_code;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -429,11 +591,11 @@ main(int argc, char **argv)
AH->numWorkers = numWorkers;
if (opts->tocSummary)
- PrintTOCSummary(AH);
+ PrintTOCSummary(AH, append_data);
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
/* done, print a summary of ignored errors */
@@ -451,7 +613,8 @@ main(int argc, char **argv)
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n"
+ "If archive is created by pg_dumpall, then restores multiple databases also. \n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -469,6 +632,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -481,6 +645,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -513,8 +678,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be combined\n"
+ "and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -619,3 +784,645 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * IsFileExistsInDirectory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+IsFileExistsInDirectory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.dat file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+ReadOneStatement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+ int m;
+
+ StringInfoData q;
+ initStringInfo(&q);
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != '\n' && c != ';')
+ {
+ appendStringInfoChar(inBuf, (char) c);
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != ';' && c != '\n')
+ appendStringInfoChar(inBuf, (char) c);
+ else
+ break;
+ }
+ }
+
+ if (c == '\'' || c == '"')
+ {
+ appendStringInfoChar(&q, (char) c);
+ m = c;
+
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(&q, (char) c);
+
+ if(c == m)
+ {
+ appendStringInfoString(inBuf, q.data);
+ resetStringInfo(&q);
+ break;
+ }
+ }
+ }
+
+ if (c == ';')
+ {
+ appendStringInfoChar(inBuf, (char) ';');
+ break;
+ }
+
+ if (c == '\n')
+ appendStringInfoChar(inBuf, (char) '\n');
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will remove entries from dbname_oid_list that pattern matching any
+ * in the db_exclude_patterns list. dbname_oid_list maybe inplace modified.
+ *
+ * returns, number of database will be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
+ SimpleDatabaseOidListCell *dboidprecell = NULL;
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ /* Return 0 if there is no database to restore. */
+ if (dboid_cell == NULL)
+ return 0;
+
+ query = createPQExpBuffer();
+
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ while (dboid_cell != NULL)
+ {
+ bool skip_db_restore = false;
+ SimpleDatabaseOidListCell *next = dboid_cell->next;
+
+ for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ {
+ /*
+ * the construct pattern matching query:
+ * SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE
+ * pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from the
+ * dbname_oid_list, which is initially extracted from the map.dat
+ * file located in the backup directory. that's why we need
+ * quote_literal_cstr.
+ *
+ * If no db connection, then consider PATTERN as NAME.
+ */
+ if (pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0)
+ skip_db_restore = true;
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, celldb->val, false,
+ false, NULL, quote_literal_cstr(dboid_cell->db_name),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ celldb->val);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database \"%s\" is matching with exclude pattern: \"%s\"", dboid_cell->db_name, celldb->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /* Increment count if database needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
+ simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ }
+ else
+ {
+ count_db++;
+ dboidprecell = dboid_cell;
+ }
+
+ /* Process next dbname from dbname list. */
+ dboid_cell = next;
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ /*
+ * If there is only global.dat file in dump, then return from here as there
+ * is no database to restore.
+ */
+ if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ {
+ pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid = InvalidOid;
+ char db_oid_str[MAXPGPATH + 1] = {'\0'};
+ char dbname[MAXPGPATH + 1] = {'\0'};
+
+ /* Extract dboid. */
+ sscanf(line, "%u" , &db_oid);
+ sscanf(line, "%s" , db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found database \"%s\" (OID: %u) in map.dat file while restoring.", dbname, db_oid);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
+ SimpleDatabaseOidListCell *dboid_cell;
+ int exit_code = 0;
+ int num_db_restore = 0;
+ int num_total_db;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /*
+ * If map.dat has no entry, return from here after processing
+ * global.dat file.
+ */
+ if (dbname_oid_list.head == NULL)
+ {
+ process_global_sql_commands(conn, dumpdirpath, opts->filename);
+ return 0;
+ }
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ }
+ }
+
+ /*
+ * processing pg_retsore --exclude-database=PATTERN/NAME if no connection.
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* TODO: MAX_ON_EXIT_NICELY is 100 now... max AH handle register on exit .*/
+ if (num_db_restore > MAX_ON_EXIT_NICELY)
+ {
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+ pg_fatal("cound not restore more than %d databases by single pg_restore, here total database:%d",
+ MAX_ON_EXIT_NICELY,
+ num_db_restore);
+ }
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL)
+ {
+ pg_log_info("no database needs to restore out of %d databases", num_total_db);
+ return 0;
+ }
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * XXX: TODO till now, we made a list of databases, those needs to be restored
+ * after skipping names of exclude-database. Now we can launch parallel
+ * workers to restore these databases.
+ */
+ dboid_cell = dbname_oid_list.head;
+
+ while(dboid_cell != NULL)
+ {
+ char subdirpath[MAXPGPATH];
+ int dbexit_code;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored into
+ * already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+
+ pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+
+ /* Restore single database and save exit_code. */
+ dbexit_code = restoreOneDatabase(subdirpath, opts, numWorkers,
+ true, dboid_cell->db_name);
+
+ /* Store exit_code to report it back. */
+ if (exit_code == 0 && dbexit_code != 0)
+ exit_code = dbexit_code;
+
+ dboid_cell = dboid_cell->next;
+ }
+
+ /* Log number of processed databases.*/
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+
+ return exit_code;
+}
+
+/*
+ * process_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ */
+static void
+process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Now open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ copy_global_file_to_out_file(outfile, pfile);
+ return;
+ }
+
+ /* Init sqlstatement to append commands. */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements. */
+ while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ {
+ pg_log_info("executing query: %s", sqlstatement.data);
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ fclose(pfile);
+}
+
+/*
+ * copy_global_file_to_out_file
+ *
+ * This will copy global.dat file into out file. If "-" is used as outfile,
+ * then print commands to the stdout.
+ */
+static void
+copy_global_file_to_out_file(const char *outfile, FILE *pfile)
+{
+ char out_file_path[MAXPGPATH];
+ FILE *OPF;
+ int c;
+
+ /* "-" is used for stdout. */
+ if (strcmp(outfile, "-") == 0)
+ OPF = stdout;
+ else
+ {
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ OPF = fopen(out_file_path, PG_BINARY_W);
+
+ if (OPF == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", outfile);
+ }
+ }
+
+ /* Now append global.dat into out file. */
+ while ((c = fgetc(pfile)) != EOF)
+ fputc(c, OPF);
+
+ fclose(pfile);
+
+ /* Close out file. */
+ if (strcmp(outfile, "-") != 0)
+ fclose(OPF);
+}
+
+/*
+ * simple_db_oid_list_append
+ *
+ * appends a node to the list in the end.
+ */
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname)
+{
+ SimpleDatabaseOidListCell *cell;
+
+ cell = pg_malloc_object(SimpleDatabaseOidListCell);
+
+ cell->next = NULL;
+ cell->db_oid = db_oid;
+ cell->db_name = pg_strdup(dbname);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * simple_db_oid_full_list_delete
+ *
+ * delete all cell from dbname and dboid list.
+ */
+static void
+simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
+{
+ SimpleDatabaseOidListCell *cell = list->head;
+ SimpleDatabaseOidListCell *nextcell = NULL;
+
+ while (cell)
+ {
+ nextcell = cell->next;
+ pfree (cell);
+ cell = nextcell;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_string_full_list_delete
+ *
+ * delete all cell from string list.
+ */
+static void
+simple_string_full_list_delete(SimpleStringList *list)
+{
+ SimpleStringListCell *cell = list->head;
+ SimpleStringListCell *cellnext = NULL;
+
+ while (cell)
+ {
+ cellnext = cell->next;
+ pfree(cell);
+ cell = cellnext;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_db_oid_list_delete
+ *
+ * delete cell from database and oid list.
+ */
+static void
+simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev)
+{
+ if (prev == NULL)
+ {
+ list->head = cell->next;
+ pfree(cell);
+ }
+ else
+ {
+ prev->next = cell->next;
+ pfree(cell);
+ }
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr
+ *
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 214240f1ae5..de41ec06d86
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -219,6 +219,11 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -226,4 +231,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 80aa50d55a4..02a01251b04 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2676,6 +2676,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.39.3
^ permalink raw reply [nested|flat] 36+ messages in thread
end of thread, other threads:[~2025-02-19 12:37 UTC | newest]
Thread overview: 36+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-01-20 16:01 Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-01-21 04:05 ` jian he <[email protected]>
2025-01-21 09:29 ` jian he <[email protected]>
2025-01-21 18:56 ` Mahendra Singh Thalor <[email protected]>
2025-01-23 09:28 ` jian he <[email protected]>
2025-01-23 10:35 ` Mahendra Singh Thalor <[email protected]>
2025-01-24 15:20 ` jian he <[email protected]>
2025-01-26 14:46 ` jian he <[email protected]>
2025-01-26 15:48 ` Mahendra Singh Thalor <[email protected]>
2025-01-28 04:49 ` Srinath Reddy <[email protected]>
2025-01-28 06:21 ` Srinath Reddy <[email protected]>
2025-01-28 06:27 ` Mahendra Singh Thalor <[email protected]>
2025-01-28 12:02 ` Mahendra Singh Thalor <[email protected]>
2025-01-29 09:38 ` jian he <[email protected]>
2025-01-31 03:52 ` jian he <[email protected]>
2025-01-31 08:51 ` jian he <[email protected]>
2025-02-01 16:06 ` Srinath Reddy <[email protected]>
2025-02-02 20:19 ` Mahendra Singh Thalor <[email protected]>
2025-02-03 08:53 ` Srinath Reddy <[email protected]>
2025-02-03 09:02 ` Srinath Reddy <[email protected]>
2025-02-03 09:14 ` jian he <[email protected]>
2025-02-03 12:49 ` jian he <[email protected]>
2025-02-03 18:04 ` Mahendra Singh Thalor <[email protected]>
2025-02-04 02:04 ` jian he <[email protected]>
2025-02-04 07:42 ` jian he <[email protected]>
2025-02-11 05:40 ` Mahendra Singh Thalor <[email protected]>
2025-02-11 15:09 ` jian he <[email protected]>
2025-02-11 17:17 ` Mahendra Singh Thalor <[email protected]>
2025-02-12 07:14 ` jian he <[email protected]>
2025-02-13 11:25 ` Mahendra Singh Thalor <[email protected]>
2025-02-18 04:30 ` Srinath Reddy <[email protected]>
2025-02-19 11:32 ` Mahendra Singh Thalor <[email protected]>
2025-02-19 11:37 ` jian he <[email protected]>
2025-02-18 06:10 ` jian he <[email protected]>
2025-02-18 08:31 ` jian he <[email protected]>
2025-02-19 12:37 ` Mahendra Singh Thalor <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox