public inbox for [email protected]
help / color / mirror / Atom feedRe: Non-text mode for pg_dumpall
65+ messages / 11 participants
[nested] [flat]
* Re: Non-text mode for pg_dumpall
@ 2025-04-04 20:11 Andrew Dunstan <[email protected]>
0 siblings, 4 replies; 65+ messages in thread
From: Andrew Dunstan @ 2025-04-04 20:11 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-04-04 Fr 5:12 AM, Mahendra Singh Thalor wrote:
> On Fri, 4 Apr 2025 at 13:52, Mahendra Singh Thalor<[email protected]> wrote:
>> On Fri, 4 Apr 2025 at 01:17, Andrew Dunstan<[email protected]> wrote:
>>>
>>> On 2025-04-01 Tu 1:59 AM, Mahendra Singh Thalor wrote:
>>>> On Mon, 31 Mar 2025 at 23:43, Álvaro Herrera<[email protected]> wrote:
>>>>> Hi
>>>>>
>>>>> FWIW I don't think the on_exit_nicely business is in final shape just
>>>>> yet. We're doing something super strange and novel about keeping track
>>>>> of an array index, so that we can modify it later. Or something like
>>>>> that, I think? That doesn't sound all that nice to me. Elsewhere it
>>>>> was suggested that we need some way to keep track of the list of things
>>>>> that need cleanup (a list of connections IIRC?) -- perhaps in a
>>>>> thread-local variable or a global or something -- and we install the
>>>>> cleanup function once, and that reads from the variable. The program
>>>>> can add things to the list, or remove them, at will; and we don't need
>>>>> to modify the cleanup function in any way.
>>>>>
>>>>> --
>>>>> Álvaro Herrera Breisgau, Deutschland —https://www.EnterpriseDB.com/
>>>> Thanks Álvaro for the feedback.
>>>>
>>>> I removed the old handling of on_exit_nicely_list from the last patch
>>>> set and added one simple function to just update the archive handle in
>>>> shutdown_info. (shutdown_info.AHX = AHX;)
>>>>
>>>> For first database, we will add entry into on_exit_nicely_list array
>>>> and for rest database, we will update only shutdown_info as we already
>>>> closed connection for previous database.With this fix, we will not
>>>> touch entry of on_exit_nicely_list for each database.
>>>>
>>>> Here, I am attaching updated patches.
>>>>
>>>
>>> OK, looks good. here's my latest. I'm currently working on tidying up
>>> docco and comments.
>>>
>>>
>>> cheers
>>>
>>>
>>> andrew
>>>
>>>
>>>
>>>
>>> --
>>> Andrew Dunstan
>>> EDB:https://www.enterprisedb.com
>> Thanks Andrew for the updated patches.
>>
>> Here, I am attaching a delta patch with some more TAP-test cases.
>>
> Here, I am attaching an updated delta patch which has some more TAP
> tests. Please include these tests also. This patch can be applied on
> v20250403_0004* patch.
>
Thanks. I have pushed these now with a few further small tweaks.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-04-10 17:13 Mahendra Singh Thalor <[email protected]>
parent: Andrew Dunstan <[email protected]>
3 siblings, 0 replies; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-04-10 17:13 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Sat, 5 Apr 2025 at 01:41, Andrew Dunstan <[email protected]> wrote:
>
>
> On 2025-04-04 Fr 5:12 AM, Mahendra Singh Thalor wrote:
>
> On Fri, 4 Apr 2025 at 13:52, Mahendra Singh Thalor <[email protected]> wrote:
>
> On Fri, 4 Apr 2025 at 01:17, Andrew Dunstan <[email protected]> wrote:
>
> On 2025-04-01 Tu 1:59 AM, Mahendra Singh Thalor wrote:
>
> On Mon, 31 Mar 2025 at 23:43, Álvaro Herrera <[email protected]> wrote:
>
> Hi
>
> FWIW I don't think the on_exit_nicely business is in final shape just
> yet. We're doing something super strange and novel about keeping track
> of an array index, so that we can modify it later. Or something like
> that, I think? That doesn't sound all that nice to me. Elsewhere it
> was suggested that we need some way to keep track of the list of things
> that need cleanup (a list of connections IIRC?) -- perhaps in a
> thread-local variable or a global or something -- and we install the
> cleanup function once, and that reads from the variable. The program
> can add things to the list, or remove them, at will; and we don't need
> to modify the cleanup function in any way.
>
> --
> Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
>
> Thanks Álvaro for the feedback.
>
> I removed the old handling of on_exit_nicely_list from the last patch
> set and added one simple function to just update the archive handle in
> shutdown_info. (shutdown_info.AHX = AHX;)
>
> For first database, we will add entry into on_exit_nicely_list array
> and for rest database, we will update only shutdown_info as we already
> closed connection for previous database.With this fix, we will not
> touch entry of on_exit_nicely_list for each database.
>
> Here, I am attaching updated patches.
>
>
> OK, looks good. here's my latest. I'm currently working on tidying up
> docco and comments.
>
>
> cheers
>
>
> andrew
>
>
>
>
> --
> Andrew Dunstan
> EDB: https://www.enterprisedb.com
>
> Thanks Andrew for the updated patches.
>
> Here, I am attaching a delta patch with some more TAP-test cases.
>
> Here, I am attaching an updated delta patch which has some more TAP
> tests. Please include these tests also. This patch can be applied on
> v20250403_0004* patch.
>
>
>
> Thanks. I have pushed these now with a few further small tweaks.
>
>
> cheers
>
>
> andrew
>
> --
> Andrew Dunstan
> EDB: https://www.enterprisedb.com
Thanks Andrew for committing this.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-04-10 18:38 Ranier Vilela <[email protected]>
parent: Andrew Dunstan <[email protected]>
3 siblings, 1 reply; 65+ messages in thread
From: Ranier Vilela @ 2025-04-10 18:38 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Hi.
Em sex., 4 de abr. de 2025 às 17:11, Andrew Dunstan <[email protected]>
escreveu:
>
> On 2025-04-04 Fr 5:12 AM, Mahendra Singh Thalor wrote:
>
> On Fri, 4 Apr 2025 at 13:52, Mahendra Singh Thalor <[email protected]> <[email protected]> wrote:
>
> On Fri, 4 Apr 2025 at 01:17, Andrew Dunstan <[email protected]> <[email protected]> wrote:
>
> On 2025-04-01 Tu 1:59 AM, Mahendra Singh Thalor wrote:
>
> On Mon, 31 Mar 2025 at 23:43, Álvaro Herrera <[email protected]> <[email protected]> wrote:
>
> Hi
>
> FWIW I don't think the on_exit_nicely business is in final shape just
> yet. We're doing something super strange and novel about keeping track
> of an array index, so that we can modify it later. Or something like
> that, I think? That doesn't sound all that nice to me. Elsewhere it
> was suggested that we need some way to keep track of the list of things
> that need cleanup (a list of connections IIRC?) -- perhaps in a
> thread-local variable or a global or something -- and we install the
> cleanup function once, and that reads from the variable. The program
> can add things to the list, or remove them, at will; and we don't need
> to modify the cleanup function in any way.
>
> --
> Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
>
> Thanks Álvaro for the feedback.
>
> I removed the old handling of on_exit_nicely_list from the last patch
> set and added one simple function to just update the archive handle in
> shutdown_info. (shutdown_info.AHX = AHX;)
>
> For first database, we will add entry into on_exit_nicely_list array
> and for rest database, we will update only shutdown_info as we already
> closed connection for previous database.With this fix, we will not
> touch entry of on_exit_nicely_list for each database.
>
> Here, I am attaching updated patches.
>
>
> OK, looks good. here's my latest. I'm currently working on tidying up
> docco and comments.
>
>
> cheers
>
>
> andrew
>
>
>
>
> --
> Andrew Dunstan
> EDB: https://www.enterprisedb.com
>
> Thanks Andrew for the updated patches.
>
> Here, I am attaching a delta patch with some more TAP-test cases.
>
>
> Here, I am attaching an updated delta patch which has some more TAP
> tests. Please include these tests also. This patch can be applied on
> v20250403_0004* patch.
>
>
>
>
> Thanks. I have pushed these now with a few further small tweaks.
>
Sorry if it is not the right place.
Coverity has another resource leak alert.
trivial patch attached.
best regards,
Ranier Vilela
>
Attachments:
[application/octet-stream] fix_resource_leak_pg_restore.patch (434B, 3-fix_resource_leak_pg_restore.patch)
download | inline diff:
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 06c28ab314..eb3109d719 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -905,6 +905,7 @@ read_one_statement(StringInfo inBuf, FILE *pfile)
if (c == '\n')
appendStringInfoChar(inBuf, (char) '\n');
}
+ destroyStringInfo(&q);
/* No input before EOF signal means time to quit. */
if (c == EOF && inBuf->len == 0)
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-04-10 18:58 Andrew Dunstan <[email protected]>
parent: Ranier Vilela <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Andrew Dunstan @ 2025-04-10 18:58 UTC (permalink / raw)
To: Ranier Vilela <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-04-10 Th 2:38 PM, Ranier Vilela wrote:
>
>
>
> Thanks. I have pushed these now with a few further small tweaks.
>
> Sorry if it is not the right place.
> Coverity has another resource leak alert.
>
> trivial patch attached.
>
>
Thanks for checking. Pushed.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-04-10 21:45 Ranier Vilela <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Ranier Vilela @ 2025-04-10 21:45 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Em qui., 10 de abr. de 2025 às 15:58, Andrew Dunstan <[email protected]>
escreveu:
>
> On 2025-04-10 Th 2:38 PM, Ranier Vilela wrote:
>
>
>>
>> Thanks. I have pushed these now with a few further small tweaks.
>>
> Sorry if it is not the right place.
> Coverity has another resource leak alert.
>
> trivial patch attached.
>
>
>
>
> Thanks for checking. Pushed.
>
Andew, I think that the commit wasn't very correct.
Now the variable *q* is being destroyed inside the loop.
The patch was destroying the variable *q* (stringinfo),
after the loop while.
best regards,
Ranier Vilela
>
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-04-15 18:30 Mahendra Singh Thalor <[email protected]>
parent: Andrew Dunstan <[email protected]>
3 siblings, 0 replies; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-04-15 18:30 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Sat, 5 Apr 2025 at 01:41, Andrew Dunstan <[email protected]> wrote:
>
>
> On 2025-04-04 Fr 5:12 AM, Mahendra Singh Thalor wrote:
>
> On Fri, 4 Apr 2025 at 13:52, Mahendra Singh Thalor <[email protected]> wrote:
>
> On Fri, 4 Apr 2025 at 01:17, Andrew Dunstan <[email protected]> wrote:
>
> On 2025-04-01 Tu 1:59 AM, Mahendra Singh Thalor wrote:
>
> On Mon, 31 Mar 2025 at 23:43, Álvaro Herrera <[email protected]> wrote:
>
> Hi
>
> FWIW I don't think the on_exit_nicely business is in final shape just
> yet. We're doing something super strange and novel about keeping track
> of an array index, so that we can modify it later. Or something like
> that, I think? That doesn't sound all that nice to me. Elsewhere it
> was suggested that we need some way to keep track of the list of things
> that need cleanup (a list of connections IIRC?) -- perhaps in a
> thread-local variable or a global or something -- and we install the
> cleanup function once, and that reads from the variable. The program
> can add things to the list, or remove them, at will; and we don't need
> to modify the cleanup function in any way.
>
> --
> Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
>
> Thanks Álvaro for the feedback.
>
> I removed the old handling of on_exit_nicely_list from the last patch
> set and added one simple function to just update the archive handle in
> shutdown_info. (shutdown_info.AHX = AHX;)
>
> For first database, we will add entry into on_exit_nicely_list array
> and for rest database, we will update only shutdown_info as we already
> closed connection for previous database.With this fix, we will not
> touch entry of on_exit_nicely_list for each database.
>
> Here, I am attaching updated patches.
>
>
> OK, looks good. here's my latest. I'm currently working on tidying up
> docco and comments.
>
>
> cheers
>
>
> andrew
>
>
>
>
> --
> Andrew Dunstan
> EDB: https://www.enterprisedb.com
>
> Thanks Andrew for the updated patches.
>
> Here, I am attaching a delta patch with some more TAP-test cases.
>
> Here, I am attaching an updated delta patch which has some more TAP
> tests. Please include these tests also. This patch can be applied on
> v20250403_0004* patch.
>
>
>
> Thanks. I have pushed these now with a few further small tweaks.
>
>
> cheers
>
>
> andrew
>
> --
> Andrew Dunstan
> EDB: https://www.enterprisedb.com
Hi Andrew,
I did some refactoring to find out dump file extensions(.dmp/.tar etc)
in pg_restore. With the attached patch, we will not try to find out
file extension with each database, rather we will find out before the
loop.
Here, I am attaching a patch for the same. Please have a look over this.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v01-pg_restore-refactor-code-of-dump-file-extenion.patch (4.6K, 2-v01-pg_restore-refactor-code-of-dump-file-extenion.patch)
download | inline diff:
From b4721f4c91017297cfbdeaa458291f7a97b023d7 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Tue, 15 Apr 2025 23:45:44 +0530
Subject: [PATCH] pg_restore: refactor code of dump file extenion
After this refactor, we will find file extension once but earlier we were trying
to get extension for each database in loop.
---
src/bin/pg_dump/pg_restore.c | 82 +++++++++++++++++++++++++++---------
1 file changed, 61 insertions(+), 21 deletions(-)
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index ff4bb320fc9..e6486957620 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -71,6 +71,8 @@ static int get_dbnames_list_to_restore(PGconn *conn,
SimpleStringList db_exclude_patterns);
static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
SimpleOidStringList *dbname_oid_list);
+static
+char *get_dump_file_exten(const char *dumpdirpath, Oid dboid, const ArchiveFormat format);
int
main(int argc, char **argv)
@@ -1109,6 +1111,7 @@ restore_all_databases(PGconn *conn, const char *dumpdirpath,
bool dumpData = opts->dumpData;
bool dumpSchema = opts->dumpSchema;
bool dumpStatistics = opts->dumpSchema;
+ const char *file_exten;
/* Save db name to reuse it for all the database. */
if (opts->cparams.dbname)
@@ -1163,6 +1166,9 @@ restore_all_databases(PGconn *conn, const char *dumpdirpath,
pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+ /* Now get dump file extention. */
+ file_exten = get_dump_file_exten(dumpdirpath, dbname_oid_list.head->oid, opts->format);
+
/*
* Till now, we made a list of databases, those needs to be restored after
* skipping names of exclude-database. Now we can launch parallel workers
@@ -1172,8 +1178,6 @@ restore_all_databases(PGconn *conn, const char *dumpdirpath,
db_cell; db_cell = db_cell->next)
{
char subdirpath[MAXPGPATH];
- char subdirdbpath[MAXPGPATH];
- char dbfilename[MAXPGPATH];
int n_errors;
/* ignore dbs marked for skipping */
@@ -1190,25 +1194,8 @@ restore_all_databases(PGconn *conn, const char *dumpdirpath,
opts->cparams.override_dbname = NULL;
}
- snprintf(subdirdbpath, MAXPGPATH, "%s/databases", dumpdirpath);
-
- /*
- * Look for the database dump file/dir. If there is an {oid}.tar or
- * {oid}.dmp file, use it. Otherwise try to use a directory called
- * {oid}
- */
- snprintf(dbfilename, MAXPGPATH, "%u.tar", db_cell->oid);
- if (file_exists_in_directory(subdirdbpath, dbfilename))
- snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", dumpdirpath, db_cell->oid);
- else
- {
- snprintf(dbfilename, MAXPGPATH, "%u.dmp", db_cell->oid);
-
- if (file_exists_in_directory(subdirdbpath, dbfilename))
- snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", dumpdirpath, db_cell->oid);
- else
- snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, db_cell->oid);
- }
+ /* Set particular dump file path. */
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u%s", dumpdirpath, db_cell->oid, file_exten);
pg_log_info("restoring database \"%s\"", db_cell->str);
@@ -1384,3 +1371,56 @@ copy_or_print_global_file(const char *outfile, FILE *pfile)
if (strcmp(outfile, "-") != 0)
fclose(OPF);
}
+
+/*
+ * get_dump_file_exten
+ *
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+static
+char *get_dump_file_exten(const char *dumpdirpath, Oid dboid, const ArchiveFormat format)
+{
+ char *file_exten = "";
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", dumpdirpath);
+
+ switch(format)
+ {
+ case archCustom:
+ file_exten = ".dmp";
+ break;
+ case archTar:
+ file_exten = ".tar";
+ break;
+ case archDirectory:
+ file_exten = "";
+ break;
+ case archUnknown: /* based on file exist, try to get file extension name. */
+ snprintf(dbfilename, MAXPGPATH, "%u", dboid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ file_exten = "";
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dboid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ file_exten = ".dmp";
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dboid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ file_exten = ".tar";
+ else
+ file_exten = "";
+ }
+ }
+ break;
+ default:
+ pg_fatal("unrecognized file format \"%d\"", format);
+ }
+
+ return file_exten;
+}
--
2.39.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-08 21:28 Noah Misch <[email protected]>
parent: Andrew Dunstan <[email protected]>
3 siblings, 1 reply; 65+ messages in thread
From: Noah Misch @ 2025-07-08 21:28 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Fri, Apr 04, 2025 at 04:11:05PM -0400, Andrew Dunstan wrote:
> Thanks. I have pushed these now with a few further small tweaks.
This drops all databases:
pg_dumpall --clean -Fd -f /tmp/dump
pg_restore -d template1 --globals-only /tmp/dump
That didn't match my expectations given this help text:
$ pg_restore --help|grep global
-g, --globals-only restore only global objects, no databases
This happens in dropDBs(). I found that by searching pg_dumpall.c for "OPF",
which finds all the content we can write to globals.dat.
commit 1495eff wrote:
> --- a/src/bin/pg_dump/pg_dumpall.c
> +++ b/src/bin/pg_dump/pg_dumpall.c
> @@ -1612,9 +1683,27 @@ dumpDatabases(PGconn *conn)
> continue;
> }
>
> + /*
> + * If this is not a plain format dump, then append dboid and dbname to
> + * the map.dat file.
> + */
> + if (archDumpFormat != archNull)
> + {
> + if (archDumpFormat == archCustom)
> + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
> + else if (archDumpFormat == archTar)
> + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
> + else
> + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
Use appendShellString() instead. Plain mode already does that for the
"pg_dumpall -f" argument, which is part of db_subdir here. We don't want
weird filename characters to work out differently for plain vs. non-plain
mode. Also, it's easier to search for appendShellString() than to search for
open-coded shell quoting.
> @@ -1641,19 +1727,30 @@ dumpDatabases(PGconn *conn)
> if (filename)
> fclose(OPF);
>
> - ret = runPgDump(dbname, create_opts);
> + ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
> if (ret != 0)
> pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
>
> if (filename)
> {
> - OPF = fopen(filename, PG_BINARY_A);
> + char global_path[MAXPGPATH];
> +
> + if (archDumpFormat != archNull)
> + snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
> + else
> + snprintf(global_path, MAXPGPATH, "%s", filename);
> +
> + OPF = fopen(global_path, PG_BINARY_A);
> if (!OPF)
> pg_fatal("could not re-open the output file \"%s\": %m",
> - filename);
> + global_path);
Minor item: plain mode benefits from reopening, because pg_dump appended to
the plain output file. There's no analogous need to reopen global.dat, since
just this one process writes to global.dat.
> @@ -1672,17 +1770,36 @@ runPgDump(const char *dbname, const char *create_opts)
> initPQExpBuffer(&connstrbuf);
> initPQExpBuffer(&cmd);
>
> - printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
> - pgdumpopts->data, create_opts);
> -
> /*
> - * If we have a filename, use the undocumented plain-append pg_dump
> - * format.
> + * If this is not a plain format dump, then append file name and dump
> + * format to the pg_dump command to get archive dump.
> */
> - if (filename)
> - appendPQExpBufferStr(&cmd, " -Fa ");
> + if (archDumpFormat != archNull)
> + {
> + printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
> + dbfile, create_opts);
> +
> + if (archDumpFormat == archDirectory)
> + appendPQExpBufferStr(&cmd, " --format=directory ");
> + else if (archDumpFormat == archCustom)
> + appendPQExpBufferStr(&cmd, " --format=custom ");
> + else if (archDumpFormat == archTar)
> + appendPQExpBufferStr(&cmd, " --format=tar ");
> + }
> else
> - appendPQExpBufferStr(&cmd, " -Fp ");
> + {
> + printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
> + pgdumpopts->data, create_opts);
This uses pgdumpopts for plain mode only, so many pg_dumpall options silently
have no effect in non-plain mode. Example:
strace -f pg_dumpall --lock-wait-timeout=10 2>&1 >/dev/null | grep exec
strace -f pg_dumpall --lock-wait-timeout=10 -Fd -f /tmp/dump3 2>&1 >/dev/null | grep exec
> --- a/src/bin/pg_dump/pg_restore.c
> +++ b/src/bin/pg_dump/pg_restore.c
> +/*
> + * read_one_statement
> + *
> + * This will start reading from passed file pointer using fgetc and read till
> + * semicolon(sql statement terminator for global.dat file)
> + *
> + * EOF is returned if end-of-file input is seen; time to shut down.
What makes it okay to use this particular subset of SQL lexing?
> +/*
> + * get_dbnames_list_to_restore
> + *
> + * This will mark for skipping any entries from dbname_oid_list that pattern match an
> + * entry in the db_exclude_patterns list.
> + *
> + * Returns the number of database to be restored.
> + *
> + */
> +static int
> +get_dbnames_list_to_restore(PGconn *conn,
> + SimpleOidStringList *dbname_oid_list,
> + SimpleStringList db_exclude_patterns)
> +{
> + int count_db = 0;
> + PQExpBuffer query;
> + PGresult *res;
> +
> + query = createPQExpBuffer();
> +
> + if (!conn)
> + pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
When do we not have a connection here? We'd need to document this behavior
variation if it stays, but I'd prefer if we can just rely on having a
connection.
> + /* If database is already created, then don't set createDB flag. */
> + if (opts->cparams.dbname)
> + {
> + PGconn *test_conn;
> +
> + test_conn = ConnectDatabase(db_cell->str, NULL, opts->cparams.pghost,
> + opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
> + false, progname, NULL, NULL, NULL, NULL);
> + if (test_conn)
> + {
> + PQfinish(test_conn);
> +
> + /* Use already created database for connection. */
> + opts->createDB = 0;
> + opts->cparams.dbname = db_cell->str;
> + }
> + else
> + {
> + /* we'll have to create it */
> + opts->createDB = 1;
> + opts->cparams.dbname = connected_db;
> + }
In released versions, "pg_restore --create" fails if the database exists, and
pg_restore w/o --create fails unless the database exists. I think we should
continue that pattern in this new feature. If not, pg_restore should document
how it treats pg_dumpall-sourced dumps with the "create if not exists"
semantics appearing here.
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-09 18:51 Mahendra Singh Thalor <[email protected]>
parent: Noah Misch <[email protected]>
0 siblings, 2 replies; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-07-09 18:51 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Thanks Noah for the comments.
On Wed, 9 Jul 2025 at 02:58, Noah Misch <[email protected]> wrote:
>
> On Fri, Apr 04, 2025 at 04:11:05PM -0400, Andrew Dunstan wrote:
> > Thanks. I have pushed these now with a few further small tweaks.
>
> This drops all databases:
>
> pg_dumpall --clean -Fd -f /tmp/dump
> pg_restore -d template1 --globals-only /tmp/dump
>
> That didn't match my expectations given this help text:
>
> $ pg_restore --help|grep global
> -g, --globals-only restore only global objects, no databases
Databases are global objects so due to --clean command, we are putting
drop commands in global.dat for all the databases. While restoring, we
used the "--globals-only" option so we are dropping all these
databases by global.dat file.
Please let us know your expectations for this specific case.
>
> This happens in dropDBs(). I found that by searching pg_dumpall.c for "OPF",
> which finds all the content we can write to globals.dat.
>
> commit 1495eff wrote:
> > --- a/src/bin/pg_dump/pg_dumpall.c
> > +++ b/src/bin/pg_dump/pg_dumpall.c
>
> > @@ -1612,9 +1683,27 @@ dumpDatabases(PGconn *conn)
> > continue;
> > }
> >
> > + /*
> > + * If this is not a plain format dump, then append dboid and dbname to
> > + * the map.dat file.
> > + */
> > + if (archDumpFormat != archNull)
> > + {
> > + if (archDumpFormat == archCustom)
> > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
> > + else if (archDumpFormat == archTar)
> > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
> > + else
> > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
>
> Use appendShellString() instead. Plain mode already does that for the
> "pg_dumpall -f" argument, which is part of db_subdir here. We don't want
> weird filename characters to work out differently for plain vs. non-plain
> mode. Also, it's easier to search for appendShellString() than to search for
> open-coded shell quoting.
Yes, we can use appendShellString also. We are using snprintf in the
pg_dump.c file also.
Ex: snprintf(tagbuf, sizeof(tagbuf), "LARGE OBJECTS %u..%u",
loinfo->looids[0], loinfo->looids[loinfo->numlos - 1]);
If we want to use appendShellString, I can write a patch for these.
Please let me know your opinion.
>
> > @@ -1641,19 +1727,30 @@ dumpDatabases(PGconn *conn)
> > if (filename)
> > fclose(OPF);
> >
> > - ret = runPgDump(dbname, create_opts);
> > + ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
> > if (ret != 0)
> > pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
> >
> > if (filename)
> > {
> > - OPF = fopen(filename, PG_BINARY_A);
> > + char global_path[MAXPGPATH];
> > +
> > + if (archDumpFormat != archNull)
> > + snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
> > + else
> > + snprintf(global_path, MAXPGPATH, "%s", filename);
> > +
> > + OPF = fopen(global_path, PG_BINARY_A);
> > if (!OPF)
> > pg_fatal("could not re-open the output file \"%s\": %m",
> > - filename);
> > + global_path);
>
> Minor item: plain mode benefits from reopening, because pg_dump appended to
> the plain output file. There's no analogous need to reopen global.dat, since
> just this one process writes to global.dat.
yes, only once we need to open global.dat file but to keep simple
code, we kept old code.
>
> > @@ -1672,17 +1770,36 @@ runPgDump(const char *dbname, const char *create_opts)
> > initPQExpBuffer(&connstrbuf);
> > initPQExpBuffer(&cmd);
> >
> > - printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
> > - pgdumpopts->data, create_opts);
> > -
> > /*
> > - * If we have a filename, use the undocumented plain-append pg_dump
> > - * format.
> > + * If this is not a plain format dump, then append file name and dump
> > + * format to the pg_dump command to get archive dump.
> > */
> > - if (filename)
> > - appendPQExpBufferStr(&cmd, " -Fa ");
> > + if (archDumpFormat != archNull)
> > + {
> > + printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
> > + dbfile, create_opts);
> > +
> > + if (archDumpFormat == archDirectory)
> > + appendPQExpBufferStr(&cmd, " --format=directory ");
> > + else if (archDumpFormat == archCustom)
> > + appendPQExpBufferStr(&cmd, " --format=custom ");
> > + else if (archDumpFormat == archTar)
> > + appendPQExpBufferStr(&cmd, " --format=tar ");
> > + }
> > else
> > - appendPQExpBufferStr(&cmd, " -Fp ");
> > + {
> > + printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
> > + pgdumpopts->data, create_opts);
>
> This uses pgdumpopts for plain mode only, so many pg_dumpall options silently
> have no effect in non-plain mode. Example:
>
> strace -f pg_dumpall --lock-wait-timeout=10 2>&1 >/dev/null | grep exec
> strace -f pg_dumpall --lock-wait-timeout=10 -Fd -f /tmp/dump3 2>&1 >/dev/null | grep exec
Agreed. We can add pgdumpopts->data to all the dump formats.
>
> > --- a/src/bin/pg_dump/pg_restore.c
> > +++ b/src/bin/pg_dump/pg_restore.c
>
> > +/*
> > + * read_one_statement
> > + *
> > + * This will start reading from passed file pointer using fgetc and read till
> > + * semicolon(sql statement terminator for global.dat file)
> > + *
> > + * EOF is returned if end-of-file input is seen; time to shut down.
>
> What makes it okay to use this particular subset of SQL lexing?
To support complex syntax, we used this code from another file.
>
> > +/*
> > + * get_dbnames_list_to_restore
> > + *
> > + * This will mark for skipping any entries from dbname_oid_list that pattern match an
> > + * entry in the db_exclude_patterns list.
> > + *
> > + * Returns the number of database to be restored.
> > + *
> > + */
> > +static int
> > +get_dbnames_list_to_restore(PGconn *conn,
> > + SimpleOidStringList *dbname_oid_list,
> > + SimpleStringList db_exclude_patterns)
> > +{
> > + int count_db = 0;
> > + PQExpBuffer query;
> > + PGresult *res;
> > +
> > + query = createPQExpBuffer();
> > +
> > + if (!conn)
> > + pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
>
> When do we not have a connection here? We'd need to document this behavior
> variation if it stays, but I'd prefer if we can just rely on having a
> connection.
Yes, we can document this behavior.
>
> > + /* If database is already created, then don't set createDB flag. */
> > + if (opts->cparams.dbname)
> > + {
> > + PGconn *test_conn;
> > +
> > + test_conn = ConnectDatabase(db_cell->str, NULL, opts->cparams.pghost,
> > + opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
> > + false, progname, NULL, NULL, NULL, NULL);
> > + if (test_conn)
> > + {
> > + PQfinish(test_conn);
> > +
> > + /* Use already created database for connection. */
> > + opts->createDB = 0;
> > + opts->cparams.dbname = db_cell->str;
> > + }
> > + else
> > + {
> > + /* we'll have to create it */
> > + opts->createDB = 1;
> > + opts->cparams.dbname = connected_db;
> > + }
>
> In released versions, "pg_restore --create" fails if the database exists, and
> pg_restore w/o --create fails unless the database exists. I think we should
> continue that pattern in this new feature. If not, pg_restore should document
> how it treats pg_dumpall-sourced dumps with the "create if not exists"
> semantics appearing here.
Yes, we can document this behavior also.
I am working on all these review comments and I will post a patch in
the coming days.
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-16 00:19 Noah Misch <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
1 sibling, 1 reply; 65+ messages in thread
From: Noah Misch @ 2025-07-16 00:19 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Thu, Jul 10, 2025 at 12:21:03AM +0530, Mahendra Singh Thalor wrote:
> On Wed, 9 Jul 2025 at 02:58, Noah Misch <[email protected]> wrote:
> > On Fri, Apr 04, 2025 at 04:11:05PM -0400, Andrew Dunstan wrote:
> > > Thanks. I have pushed these now with a few further small tweaks.
> >
> > This drops all databases:
> >
> > pg_dumpall --clean -Fd -f /tmp/dump
> > pg_restore -d template1 --globals-only /tmp/dump
> >
> > That didn't match my expectations given this help text:
> >
> > $ pg_restore --help|grep global
> > -g, --globals-only restore only global objects, no databases
>
> Databases are global objects so due to --clean command, we are putting
> drop commands in global.dat for all the databases. While restoring, we
> used the "--globals-only" option so we are dropping all these
> databases by global.dat file.
>
> Please let us know your expectations for this specific case.
Be consistent with "pg_dump". A quick check suggests "pg_dump --clean"
affects plain format only. For non-plain formats, only the pg_restore
argument governs the final commands:
$ rm -r /tmp/dump; pg_dump --clean -Fd -f /tmp/dump && pg_restore -f- /tmp/dump | grep DROP
$ rm -r /tmp/dump; pg_dump --clean -Fd -f /tmp/dump && pg_restore --clean -f- /tmp/dump | grep DROP
DROP TABLE public.example;
That said, you should audit code referencing the --clean flag and see if
there's more to it than that quick test suggests. Note that aligning with
pg_dump will require changes for object types beyond databases. "pg_restore
--clean" of a global dump should emit DROP TABLESPACE and DROP ROLE as
appropriate, regardless of whether the original pg_dumpall had --clean.
For my earlier example (pg_dumpall --clean; pg_restore --globals-only) I
expect the same outcome as plain-format "pg_dumpall --globals-only", which is
no databases dropped or created. The help line says "no databases". Plain
"pg_dumpall --globals-only" and even "pg_dumpall --globals-only --clean" do
not drop or create databases.
> > commit 1495eff wrote:
> > > --- a/src/bin/pg_dump/pg_dumpall.c
> > > +++ b/src/bin/pg_dump/pg_dumpall.c
> >
> > > @@ -1612,9 +1683,27 @@ dumpDatabases(PGconn *conn)
> > > continue;
> > > }
> > >
> > > + /*
> > > + * If this is not a plain format dump, then append dboid and dbname to
> > > + * the map.dat file.
> > > + */
> > > + if (archDumpFormat != archNull)
> > > + {
> > > + if (archDumpFormat == archCustom)
> > > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
> > > + else if (archDumpFormat == archTar)
> > > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
> > > + else
> > > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
> >
> > Use appendShellString() instead. Plain mode already does that for the
> > "pg_dumpall -f" argument, which is part of db_subdir here. We don't want
> > weird filename characters to work out differently for plain vs. non-plain
> > mode. Also, it's easier to search for appendShellString() than to search for
> > open-coded shell quoting.
>
> Yes, we can use appendShellString also. We are using snprintf in the
> pg_dump.c file also.
> Ex: snprintf(tagbuf, sizeof(tagbuf), "LARGE OBJECTS %u..%u",
> loinfo->looids[0], loinfo->looids[loinfo->numlos - 1]);
It's true snprintf() is not banned in these programs, but don't use it to do
the quoting for OS shell command lines or fragments thereof. dbfilepath is a
fragment of an OS shell command line. The LARGE OBJECTS string is not one of
those. Hence, the LARGE OBJECTS scenario should keep using snprintf().
> If we want to use appendShellString, I can write a patch for these.
> Please let me know your opinion.
Use appendShellString() for shell quoting. Don't attempt to use it for other
purposes.
> > > --- a/src/bin/pg_dump/pg_restore.c
> > > +++ b/src/bin/pg_dump/pg_restore.c
> >
> > > +/*
> > > + * read_one_statement
> > > + *
> > > + * This will start reading from passed file pointer using fgetc and read till
> > > + * semicolon(sql statement terminator for global.dat file)
> > > + *
> > > + * EOF is returned if end-of-file input is seen; time to shut down.
> >
> > What makes it okay to use this particular subset of SQL lexing?
>
> To support complex syntax, we used this code from another file.
I'm hearing that you copied this code from somewhere. Running
"git grep 'time to shut down'" suggests you copied it from
InteractiveBackend(). Is that right? I do see other similarities between
read_one_statement() and InteractiveBackend().
Copying InteractiveBackend() provides negligible assurance that this is the
right subset of SQL lexing. Only single-user mode uses InteractiveBackend().
Single-user mode survives mostly as a last resort for recovering from having
reached xidStopLimit, is rarely used, and only superusers write queries to it.
> > > +/*
> > > + * get_dbnames_list_to_restore
> > > + *
> > > + * This will mark for skipping any entries from dbname_oid_list that pattern match an
> > > + * entry in the db_exclude_patterns list.
> > > + *
> > > + * Returns the number of database to be restored.
> > > + *
> > > + */
> > > +static int
> > > +get_dbnames_list_to_restore(PGconn *conn,
> > > + SimpleOidStringList *dbname_oid_list,
> > > + SimpleStringList db_exclude_patterns)
> > > +{
> > > + int count_db = 0;
> > > + PQExpBuffer query;
> > > + PGresult *res;
> > > +
> > > + query = createPQExpBuffer();
> > > +
> > > + if (!conn)
> > > + pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
> >
> > When do we not have a connection here? We'd need to document this behavior
> > variation if it stays, but I'd prefer if we can just rely on having a
> > connection.
>
> Yes, we can document this behavior.
My review asked a question there. I don't see an answer to that question.
Would you answer that question?
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-17 10:16 Mahendra Singh Thalor <[email protected]>
parent: Noah Misch <[email protected]>
0 siblings, 3 replies; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-07-17 10:16 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Thanks Noah for the feedback.
On Wed, 16 Jul 2025 at 05:50, Noah Misch <[email protected]> wrote:
>
> On Thu, Jul 10, 2025 at 12:21:03AM +0530, Mahendra Singh Thalor wrote:
> > On Wed, 9 Jul 2025 at 02:58, Noah Misch <[email protected]> wrote:
> > > On Fri, Apr 04, 2025 at 04:11:05PM -0400, Andrew Dunstan wrote:
> > > > Thanks. I have pushed these now with a few further small tweaks.
> > >
> > > This drops all databases:
> > >
> > > pg_dumpall --clean -Fd -f /tmp/dump
> > > pg_restore -d template1 --globals-only /tmp/dump
> > >
> > > That didn't match my expectations given this help text:
> > >
> > > $ pg_restore --help|grep global
> > > -g, --globals-only restore only global objects, no databases
> >
> > Databases are global objects so due to --clean command, we are putting
> > drop commands in global.dat for all the databases. While restoring, we
> > used the "--globals-only" option so we are dropping all these
> > databases by global.dat file.
> >
> > Please let us know your expectations for this specific case.
>
> Be consistent with "pg_dump". A quick check suggests "pg_dump --clean"
> affects plain format only. For non-plain formats, only the pg_restore
> argument governs the final commands:
>
> $ rm -r /tmp/dump; pg_dump --clean -Fd -f /tmp/dump && pg_restore -f- /tmp/dump | grep DROP
> $ rm -r /tmp/dump; pg_dump --clean -Fd -f /tmp/dump && pg_restore --clean -f- /tmp/dump | grep DROP
> DROP TABLE public.example;
>
> That said, you should audit code referencing the --clean flag and see if
> there's more to it than that quick test suggests. Note that aligning with
> pg_dump will require changes for object types beyond databases. "pg_restore
> --clean" of a global dump should emit DROP TABLESPACE and DROP ROLE as
> appropriate, regardless of whether the original pg_dumpall had --clean.
>
> For my earlier example (pg_dumpall --clean; pg_restore --globals-only) I
> expect the same outcome as plain-format "pg_dumpall --globals-only", which is
> no databases dropped or created. The help line says "no databases". Plain
> "pg_dumpall --globals-only" and even "pg_dumpall --globals-only --clean" do
> not drop or create databases.
To pg_restore, we are giving a dump of pg_dumpall which has a
global.dat file and we have drop commands in the global.dat file so
when we are using 'globals-only', we are dropping databases as we have
DROP commands.
As of now, we don't have any filter for global.dat file in restore. If
a user wants to restore only globals(without droping db), then they
should use 'globals-only' in pg_dumpall.
Or if we don't want to DROP databases by global.dat file, then we
should add a filter in pg_restore (hard to implement as we have SQL
commands in global.dat file). I think, for this case, we can do some
more doc changes.
Example: pg_restore --globals-only : this will restore the global.dat
file(including all drop commands). It might drop databases if any drop
commands.
@Andrew Dunstan Please add your opinion.
>
> > > commit 1495eff wrote:
> > > > --- a/src/bin/pg_dump/pg_dumpall.c
> > > > +++ b/src/bin/pg_dump/pg_dumpall.c
> > >
> > > > @@ -1612,9 +1683,27 @@ dumpDatabases(PGconn *conn)
> > > > continue;
> > > > }
> > > >
> > > > + /*
> > > > + * If this is not a plain format dump, then append dboid and dbname to
> > > > + * the map.dat file.
> > > > + */
> > > > + if (archDumpFormat != archNull)
> > > > + {
> > > > + if (archDumpFormat == archCustom)
> > > > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
> > > > + else if (archDumpFormat == archTar)
> > > > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
> > > > + else
> > > > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
> > >
> > > Use appendShellString() instead. Plain mode already does that for the
> > > "pg_dumpall -f" argument, which is part of db_subdir here. We don't want
> > > weird filename characters to work out differently for plain vs. non-plain
> > > mode. Also, it's easier to search for appendShellString() than to search for
> > > open-coded shell quoting.
> >
> > Yes, we can use appendShellString also. We are using snprintf in the
> > pg_dump.c file also.
> > Ex: snprintf(tagbuf, sizeof(tagbuf), "LARGE OBJECTS %u..%u",
> > loinfo->looids[0], loinfo->looids[loinfo->numlos - 1]);
>
> It's true snprintf() is not banned in these programs, but don't use it to do
> the quoting for OS shell command lines or fragments thereof. dbfilepath is a
> fragment of an OS shell command line. The LARGE OBJECTS string is not one of
> those. Hence, the LARGE OBJECTS scenario should keep using snprintf().
>
> > If we want to use appendShellString, I can write a patch for these.
> > Please let me know your opinion.
>
> Use appendShellString() for shell quoting. Don't attempt to use it for other
> purposes.
Okay. Fixed in attached patch.
>
> > > > --- a/src/bin/pg_dump/pg_restore.c
> > > > +++ b/src/bin/pg_dump/pg_restore.c
> > >
> > > > +/*
> > > > + * read_one_statement
> > > > + *
> > > > + * This will start reading from passed file pointer using fgetc and read till
> > > > + * semicolon(sql statement terminator for global.dat file)
> > > > + *
> > > > + * EOF is returned if end-of-file input is seen; time to shut down.
> > >
> > > What makes it okay to use this particular subset of SQL lexing?
> >
> > To support complex syntax, we used this code from another file.
>
> I'm hearing that you copied this code from somewhere. Running
> "git grep 'time to shut down'" suggests you copied it from
> InteractiveBackend(). Is that right? I do see other similarities between
> read_one_statement() and InteractiveBackend().
>
> Copying InteractiveBackend() provides negligible assurance that this is the
> right subset of SQL lexing. Only single-user mode uses InteractiveBackend().
> Single-user mode survives mostly as a last resort for recovering from having
> reached xidStopLimit, is rarely used, and only superusers write queries to it.
Yes, we copied this from InteractiveBackend to read statements from
global.dat file.
>
> > > > +/*
> > > > + * get_dbnames_list_to_restore
> > > > + *
> > > > + * This will mark for skipping any entries from dbname_oid_list that pattern match an
> > > > + * entry in the db_exclude_patterns list.
> > > > + *
> > > > + * Returns the number of database to be restored.
> > > > + *
> > > > + */
> > > > +static int
> > > > +get_dbnames_list_to_restore(PGconn *conn,
> > > > + SimpleOidStringList *dbname_oid_list,
> > > > + SimpleStringList db_exclude_patterns)
> > > > +{
> > > > + int count_db = 0;
> > > > + PQExpBuffer query;
> > > > + PGresult *res;
> > > > +
> > > > + query = createPQExpBuffer();
> > > > +
> > > > + if (!conn)
> > > > + pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
> > >
> > > When do we not have a connection here? We'd need to document this behavior
> > > variation if it stays, but I'd prefer if we can just rely on having a
> > > connection.
> >
> > Yes, we can document this behavior.
>
> My review asked a question there. I don't see an answer to that question.
> Would you answer that question?
Example: if there is no active database, even postgres/template1, then
we will consider PATTEREN as NAME. This is the rare case.
In attached patch, I added one doc line also for this case.
> > @@ -1612,9 +1683,27 @@ dumpDatabases(PGconn *conn)
> > continue;
> > }
> >
> > + /*
> > + * If this is not a plain format dump, then append dboid and dbname to
> > + * the map.dat file.
> > + */
> > + if (archDumpFormat != archNull)
> > + {
> > + if (archDumpFormat == archCustom)
> > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
> > + else if (archDumpFormat == archTar)
> > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
> > + else
> > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
>
> Use appendShellString() instead. Plain mode already does that for the
> "pg_dumpall -f" argument, which is part of db_subdir here. We don't want
> weird filename characters to work out differently for plain vs. non-plain
> mode. Also, it's easier to search for appendShellString() than to search for
> open-coded shell quoting.
Fixed.
>
> > @@ -1641,19 +1727,30 @@ dumpDatabases(PGconn *conn)
> > if (filename)
> > fclose(OPF);
> >
> > - ret = runPgDump(dbname, create_opts);
> > + ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
> > if (ret != 0)
> > pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
> >
> > if (filename)
> > {
> > - OPF = fopen(filename, PG_BINARY_A);
> > + char global_path[MAXPGPATH];
> > +
> > + if (archDumpFormat != archNull)
> > + snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
> > + else
> > + snprintf(global_path, MAXPGPATH, "%s", filename);
> > +
> > + OPF = fopen(global_path, PG_BINARY_A);
> > if (!OPF)
> > pg_fatal("could not re-open the output file \"%s\": %m",
> > - filename);
> > + global_path);
>
> Minor item: plain mode benefits from reopening, because pg_dump appended to
> the plain output file. There's no analogous need to reopen global.dat, since
> just this one process writes to global.dat.
Fixed.
>
> > @@ -1672,17 +1770,36 @@ runPgDump(const char *dbname, const char *create_opts)
> > initPQExpBuffer(&connstrbuf);
> > initPQExpBuffer(&cmd);
> >
> > - printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
> > - pgdumpopts->data, create_opts);
> > -
> > /*
> > - * If we have a filename, use the undocumented plain-append pg_dump
> > - * format.
> > + * If this is not a plain format dump, then append file name and dump
> > + * format to the pg_dump command to get archive dump.
> > */
> > - if (filename)
> > - appendPQExpBufferStr(&cmd, " -Fa ");
> > + if (archDumpFormat != archNull)
> > + {
> > + printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
> > + dbfile, create_opts);
> > +
> > + if (archDumpFormat == archDirectory)
> > + appendPQExpBufferStr(&cmd, " --format=directory ");
> > + else if (archDumpFormat == archCustom)
> > + appendPQExpBufferStr(&cmd, " --format=custom ");
> > + else if (archDumpFormat == archTar)
> > + appendPQExpBufferStr(&cmd, " --format=tar ");
> > + }
> > else
> > - appendPQExpBufferStr(&cmd, " -Fp ");
> > + {
> > + printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
> > + pgdumpopts->data, create_opts);
>
> This uses pgdumpopts for plain mode only, so many pg_dumpall options silently
> have no effect in non-plain mode. Example:
>
> strace -f pg_dumpall --lock-wait-timeout=10 2>&1 >/dev/null | grep exec
> strace -f pg_dumpall --lock-wait-timeout=10 -Fd -f /tmp/dump3 2>&1 >/dev/null | grep exec
Fixed.
> > + /* If database is already created, then don't set createDB flag. */
> > + if (opts->cparams.dbname)
> > + {
> > + PGconn *test_conn;
> > +
> > + test_conn = ConnectDatabase(db_cell->str, NULL, opts->cparams.pghost,
> > + opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
> > + false, progname, NULL, NULL, NULL, NULL);
> > + if (test_conn)
> > + {
> > + PQfinish(test_conn);
> > +
> > + /* Use already created database for connection. */
> > + opts->createDB = 0;
> > + opts->cparams.dbname = db_cell->str;
> > + }
> > + else
> > + {
> > + /* we'll have to create it */
> > + opts->createDB = 1;
> > + opts->cparams.dbname = connected_db;
> > + }
>
> In released versions, "pg_restore --create" fails if the database exists, and
> pg_restore w/o --create fails unless the database exists. I think we should
> continue that pattern in this new feature. If not, pg_restore should document
> how it treats pg_dumpall-sourced dumps with the "create if not exists"
> semantics appearing here.
Added one more doc line for this case.
Here, I am attaching a patch. Please let me know feedback.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v01_db2978-WIP-implement-setNodeValue-function.patch (5.2K, 2-v01_db2978-WIP-implement-setNodeValue-function.patch)
download | inline diff:
From ff9b28250dc0c016b37477536ece0b14b6d07c03 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Wed, 9 Jul 2025 11:42:47 +0530
Subject: [PATCH] implement setNodeValue function
DB-2978
---
contrib/dbms_xmldom/dbms_xmldom.c | 58 ++++++++++++++++++-
contrib/dbms_xmldom/dbms_xmldom.sql.in | 13 +++++
contrib/dbms_xmldom/dbms_xmldom_public.sql.in | 1 +
3 files changed, 71 insertions(+), 1 deletion(-)
diff --git a/contrib/dbms_xmldom/dbms_xmldom.c b/contrib/dbms_xmldom/dbms_xmldom.c
index ed49245b0cb..a297ddca6d3 100644
--- a/contrib/dbms_xmldom/dbms_xmldom.c
+++ b/contrib/dbms_xmldom/dbms_xmldom.c
@@ -42,6 +42,7 @@ PG_FUNCTION_INFO_V1(dbms_xmldom_free_document);
PG_FUNCTION_INFO_V1(dbms_xmldom_set_version);
PG_FUNCTION_INFO_V1(dbms_xmldom_get_nodename);
PG_FUNCTION_INFO_V1(dbms_xmldom_get_nodevalue);
+PG_FUNCTION_INFO_V1(dbms_xmldom_set_nodevalue);
PG_FUNCTION_INFO_V1(dbms_xmldom_get_firstchild);
PG_FUNCTION_INFO_V1(dbms_xmldom_get_childnodes);
PG_FUNCTION_INFO_V1(dbms_xmldom_get_nodelistlength);
@@ -49,7 +50,6 @@ PG_FUNCTION_INFO_V1(dbms_xmldom_get_nodelistitem);
PG_FUNCTION_INFO_V1(dbms_xmldom_make_element);
PG_FUNCTION_INFO_V1(dbms_xmldom_replace_child);
PG_FUNCTION_INFO_V1(dbms_xmldom_remove_child);
-PG_FUNCTION_INFO_V1(dbms_xmldom_set_node_value);
/* Function Declarations */
@@ -1287,6 +1287,62 @@ dbms_xmldom_get_nodevalue(PG_FUNCTION_ARGS)
PG_RETURN_VARCHAR_P(cstring_to_text((char *) nodevalue));
}
+/*
+ * dbms_xmldom_set_nodevalue
+ *
+ * Implements the functionality of dbms_xmldom.setNodeValue.
+ * sets the value to DOMNode
+ */
+Datum
+dbms_xmldom_set_nodevalue(PG_FUNCTION_ARGS)
+{
+ char docHashKey[HASHKEYLEN];
+ uint32 docid = 0;
+ uint64 nodeid = 0;
+ DocInfoPtr docInfo = NULL;
+ DocNodeInfoPtr nodeInfo = NULL;
+ char *nodevalue = NULL;
+ char *nodevaluenull = NULL;
+
+ /* If node is NULL, then return. */
+ if (PG_ARGISNULL(0))
+ PG_RETURN_NULL();
+
+ if (PG_ARGISNULL(1) || strlen(text_to_cstring(PG_GETARG_TEXT_P(1))) == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid value for parameter \"%s\"", "tagName")));
+
+ nodevalue = text_to_cstring(PG_GETARG_TEXT_P(1));
+
+ memcpy(docHashKey, VARDATA_ANY(PG_GETARG_TEXT_P(0)), HASHKEYLEN);
+ extract_ids(docHashKey, &docid, &nodeid);
+
+ docInfo = getDocInfo(docid);
+ if (!docInfo)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ err_redwood_sqlcode(-31181),
+ errmsg("invalid value for parameter \"%s\"", "n")));
+
+ nodeInfo =
+ (DocNodeInfoPtr) hash_search(docInfo->nodeInfo,
+ &nodeid,
+ HASH_FIND,
+ NULL);
+
+ Assert(nodeInfo && nodeInfo->node);
+
+ /* Now set new vaule. */
+// xmlNodeSetContent((xmlNodePtr) nodeInfo->node, (const xmlChar *) nodevaluenull);
+// xmlNodeAddContent((xmlNodePtr) nodeInfo->node, (const xmlChar *) nodevalue);
+
+ xmlNodeSetName((xmlNodePtr) nodeInfo->node, (const xmlChar *) nodevalue);
+// xmlNodeSetContent((xmlNodePtr) nodeInfo->node, (const xmlChar *) nodevalue);
+
+ PG_RETURN_TEXT_P(cstring_to_text_with_len(docHashKey, HASHKEYLEN));
+}
+
/*
* dbms_xmldom_get_firstchild
*
diff --git a/contrib/dbms_xmldom/dbms_xmldom.sql.in b/contrib/dbms_xmldom/dbms_xmldom.sql.in
index 2236612d754..2eae94ec656 100644
--- a/contrib/dbms_xmldom/dbms_xmldom.sql.in
+++ b/contrib/dbms_xmldom/dbms_xmldom.sql.in
@@ -65,6 +65,10 @@ CREATE FUNCTION dbms_xmldom_getNodeValue(nodeid RAW) RETURNS VARCHAR2
AS '$libdir/dbms_xmldom', 'dbms_xmldom_get_nodevalue'
LANGUAGE C IMMUTABLE PARALLEL SAFE;
+CREATE FUNCTION dbms_xmldom_setNodeValue(nodeid RAW, name IN VARCHAR2) RETURNS RAW
+AS '$libdir/dbms_xmldom', 'dbms_xmldom_set_nodevalue'
+LANGUAGE C IMMUTABLE PARALLEL SAFE;
+
CREATE FUNCTION dbms_xmldom_getFirstChild(nodeid RAW) RETURNS RAW
AS '$libdir/dbms_xmldom', 'dbms_xmldom_get_firstchild'
LANGUAGE C IMMUTABLE PARALLEL SAFE;
@@ -145,6 +149,14 @@ CREATE OR REPLACE PACKAGE BODY dbms_xmldom IS
return dbms_xmldom_getNodeValue(n.id);
END;
+ FUNCTION setNodeValue(n domnode, name IN VARCHAR2) RETURN DOMNode SET search_path = pg_catalog, pg_temp IS
+ DECLARE
+ node DOMNode;
+ BEGIN
+ n.id = dbms_xmldom_setNodeValue(n.id, name);
+ return node;
+ END;
+
FUNCTION getFirstChild(n DOMNode) RETURN DOMNode SET search_path = pg_catalog, pg_temp IS
DECLARE
node DOMNode;
@@ -199,6 +211,7 @@ CREATE OR REPLACE PACKAGE BODY dbms_xmldom IS
return node;
END;
+
FUNCTION createElement(doc DOMDocument, tagName IN VARCHAR2) RETURN DOMElement SET search_path = pg_catalog, pg_temp IS
DECLARE
node DOMElement;
diff --git a/contrib/dbms_xmldom/dbms_xmldom_public.sql.in b/contrib/dbms_xmldom/dbms_xmldom_public.sql.in
index b7b1686e1fe..8fd2463394d 100644
--- a/contrib/dbms_xmldom/dbms_xmldom_public.sql.in
+++ b/contrib/dbms_xmldom/dbms_xmldom_public.sql.in
@@ -32,6 +32,7 @@ CREATE OR REPLACE PACKAGE dbms_xmldom AUTHID CURRENT_USER AS
FUNCTION getNodeName(n DOMNode) RETURN VARCHAR2;
FUNCTION getNodeValue(n domnode) RETURN VARCHAR2;
+ FUNCTION setNodeValue(n domnode, name IN VARCHAR2) RETURN DOMNode;
FUNCTION getFirstChild(n DOMNode) RETURN DOMNode;
FUNCTION getChildNodes(n DOMNode) RETURN DOMNodeList;
FUNCTION appendChild(n DOMNode, newChild IN DOMNode) RETURN DOMNode;
--
2.39.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-17 10:18 Mahendra Singh Thalor <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
2 siblings, 1 reply; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-07-17 10:18 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Attaching the correct patch.
Sorry, I attached the wrong patch in my last email.
On Thu, 17 Jul 2025 at 15:46, Mahendra Singh Thalor <[email protected]> wrote:
>
> Thanks Noah for the feedback.
>
> On Wed, 16 Jul 2025 at 05:50, Noah Misch <[email protected]> wrote:
> >
> > On Thu, Jul 10, 2025 at 12:21:03AM +0530, Mahendra Singh Thalor wrote:
> > > On Wed, 9 Jul 2025 at 02:58, Noah Misch <[email protected]> wrote:
> > > > On Fri, Apr 04, 2025 at 04:11:05PM -0400, Andrew Dunstan wrote:
> > > > > Thanks. I have pushed these now with a few further small tweaks.
> > > >
> > > > This drops all databases:
> > > >
> > > > pg_dumpall --clean -Fd -f /tmp/dump
> > > > pg_restore -d template1 --globals-only /tmp/dump
> > > >
> > > > That didn't match my expectations given this help text:
> > > >
> > > > $ pg_restore --help|grep global
> > > > -g, --globals-only restore only global objects, no databases
> > >
> > > Databases are global objects so due to --clean command, we are putting
> > > drop commands in global.dat for all the databases. While restoring, we
> > > used the "--globals-only" option so we are dropping all these
> > > databases by global.dat file.
> > >
> > > Please let us know your expectations for this specific case.
> >
> > Be consistent with "pg_dump". A quick check suggests "pg_dump --clean"
> > affects plain format only. For non-plain formats, only the pg_restore
> > argument governs the final commands:
> >
> > $ rm -r /tmp/dump; pg_dump --clean -Fd -f /tmp/dump && pg_restore -f- /tmp/dump | grep DROP
> > $ rm -r /tmp/dump; pg_dump --clean -Fd -f /tmp/dump && pg_restore --clean -f- /tmp/dump | grep DROP
> > DROP TABLE public.example;
> >
> > That said, you should audit code referencing the --clean flag and see if
> > there's more to it than that quick test suggests. Note that aligning with
> > pg_dump will require changes for object types beyond databases. "pg_restore
> > --clean" of a global dump should emit DROP TABLESPACE and DROP ROLE as
> > appropriate, regardless of whether the original pg_dumpall had --clean.
> >
> > For my earlier example (pg_dumpall --clean; pg_restore --globals-only) I
> > expect the same outcome as plain-format "pg_dumpall --globals-only", which is
> > no databases dropped or created. The help line says "no databases". Plain
> > "pg_dumpall --globals-only" and even "pg_dumpall --globals-only --clean" do
> > not drop or create databases.
>
> To pg_restore, we are giving a dump of pg_dumpall which has a
> global.dat file and we have drop commands in the global.dat file so
> when we are using 'globals-only', we are dropping databases as we have
> DROP commands.
> As of now, we don't have any filter for global.dat file in restore. If
> a user wants to restore only globals(without droping db), then they
> should use 'globals-only' in pg_dumpall.
> Or if we don't want to DROP databases by global.dat file, then we
> should add a filter in pg_restore (hard to implement as we have SQL
> commands in global.dat file). I think, for this case, we can do some
> more doc changes.
> Example: pg_restore --globals-only : this will restore the global.dat
> file(including all drop commands). It might drop databases if any drop
> commands.
> @Andrew Dunstan Please add your opinion.
>
> >
> > > > commit 1495eff wrote:
> > > > > --- a/src/bin/pg_dump/pg_dumpall.c
> > > > > +++ b/src/bin/pg_dump/pg_dumpall.c
> > > >
> > > > > @@ -1612,9 +1683,27 @@ dumpDatabases(PGconn *conn)
> > > > > continue;
> > > > > }
> > > > >
> > > > > + /*
> > > > > + * If this is not a plain format dump, then append dboid and dbname to
> > > > > + * the map.dat file.
> > > > > + */
> > > > > + if (archDumpFormat != archNull)
> > > > > + {
> > > > > + if (archDumpFormat == archCustom)
> > > > > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
> > > > > + else if (archDumpFormat == archTar)
> > > > > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
> > > > > + else
> > > > > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
> > > >
> > > > Use appendShellString() instead. Plain mode already does that for the
> > > > "pg_dumpall -f" argument, which is part of db_subdir here. We don't want
> > > > weird filename characters to work out differently for plain vs. non-plain
> > > > mode. Also, it's easier to search for appendShellString() than to search for
> > > > open-coded shell quoting.
> > >
> > > Yes, we can use appendShellString also. We are using snprintf in the
> > > pg_dump.c file also.
> > > Ex: snprintf(tagbuf, sizeof(tagbuf), "LARGE OBJECTS %u..%u",
> > > loinfo->looids[0], loinfo->looids[loinfo->numlos - 1]);
> >
> > It's true snprintf() is not banned in these programs, but don't use it to do
> > the quoting for OS shell command lines or fragments thereof. dbfilepath is a
> > fragment of an OS shell command line. The LARGE OBJECTS string is not one of
> > those. Hence, the LARGE OBJECTS scenario should keep using snprintf().
> >
> > > If we want to use appendShellString, I can write a patch for these.
> > > Please let me know your opinion.
> >
> > Use appendShellString() for shell quoting. Don't attempt to use it for other
> > purposes.
>
> Okay. Fixed in attached patch.
>
> >
> > > > > --- a/src/bin/pg_dump/pg_restore.c
> > > > > +++ b/src/bin/pg_dump/pg_restore.c
> > > >
> > > > > +/*
> > > > > + * read_one_statement
> > > > > + *
> > > > > + * This will start reading from passed file pointer using fgetc and read till
> > > > > + * semicolon(sql statement terminator for global.dat file)
> > > > > + *
> > > > > + * EOF is returned if end-of-file input is seen; time to shut down.
> > > >
> > > > What makes it okay to use this particular subset of SQL lexing?
> > >
> > > To support complex syntax, we used this code from another file.
> >
> > I'm hearing that you copied this code from somewhere. Running
> > "git grep 'time to shut down'" suggests you copied it from
> > InteractiveBackend(). Is that right? I do see other similarities between
> > read_one_statement() and InteractiveBackend().
> >
> > Copying InteractiveBackend() provides negligible assurance that this is the
> > right subset of SQL lexing. Only single-user mode uses InteractiveBackend().
> > Single-user mode survives mostly as a last resort for recovering from having
> > reached xidStopLimit, is rarely used, and only superusers write queries to it.
>
> Yes, we copied this from InteractiveBackend to read statements from
> global.dat file.
>
> >
> > > > > +/*
> > > > > + * get_dbnames_list_to_restore
> > > > > + *
> > > > > + * This will mark for skipping any entries from dbname_oid_list that pattern match an
> > > > > + * entry in the db_exclude_patterns list.
> > > > > + *
> > > > > + * Returns the number of database to be restored.
> > > > > + *
> > > > > + */
> > > > > +static int
> > > > > +get_dbnames_list_to_restore(PGconn *conn,
> > > > > + SimpleOidStringList *dbname_oid_list,
> > > > > + SimpleStringList db_exclude_patterns)
> > > > > +{
> > > > > + int count_db = 0;
> > > > > + PQExpBuffer query;
> > > > > + PGresult *res;
> > > > > +
> > > > > + query = createPQExpBuffer();
> > > > > +
> > > > > + if (!conn)
> > > > > + pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
> > > >
> > > > When do we not have a connection here? We'd need to document this behavior
> > > > variation if it stays, but I'd prefer if we can just rely on having a
> > > > connection.
> > >
> > > Yes, we can document this behavior.
> >
> > My review asked a question there. I don't see an answer to that question.
> > Would you answer that question?
>
> Example: if there is no active database, even postgres/template1, then
> we will consider PATTEREN as NAME. This is the rare case.
> In attached patch, I added one doc line also for this case.
>
> > > @@ -1612,9 +1683,27 @@ dumpDatabases(PGconn *conn)
> > > continue;
> > > }
> > >
> > > + /*
> > > + * If this is not a plain format dump, then append dboid and dbname to
> > > + * the map.dat file.
> > > + */
> > > + if (archDumpFormat != archNull)
> > > + {
> > > + if (archDumpFormat == archCustom)
> > > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
> > > + else if (archDumpFormat == archTar)
> > > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
> > > + else
> > > + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
> >
> > Use appendShellString() instead. Plain mode already does that for the
> > "pg_dumpall -f" argument, which is part of db_subdir here. We don't want
> > weird filename characters to work out differently for plain vs. non-plain
> > mode. Also, it's easier to search for appendShellString() than to search for
> > open-coded shell quoting.
>
> Fixed.
>
> >
> > > @@ -1641,19 +1727,30 @@ dumpDatabases(PGconn *conn)
> > > if (filename)
> > > fclose(OPF);
> > >
> > > - ret = runPgDump(dbname, create_opts);
> > > + ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
> > > if (ret != 0)
> > > pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
> > >
> > > if (filename)
> > > {
> > > - OPF = fopen(filename, PG_BINARY_A);
> > > + char global_path[MAXPGPATH];
> > > +
> > > + if (archDumpFormat != archNull)
> > > + snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
> > > + else
> > > + snprintf(global_path, MAXPGPATH, "%s", filename);
> > > +
> > > + OPF = fopen(global_path, PG_BINARY_A);
> > > if (!OPF)
> > > pg_fatal("could not re-open the output file \"%s\": %m",
> > > - filename);
> > > + global_path);
> >
> > Minor item: plain mode benefits from reopening, because pg_dump appended to
> > the plain output file. There's no analogous need to reopen global.dat, since
> > just this one process writes to global.dat.
>
> Fixed.
>
> >
> > > @@ -1672,17 +1770,36 @@ runPgDump(const char *dbname, const char *create_opts)
> > > initPQExpBuffer(&connstrbuf);
> > > initPQExpBuffer(&cmd);
> > >
> > > - printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
> > > - pgdumpopts->data, create_opts);
> > > -
> > > /*
> > > - * If we have a filename, use the undocumented plain-append pg_dump
> > > - * format.
> > > + * If this is not a plain format dump, then append file name and dump
> > > + * format to the pg_dump command to get archive dump.
> > > */
> > > - if (filename)
> > > - appendPQExpBufferStr(&cmd, " -Fa ");
> > > + if (archDumpFormat != archNull)
> > > + {
> > > + printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
> > > + dbfile, create_opts);
> > > +
> > > + if (archDumpFormat == archDirectory)
> > > + appendPQExpBufferStr(&cmd, " --format=directory ");
> > > + else if (archDumpFormat == archCustom)
> > > + appendPQExpBufferStr(&cmd, " --format=custom ");
> > > + else if (archDumpFormat == archTar)
> > > + appendPQExpBufferStr(&cmd, " --format=tar ");
> > > + }
> > > else
> > > - appendPQExpBufferStr(&cmd, " -Fp ");
> > > + {
> > > + printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
> > > + pgdumpopts->data, create_opts);
> >
> > This uses pgdumpopts for plain mode only, so many pg_dumpall options silently
> > have no effect in non-plain mode. Example:
> >
> > strace -f pg_dumpall --lock-wait-timeout=10 2>&1 >/dev/null | grep exec
> > strace -f pg_dumpall --lock-wait-timeout=10 -Fd -f /tmp/dump3 2>&1 >/dev/null | grep exec
>
> Fixed.
>
> > > + /* If database is already created, then don't set createDB flag. */
> > > + if (opts->cparams.dbname)
> > > + {
> > > + PGconn *test_conn;
> > > +
> > > + test_conn = ConnectDatabase(db_cell->str, NULL, opts->cparams.pghost,
> > > + opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
> > > + false, progname, NULL, NULL, NULL, NULL);
> > > + if (test_conn)
> > > + {
> > > + PQfinish(test_conn);
> > > +
> > > + /* Use already created database for connection. */
> > > + opts->createDB = 0;
> > > + opts->cparams.dbname = db_cell->str;
> > > + }
> > > + else
> > > + {
> > > + /* we'll have to create it */
> > > + opts->createDB = 1;
> > > + opts->cparams.dbname = connected_db;
> > > + }
> >
> > In released versions, "pg_restore --create" fails if the database exists, and
> > pg_restore w/o --create fails unless the database exists. I think we should
> > continue that pattern in this new feature. If not, pg_restore should document
> > how it treats pg_dumpall-sourced dumps with the "create if not exists"
> > semantics appearing here.
>
> Added one more doc line for this case.
>
> Here, I am attaching a patch. Please let me know feedback.
>
> --
> Thanks and Regards
> Mahendra Singh Thalor
> EnterpriseDB: http://www.enterprisedb.com
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v01-17-july-use-appendShellString-to-append-file-names.noci (9.1K, 2-v01-17-july-use-appendShellString-to-append-file-names.noci)
download
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-17 11:11 Álvaro Herrera <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
2 siblings, 2 replies; 65+ messages in thread
From: Álvaro Herrera @ 2025-07-17 11:11 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Noah Misch <[email protected]>; Andrew Dunstan <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-Jul-17, Mahendra Singh Thalor wrote:
> To pg_restore, we are giving a dump of pg_dumpall which has a
> global.dat file and we have drop commands in the global.dat file so
> when we are using 'globals-only', we are dropping databases as we have
> DROP commands.
> As of now, we don't have any filter for global.dat file in restore. If
> a user wants to restore only globals(without droping db), then they
> should use 'globals-only' in pg_dumpall.
> Or if we don't want to DROP databases by global.dat file, then we
> should add a filter in pg_restore (hard to implement as we have SQL
> commands in global.dat file).
I think dropping database is dangerous and makes no practical sense;
doing it renders pg_dumpall --clean completely unusable. You're arguing
from the point of view of ease of implementation, but that doesn't help
users.
> I think, for this case, we can do some
> more doc changes.
> Example: pg_restore --globals-only : this will restore the global.dat
> file(including all drop commands). It might drop databases if any drop
> commands.
I don't think doc changes are useful.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"I love the Postgres community. It's all about doing things _properly_. :-)"
(David Garamond)
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-17 12:52 Mahendra Singh Thalor <[email protected]>
parent: Álvaro Herrera <[email protected]>
1 sibling, 0 replies; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-07-17 12:52 UTC (permalink / raw)
To: Álvaro Herrera <[email protected]>; +Cc: Noah Misch <[email protected]>; Andrew Dunstan <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Thanks Álvaro for the feedback.
On Thu, 17 Jul 2025 at 16:41, Álvaro Herrera <[email protected]>
wrote:
>
> On 2025-Jul-17, Mahendra Singh Thalor wrote:
>
> > To pg_restore, we are giving a dump of pg_dumpall which has a
> > global.dat file and we have drop commands in the global.dat file so
> > when we are using 'globals-only', we are dropping databases as we have
> > DROP commands.
> > As of now, we don't have any filter for global.dat file in restore. If
> > a user wants to restore only globals(without droping db), then they
> > should use 'globals-only' in pg_dumpall.
> > Or if we don't want to DROP databases by global.dat file, then we
> > should add a filter in pg_restore (hard to implement as we have SQL
> > commands in global.dat file).
>
> I think dropping database is dangerous and makes no practical sense;
> doing it renders pg_dumpall --clean completely unusable. You're arguing
> from the point of view of ease of implementation, but that doesn't help
> users.
I have 2 more solutions for this case.
*Solution1*: dump DROP database/role/tablespace commands in global_drop.dat
(or dump only DROP DATABASE commands in global_drop.dat file) and skip
restoring this file with globals-only.
*Solution2*: add one more filter in restore to skip the "DROP DATABASE"
command as we already have one filter for "CREATE USER".
Based on *solution1*, I made a WIP patch. Here, I am attaching a patch for
feedback.
Note: please use this v02 patch for review.
>
> > I think, for this case, we can do some
> > more doc changes.
> > Example: pg_restore --globals-only : this will restore the global.dat
> > file(including all drop commands). It might drop databases if any drop
> > commands.
>
> I don't think doc changes are useful.
>
> --
> Álvaro Herrera 48°01'N 7°57'E —
https://www.EnterpriseDB.com/
> "I love the Postgres community. It's all about doing things _properly_.
:-)"
> (David Garamond)
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v02-17-july-use-appendShellString-to-append-file-names.noci (15.2K, 3-v02-17-july-use-appendShellString-to-append-file-names.noci)
download
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-17 16:09 Andrew Dunstan <[email protected]>
parent: Álvaro Herrera <[email protected]>
1 sibling, 0 replies; 65+ messages in thread
From: Andrew Dunstan @ 2025-07-17 16:09 UTC (permalink / raw)
To: Álvaro Herrera <[email protected]>; Mahendra Singh Thalor <[email protected]>; +Cc: Noah Misch <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-07-17 Th 7:11 AM, Álvaro Herrera wrote:
> On 2025-Jul-17, Mahendra Singh Thalor wrote:
>
>> To pg_restore, we are giving a dump of pg_dumpall which has a
>> global.dat file and we have drop commands in the global.dat file so
>> when we are using 'globals-only', we are dropping databases as we have
>> DROP commands.
>> As of now, we don't have any filter for global.dat file in restore. If
>> a user wants to restore only globals(without droping db), then they
>> should use 'globals-only' in pg_dumpall.
>> Or if we don't want to DROP databases by global.dat file, then we
>> should add a filter in pg_restore (hard to implement as we have SQL
>> commands in global.dat file).
> I think dropping database is dangerous and makes no practical sense;
> doing it renders pg_dumpall --clean completely unusable. You're arguing
> from the point of view of ease of implementation, but that doesn't help
> users.
Yeah. I also agree with Noah that we should be consistent with pg_dump.
And we should err on the side of caution. If we impose a little
inconvenience on the user by requiring them to drop a database, it's
better than surprising them by dropping a database when they didn't
expect it.
There are some subtleties here. pg_restore will only issue DROP DATABASE
of you use the -C flag, even if you specify --clean, so we need to be
very careful about issuing DROP DATABASE.
I confess that all this didn't occur to me when working on the commit.
>> I think, for this case, we can do some
>> more doc changes.
>> Example: pg_restore --globals-only : this will restore the global.dat
>> file(including all drop commands). It might drop databases if any drop
>> commands.
> I don't think doc changes are useful.
Yeah, I don't think this is something that can be cured by documentation.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-21 20:41 Andrew Dunstan <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Andrew Dunstan @ 2025-07-21 20:41 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; Noah Misch <[email protected]>; +Cc: Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-07-17 Th 6:18 AM, Mahendra Singh Thalor wrote
>>>>>> --- a/src/bin/pg_dump/pg_restore.c
>>>>>> +++ b/src/bin/pg_dump/pg_restore.c
>>>>>> +/*
>>>>>> + * read_one_statement
>>>>>> + *
>>>>>> + * This will start reading from passed file pointer using fgetc and read till
>>>>>> + * semicolon(sql statement terminator for global.dat file)
>>>>>> + *
>>>>>> + * EOF is returned if end-of-file input is seen; time to shut down.
>>>>> What makes it okay to use this particular subset of SQL lexing?
>>>> To support complex syntax, we used this code from another file.
>>> I'm hearing that you copied this code from somewhere. Running
>>> "git grep 'time to shut down'" suggests you copied it from
>>> InteractiveBackend(). Is that right? I do see other similarities between
>>> read_one_statement() and InteractiveBackend().
>>>
>>> Copying InteractiveBackend() provides negligible assurance that this is the
>>> right subset of SQL lexing. Only single-user mode uses InteractiveBackend().
>>> Single-user mode survives mostly as a last resort for recovering from having
>>> reached xidStopLimit, is rarely used, and only superusers write queries to it.
>> Yes, we copied this from InteractiveBackend to read statements from
>> global.dat file.
Maybe we should ensure that identifiers with CR or LF are turned into
Unicode quoted identifiers, so each SQL statement would always only
occupy one line. Or just reject role and tablespace names with CR or LF
altogether, just as we do for database names.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-22 00:53 Noah Misch <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 2 replies; 65+ messages in thread
From: Noah Misch @ 2025-07-22 00:53 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Mon, Jul 21, 2025 at 04:41:03PM -0400, Andrew Dunstan wrote:
> On 2025-07-17 Th 6:18 AM, Mahendra Singh Thalor wrote
> > > > > > > --- a/src/bin/pg_dump/pg_restore.c
> > > > > > > +++ b/src/bin/pg_dump/pg_restore.c
> > > > > > > +/*
> > > > > > > + * read_one_statement
> > > > > > > + *
> > > > > > > + * This will start reading from passed file pointer using fgetc and read till
> > > > > > > + * semicolon(sql statement terminator for global.dat file)
> > > > > > > + *
> > > > > > > + * EOF is returned if end-of-file input is seen; time to shut down.
> > > > > > What makes it okay to use this particular subset of SQL lexing?
> > > > > To support complex syntax, we used this code from another file.
> > > > I'm hearing that you copied this code from somewhere. Running
> > > > "git grep 'time to shut down'" suggests you copied it from
> > > > InteractiveBackend(). Is that right? I do see other similarities between
> > > > read_one_statement() and InteractiveBackend().
> > > >
> > > > Copying InteractiveBackend() provides negligible assurance that this is the
> > > > right subset of SQL lexing. Only single-user mode uses InteractiveBackend().
> > > > Single-user mode survives mostly as a last resort for recovering from having
> > > > reached xidStopLimit, is rarely used, and only superusers write queries to it.
> > > Yes, we copied this from InteractiveBackend to read statements from
> > > global.dat file.
>
> Maybe we should ensure that identifiers with CR or LF are turned into
> Unicode quoted identifiers, so each SQL statement would always only occupy
> one line.
Interesting. That might work.
> Or just reject role and tablespace names with CR or LF altogether,
> just as we do for database names.
There are other ways to get multi-line statements. Non-exhaustive list:
- pg_db_role_setting.setconfig
- pg_shdescription.description
- pg_shseclabel.label
- pg_tablespace.spcoptions (if we add a text option in the future)
I think this decision about lexing also ties to other unfinished open item
work of aligning "pg_dumpall -Fd;pg_restore [options]" behavior with "pg_dump
-Fd;pg_restore [options]". "pg_restore --no-privileges" should not restore
pg_tablespace.spcacl, and "pg_restore --no-comments" should not emit COMMENT
statements.
I suspect this is going to end with a structured dump like we use on the
pg_dump (per-database) side. It's not an accident that v17 pg_restore doesn't
lex text files to do its job. pg_dumpall deals with a more-limited set of
statements than pg_dump deals with, but they're not _that much_ more limited.
I won't veto a lexing-based approach if it gets the behaviors right, but I
don't have high hopes for it getting the behaviors right and staying that way.
(I almost said "pg_restore --no-owner" should not restore
pg_tablespace.spcowner, but v17 "pg_dumpall --no-owner" does restore it. One
could argue for or against aligning $SUBJECT behavior w/ v17's mistake there.)
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-22 01:43 Andrew Dunstan <[email protected]>
parent: Noah Misch <[email protected]>
1 sibling, 0 replies; 65+ messages in thread
From: Andrew Dunstan @ 2025-07-22 01:43 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-07-21 Mo 8:53 PM, Noah Misch wrote:
> On Mon, Jul 21, 2025 at 04:41:03PM -0400, Andrew Dunstan wrote:
>> On 2025-07-17 Th 6:18 AM, Mahendra Singh Thalor wrote
>>>>>>>> --- a/src/bin/pg_dump/pg_restore.c
>>>>>>>> +++ b/src/bin/pg_dump/pg_restore.c
>>>>>>>> +/*
>>>>>>>> + * read_one_statement
>>>>>>>> + *
>>>>>>>> + * This will start reading from passed file pointer using fgetc and read till
>>>>>>>> + * semicolon(sql statement terminator for global.dat file)
>>>>>>>> + *
>>>>>>>> + * EOF is returned if end-of-file input is seen; time to shut down.
>>>>>>> What makes it okay to use this particular subset of SQL lexing?
>>>>>> To support complex syntax, we used this code from another file.
>>>>> I'm hearing that you copied this code from somewhere. Running
>>>>> "git grep 'time to shut down'" suggests you copied it from
>>>>> InteractiveBackend(). Is that right? I do see other similarities between
>>>>> read_one_statement() and InteractiveBackend().
>>>>>
>>>>> Copying InteractiveBackend() provides negligible assurance that this is the
>>>>> right subset of SQL lexing. Only single-user mode uses InteractiveBackend().
>>>>> Single-user mode survives mostly as a last resort for recovering from having
>>>>> reached xidStopLimit, is rarely used, and only superusers write queries to it.
>>>> Yes, we copied this from InteractiveBackend to read statements from
>>>> global.dat file.
>> Maybe we should ensure that identifiers with CR or LF are turned into
>> Unicode quoted identifiers, so each SQL statement would always only occupy
>> one line.
> Interesting. That might work.
>
>> Or just reject role and tablespace names with CR or LF altogether,
>> just as we do for database names.
> There are other ways to get multi-line statements. Non-exhaustive list:
>
> - pg_db_role_setting.setconfig
> - pg_shdescription.description
> - pg_shseclabel.label
> - pg_tablespace.spcoptions (if we add a text option in the future)
>
> I think this decision about lexing also ties to other unfinished open item
> work of aligning "pg_dumpall -Fd;pg_restore [options]" behavior with "pg_dump
> -Fd;pg_restore [options]". "pg_restore --no-privileges" should not restore
> pg_tablespace.spcacl, and "pg_restore --no-comments" should not emit COMMENT
> statements.
>
> I suspect this is going to end with a structured dump like we use on the
> pg_dump (per-database) side. It's not an accident that v17 pg_restore doesn't
> lex text files to do its job. pg_dumpall deals with a more-limited set of
> statements than pg_dump deals with, but they're not _that much_ more limited.
> I won't veto a lexing-based approach if it gets the behaviors right, but I
> don't have high hopes for it getting the behaviors right and staying that way.
Yeah, that was my original idea. But maybe instead of extending the
archive mechanism, we could do something more lightweight, e.g. output
the statements as a JSON array.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-24 17:50 Noah Misch <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
2 siblings, 0 replies; 65+ messages in thread
From: Noah Misch @ 2025-07-24 17:50 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Thu, Jul 17, 2025 at 03:46:41PM +0530, Mahendra Singh Thalor wrote:
> On Wed, 16 Jul 2025 at 05:50, Noah Misch <[email protected]> wrote:
> > On Thu, Jul 10, 2025 at 12:21:03AM +0530, Mahendra Singh Thalor wrote:
> > > On Wed, 9 Jul 2025 at 02:58, Noah Misch <[email protected]> wrote:
> > > > On Fri, Apr 04, 2025 at 04:11:05PM -0400, Andrew Dunstan wrote:
> > > > > +/*
> > > > > + * get_dbnames_list_to_restore
> > > > > + *
> > > > > + * This will mark for skipping any entries from dbname_oid_list that pattern match an
> > > > > + * entry in the db_exclude_patterns list.
> > > > > + *
> > > > > + * Returns the number of database to be restored.
> > > > > + *
> > > > > + */
> > > > > +static int
> > > > > +get_dbnames_list_to_restore(PGconn *conn,
> > > > > + SimpleOidStringList *dbname_oid_list,
> > > > > + SimpleStringList db_exclude_patterns)
> > > > > +{
> > > > > + int count_db = 0;
> > > > > + PQExpBuffer query;
> > > > > + PGresult *res;
> > > > > +
> > > > > + query = createPQExpBuffer();
> > > > > +
> > > > > + if (!conn)
> > > > > + pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
> > > >
> > > > When do we not have a connection here? We'd need to document this behavior
> > > > variation if it stays, but I'd prefer if we can just rely on having a
> > > > connection.
> > >
> > > Yes, we can document this behavior.
> >
> > My review asked a question there. I don't see an answer to that question.
> > Would you answer that question?
>
> Example: if there is no active database, even postgres/template1, then
> we will consider PATTEREN as NAME. This is the rare case.
> In attached patch, I added one doc line also for this case.
If I change s/pg_log_info/pg_fatal/, check-world still passes. So no test is
reaching the !conn case. If one wanted to write a test that reaches the !conn
test, how would they do that?
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-24 18:02 Robert Haas <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
1 sibling, 0 replies; 65+ messages in thread
From: Robert Haas @ 2025-07-24 18:02 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Noah Misch <[email protected]>; Andrew Dunstan <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Wed, Jul 9, 2025 at 2:51 PM Mahendra Singh Thalor <[email protected]> wrote:
> > This drops all databases:
> >
> > pg_dumpall --clean -Fd -f /tmp/dump
> > pg_restore -d template1 --globals-only /tmp/dump
> >
> > That didn't match my expectations given this help text:
> >
> > $ pg_restore --help|grep global
> > -g, --globals-only restore only global objects, no databases
>
> Databases are global objects so due to --clean command, we are putting
> drop commands in global.dat for all the databases. While restoring, we
> used the "--globals-only" option so we are dropping all these
> databases by global.dat file.
>
> Please let us know your expectations for this specific case.
I am not sure whether pg_dumpall --clean should ever drop databases,
but it certainly shouldn't do it with --globals-only. In that case,
it's not restoring the databases, so dropping them seems
catastrophically bad.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-24 20:33 Andrew Dunstan <[email protected]>
parent: Noah Misch <[email protected]>
1 sibling, 1 reply; 65+ messages in thread
From: Andrew Dunstan @ 2025-07-24 20:33 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-07-21 Mo 8:53 PM, Noah Misch wrote:
>
> I suspect this is going to end with a structured dump like we use on the
> pg_dump (per-database) side. It's not an accident that v17 pg_restore doesn't
> lex text files to do its job. pg_dumpall deals with a more-limited set of
> statements than pg_dump deals with, but they're not _that much_ more limited.
> I won't veto a lexing-based approach if it gets the behaviors right, but I
> don't have high hopes for it getting the behaviors right and staying that way.
I have been talking offline with Mahendra about this. I agree that we
would be better off with a structured object for globals. But the thing
that's been striking me all afternoon as I have pondered it is that we
should not be designing such an animal at this stage of the cycle.
Whatever we do we're going to be stuck supporting, so I have very
reluctantly come to the conclusion that it would probably be better to
back the feature out and have another go for PG 19.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-25 16:21 Noah Misch <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Noah Misch @ 2025-07-25 16:21 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Thu, Jul 24, 2025 at 04:33:15PM -0400, Andrew Dunstan wrote:
> On 2025-07-21 Mo 8:53 PM, Noah Misch wrote:
> > I suspect this is going to end with a structured dump like we use on the
> > pg_dump (per-database) side. It's not an accident that v17 pg_restore doesn't
> > lex text files to do its job. pg_dumpall deals with a more-limited set of
> > statements than pg_dump deals with, but they're not _that much_ more limited.
> > I won't veto a lexing-based approach if it gets the behaviors right, but I
> > don't have high hopes for it getting the behaviors right and staying that way.
>
> I have been talking offline with Mahendra about this. I agree that we would
> be better off with a structured object for globals. But the thing that's
> been striking me all afternoon as I have pondered it is that we should not
> be designing such an animal at this stage of the cycle. Whatever we do we're
> going to be stuck supporting, so I have very reluctantly come to the
> conclusion that it would probably be better to back the feature out and have
> another go for PG 19.
That makes sense to me. It would be quite a sprint to get this done in time,
and that wouldn't leave much room for additional testing and feedback before
the final release. I agree with the reluctance and with the conclusion.
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-25 19:31 Andrew Dunstan <[email protected]>
parent: Noah Misch <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Andrew Dunstan @ 2025-07-25 19:31 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-07-25 Fr 12:21 PM, Noah Misch wrote:
> On Thu, Jul 24, 2025 at 04:33:15PM -0400, Andrew Dunstan wrote:
>> On 2025-07-21 Mo 8:53 PM, Noah Misch wrote:
>>> I suspect this is going to end with a structured dump like we use on the
>>> pg_dump (per-database) side. It's not an accident that v17 pg_restore doesn't
>>> lex text files to do its job. pg_dumpall deals with a more-limited set of
>>> statements than pg_dump deals with, but they're not _that much_ more limited.
>>> I won't veto a lexing-based approach if it gets the behaviors right, but I
>>> don't have high hopes for it getting the behaviors right and staying that way.
>> I have been talking offline with Mahendra about this. I agree that we would
>> be better off with a structured object for globals. But the thing that's
>> been striking me all afternoon as I have pondered it is that we should not
>> be designing such an animal at this stage of the cycle. Whatever we do we're
>> going to be stuck supporting, so I have very reluctantly come to the
>> conclusion that it would probably be better to back the feature out and have
>> another go for PG 19.
> That makes sense to me. It would be quite a sprint to get this done in time,
> and that wouldn't leave much room for additional testing and feedback before
> the final release. I agree with the reluctance and with the conclusion.
Before we throw the baby out with the bathwater, how about this
suggestion? pg_dumpall would continue to produce globals.dat, but it
wouldn't be processed by pg_restore, which would only restore the
individual databases. Or else we just don't produce globals.dat at all.
Then we could introduce a structured object that pg_restore could safely
use for release 19, and I think we'd still have something useful for
release 18.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-25 20:59 Tom Lane <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Tom Lane @ 2025-07-25 20:59 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Noah Misch <[email protected]>; Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Andrew Dunstan <[email protected]> writes:
> Before we throw the baby out with the bathwater, how about this
> suggestion? pg_dumpall would continue to produce globals.dat, but it
> wouldn't be processed by pg_restore, which would only restore the
> individual databases. Or else we just don't produce globals.dat at all.
> Then we could introduce a structured object that pg_restore could safely
> use for release 19, and I think we'd still have something useful for
> release 18.
I dunno ... that seems like a pretty weird behavior. People would
have to do a separate text-mode "pg_dumpall -g" and remember to
restore that too. Admittedly, this could be more convenient than
"pg_dumpall -g" plus separately pg_dump'ing each database, which is
what people have to do today if they want anything smarter than a flat
text dumpfile. But it still seems like a hack --- and it would not be
compatible with v19, where presumably "pg_dumpall | pg_restore"
*would* restore globals. I think that the prospect of changing
dump/restore scripts and then having to change them again in v19
isn't too appetizing.
regards, tom lane
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-27 23:56 Noah Misch <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Noah Misch @ 2025-07-27 23:56 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Fri, Jul 25, 2025 at 04:59:29PM -0400, Tom Lane wrote:
> Andrew Dunstan <[email protected]> writes:
> > Before we throw the baby out with the bathwater, how about this
> > suggestion? pg_dumpall would continue to produce globals.dat, but it
> > wouldn't be processed by pg_restore, which would only restore the
> > individual databases. Or else we just don't produce globals.dat at all.
> > Then we could introduce a structured object that pg_restore could safely
> > use for release 19, and I think we'd still have something useful for
> > release 18.
>
> I dunno ... that seems like a pretty weird behavior. People would
> have to do a separate text-mode "pg_dumpall -g" and remember to
> restore that too. Admittedly, this could be more convenient than
> "pg_dumpall -g" plus separately pg_dump'ing each database, which is
> what people have to do today if they want anything smarter than a flat
> text dumpfile. But it still seems like a hack --- and it would not be
> compatible with v19, where presumably "pg_dumpall | pg_restore"
> *would* restore globals. I think that the prospect of changing
> dump/restore scripts and then having to change them again in v19
> isn't too appetizing.
+1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-28 12:04 Andrew Dunstan <[email protected]>
parent: Noah Misch <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Andrew Dunstan @ 2025-07-28 12:04 UTC (permalink / raw)
To: Noah Misch <[email protected]>; Tom Lane <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-07-27 Su 7:56 PM, Noah Misch wrote:
> On Fri, Jul 25, 2025 at 04:59:29PM -0400, Tom Lane wrote:
>> Andrew Dunstan <[email protected]> writes:
>>> Before we throw the baby out with the bathwater, how about this
>>> suggestion? pg_dumpall would continue to produce globals.dat, but it
>>> wouldn't be processed by pg_restore, which would only restore the
>>> individual databases. Or else we just don't produce globals.dat at all.
>>> Then we could introduce a structured object that pg_restore could safely
>>> use for release 19, and I think we'd still have something useful for
>>> release 18.
>> I dunno ... that seems like a pretty weird behavior. People would
>> have to do a separate text-mode "pg_dumpall -g" and remember to
>> restore that too. Admittedly, this could be more convenient than
>> "pg_dumpall -g" plus separately pg_dump'ing each database, which is
>> what people have to do today if they want anything smarter than a flat
>> text dumpfile. But it still seems like a hack --- and it would not be
>> compatible with v19, where presumably "pg_dumpall | pg_restore"
>> *would* restore globals. I think that the prospect of changing
>> dump/restore scripts and then having to change them again in v19
>> isn't too appetizing.
> +1
OK, got it. Will revert.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-29 20:09 Andrew Dunstan <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 2 replies; 65+ messages in thread
From: Andrew Dunstan @ 2025-07-29 20:09 UTC (permalink / raw)
To: Noah Misch <[email protected]>; Tom Lane <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-07-28 Mo 8:04 AM, Andrew Dunstan wrote:
>
> On 2025-07-27 Su 7:56 PM, Noah Misch wrote:
>> On Fri, Jul 25, 2025 at 04:59:29PM -0400, Tom Lane wrote:
>>> Andrew Dunstan <[email protected]> writes:
>>>> Before we throw the baby out with the bathwater, how about this
>>>> suggestion? pg_dumpall would continue to produce globals.dat, but it
>>>> wouldn't be processed by pg_restore, which would only restore the
>>>> individual databases. Or else we just don't produce globals.dat at
>>>> all.
>>>> Then we could introduce a structured object that pg_restore could
>>>> safely
>>>> use for release 19, and I think we'd still have something useful for
>>>> release 18.
>>> I dunno ... that seems like a pretty weird behavior. People would
>>> have to do a separate text-mode "pg_dumpall -g" and remember to
>>> restore that too. Admittedly, this could be more convenient than
>>> "pg_dumpall -g" plus separately pg_dump'ing each database, which is
>>> what people have to do today if they want anything smarter than a flat
>>> text dumpfile. But it still seems like a hack --- and it would not be
>>> compatible with v19, where presumably "pg_dumpall | pg_restore"
>>> *would* restore globals. I think that the prospect of changing
>>> dump/restore scripts and then having to change them again in v19
>>> isn't too appetizing.
>> +1
>
>
> OK, got it. Will revert.
>
>
>
here's a reversion patch for master. It applies cleanly to release 18 as
well. Thanks to Mahendra Singh Thalor for helping me sanity check it
(Any issues are of course my responsibility)
I'll work on pulling the entry out of the release notes.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
Attachments:
[text/x-patch] dumpall-nontext-revert.patch (71.3K, 2-dumpall-nontext-revert.patch)
download | inline diff:
commit 77d9ccbce21
Author: Andrew Dunstan <[email protected]>
Date: Tue Jul 29 15:54:15 2025 -0400
Revert Non text modes for pg_dumpall, and pg_restore support
Recent discussions of the mechanisms used to manage global data have
raised concerns about their robustness and security. Rather than try
to deal with those concerns at a very late stage of the release cycle,
the conclusion is to revert these features and work on them for the
next release.
This reverts parts or all of the following commits:
1495eff7bdb Non text modes for pg_dumpall, correspondingly change pg_restore
5db3bf7391d Clean up from commit 1495eff7bdb
289f74d0cb2 Add more TAP tests for pg_dumpall
2ef57908067 Fix a couple of error messages and tests for them
b52a4a5f285 Clean up error messages from 1495eff7bdb
4170298b6ec Further cleanup for directory creation on pg_dump/pg_dumpall
22cb6d28950 Fix memory leak in pg_restore.c
928394b664b Improve various new-to-v18 appendStringInfo calls
39729ec01d2 Fix fat fingering in 22cb6d28950
5822bf21d50 Add missing space in pg_restore documentation.
f09088a01d3 Free memory properly in pg_restore.c
40b9c27014d pg_restore cleanups
4aad2cb7707 Portability fix: isdigit() must be passed an unsigned char.
88e947136b4 Fix typos and grammar in the code
f60420cff66 doc: Alphabetize long options for pg_dump[all].
bc35adee8d7 doc: Put new options in consistent order on man pages
a876464abc7 Message style improvements
dec6643487b Improve pg_dump/pg_dumpall help synopses and terminology
0ebd2425558 Run pgperltidy
Discussion: https://postgr.es/m/[email protected]
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 8ca68da5a55..f4cbc8288e3 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,10 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
-
- <refpurpose>
- export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
- </refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -36,7 +33,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into an SQL script file or an archive. The output contains
+ of a cluster into one script file. The script file contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -55,16 +52,11 @@ PostgreSQL documentation
</para>
<para>
- Plain text SQL scripts will be written to the standard output. Use the
+ The SQL script will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
- <para>
- Archives in other formats will be placed in a directory named using the
- <option>-f</option>/<option>--file</option>, which is required in this case.
- </para>
-
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -129,85 +121,10 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
- <varlistentry>
- <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
- <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
- <listitem>
- <para>
- Specify the format of dump files. In plain format, all the dump data is
- sent in a single text stream. This is the default.
-
- In all other modes, <application>pg_dumpall</application> first creates two files:
- <filename>global.dat</filename> and <filename>map.dat</filename>, in the directory
- specified by <option>--file</option>.
- The first file contains global data, such as roles and tablespaces. The second
- contains a mapping between database oids and names. These files are used by
- <application>pg_restore</application>. Data for individual databases is placed in
- <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
-
- <variablelist>
- <varlistentry>
- <term><literal>d</literal></term>
- <term><literal>directory</literal></term>
- <listitem>
- <para>
- Output directory-format archives for each database,
- suitable for input into pg_restore. The directory
- will have database <type>oid</type> as its name.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><literal>p</literal></term>
- <term><literal>plain</literal></term>
- <listitem>
- <para>
- Output a plain-text SQL script file (the default).
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><literal>c</literal></term>
- <term><literal>custom</literal></term>
- <listitem>
- <para>
- Output a custom-format archive for each database,
- suitable for input into pg_restore. The archive
- will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
- <type>oid</type> of the database.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><literal>t</literal></term>
- <term><literal>tar</literal></term>
- <listitem>
- <para>
- Output a tar-format archive for each database,
- suitable for input into pg_restore. The archive
- will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
- <type>oid</type> of the database.
- </para>
- </listitem>
- </varlistentry>
-
- </variablelist>
-
- Note: see <xref linkend="app-pgdump"/> for details
- of how the various non plain text archives work.
-
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index b649bd3a5ae..2abe05d47e9 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,9 +18,8 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore <productname>PostgreSQL</productname> databases from archives
- created by <application>pg_dump</application> or
- <application>pg_dumpall</application>
+ restore a <productname>PostgreSQL</productname> database from an
+ archive file created by <application>pg_dump</application>
</refpurpose>
</refnamediv>
@@ -39,14 +38,13 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database or cluster from an archive
- created by <xref linkend="app-pgdump"/> or
- <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database from an archive
+ created by <xref linkend="app-pgdump"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database or cluster to the state it was in at the time it was saved. The
- archives also allow <application>pg_restore</application> to
+ database to the state it was in at the time it was saved. The
+ archive files also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive formats are designed to be
+ prior to being restored. The archive files are designed to be
portable across architectures.
</para>
@@ -54,17 +52,10 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database.
- When restoring from a dump made by <application>pg_dumpall</application>,
- each database will be created and then the restoration will be run in that
- database.
-
- Otherwise, when a database name is not specified, a script containing the SQL
- commands necessary to rebuild the database or cluster is created and written
+ the database. Otherwise, a script containing the SQL
+ commands necessary to rebuild the database is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application> or
- <application>pg_dumpall</application>.
-
+ the plain text output format of <application>pg_dump</application>.
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -149,8 +140,6 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
- <option>--create</option> is required when restoring multiple databases
- from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -246,19 +235,6 @@ PostgreSQL documentation
</listitem>
</varlistentry>
- <varlistentry>
- <term><option>-g</option></term>
- <term><option>--globals-only</option></term>
- <listitem>
- <para>
- Restore only global objects (roles and tablespaces), no databases.
- </para>
- <para>
- This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -603,28 +579,6 @@ PostgreSQL documentation
</listitem>
</varlistentry>
- <varlistentry>
- <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
- <listitem>
- <para>
- Do not restore databases whose name matches
- <replaceable class="parameter">pattern</replaceable>.
- Multiple patterns can be excluded by writing multiple
- <option>--exclude-database</option> switches. The
- <replaceable class="parameter">pattern</replaceable> parameter is
- interpreted as a pattern according to the same rules used by
- <application>psql</application>'s <literal>\d</literal>
- commands (see <xref linkend="app-psql-patterns"/>),
- so multiple databases can also be excluded by writing wildcard
- characters in the pattern. When using wildcards, be careful to
- quote the pattern if needed to prevent shell wildcard expansion.
- </para>
- <para>
- This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 4a4ebbd8ec9..a2233b0a1b4 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -102,7 +102,6 @@ tests += {
't/003_pg_dump_with_server.pl',
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
- 't/006_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 5974d6706fd..086adcdc502 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,16 +333,6 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
-/*
- * When pg_restore restores multiple databases, then update already added entry
- * into array for cleanup.
- */
-void
-replace_on_exit_close_archive(Archive *AHX)
-{
- shutdown_info.AHX = AHX;
-}
-
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index af0007fb6d2..4ebef1e8644 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -308,7 +308,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX, bool append_data);
+extern void RestoreArchive(Archive *AHX);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 30e0da31aa3..dce88f040ac 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -87,7 +87,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec, bool append_data);
+ const pg_compress_specification compression_spec);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,14 +339,9 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/*
- * RestoreArchive
- *
- * If append_data is set, then append data into file as we are restoring dump
- * of multiple databases which was taken by pg_dumpall.
- */
+/* Public */
void
-RestoreArchive(Archive *AHX, bool append_data)
+RestoreArchive(Archive *AHX)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -463,7 +458,7 @@ RestoreArchive(Archive *AHX, bool append_data)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
+ SetOutput(AH, ropt->filename, ropt->compression_spec);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1302,7 +1297,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec, false);
+ SetOutput(AH, ropt->filename, out_compression_spec);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1681,8 +1676,7 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec,
- bool append_data)
+ const pg_compress_specification compression_spec)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1702,7 +1696,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (append_data || AH->mode == archModeAppend)
+ if (AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 365073b3eae..325b53fc9bd 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,7 +394,6 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
-extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index d94d0de2a5d..b5ba3b46dd9 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH, false);
+ RestoreArchive((Archive *) AH);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 6298edb26b5..f543d418e46 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1265,7 +1265,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout, false);
+ RestoreArchive(fout);
CloseArchive(fout);
@@ -1276,7 +1276,7 @@ main(int argc, char **argv)
static void
help(const char *progname)
{
- printf(_("%s exports a PostgreSQL database as an SQL script or to other formats.\n\n"), progname);
+ printf(_("%s dumps a database as a text file or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [DBNAME]\n"), progname);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 100317b1aa9..f69f0260256 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -65,10 +65,9 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
+static void dumpDatabases(PGconn *conn);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts,
- char *dbfile, ArchiveFormat archDumpFormat);
+static int runPgDump(const char *dbname, const char *create_opts);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -77,7 +76,6 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
-static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -150,7 +148,6 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
- {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -201,8 +198,6 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
- ArchiveFormat archDumpFormat = archNull;
- const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -252,7 +247,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -280,9 +275,7 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
- case 'F':
- formatName = pg_strdup(optarg);
- break;
+
case 'g':
globals_only = true;
break;
@@ -431,21 +424,6 @@ main(int argc, char *argv[])
exit_nicely(1);
}
- /* Get format for dump. */
- archDumpFormat = parseDumpFormat(formatName);
-
- /*
- * If a non-plain format is specified, a file name is also required as the
- * path to the main directory.
- */
- if (archDumpFormat != archNull &&
- (!filename || strcmp(filename, "") == 0))
- {
- pg_log_error("option -F/--format=d|c|t requires option -f/--file");
- pg_log_error_hint("Try \"%s --help\" for more information.", progname);
- exit_nicely(1);
- }
-
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -510,33 +488,6 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
- /*
- * Open the output file if required, otherwise use stdout. If required,
- * then create new directory and global.dat file.
- */
- if (archDumpFormat != archNull)
- {
- char global_path[MAXPGPATH];
-
- /* Create new directory or accept the empty existing directory. */
- create_or_open_dir(filename);
-
- snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
-
- OPF = fopen(global_path, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open file \"%s\": %m", global_path);
- }
- else if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -576,6 +527,19 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
+ /*
+ * Open the output file if required, otherwise use stdout
+ */
+ if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* Set the client encoding if requested.
*/
@@ -675,7 +639,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn, archDumpFormat);
+ dumpDatabases(conn);
PQfinish(conn);
@@ -688,7 +652,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync && (archDumpFormat == archNull))
+ if (dosync)
(void) fsync_fname(filename, false);
}
@@ -699,14 +663,12 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
- printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
- " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1013,6 +975,9 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
+ if (PQntuples(res) > 0)
+ fprintf(OPF, "\n--\n-- User Configurations\n--\n");
+
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
@@ -1526,7 +1491,6 @@ dumpUserConfig(PGconn *conn, const char *username)
{
PQExpBuffer buf = createPQExpBuffer();
PGresult *res;
- static bool header_done = false;
printfPQExpBuffer(buf, "SELECT unnest(setconfig) FROM pg_db_role_setting "
"WHERE setdatabase = 0 AND setrole = "
@@ -1538,13 +1502,7 @@ dumpUserConfig(PGconn *conn, const char *username)
res = executeQuery(conn, buf->data);
if (PQntuples(res) > 0)
- {
- if (!header_done)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
- header_done = true;
-
fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", username);
- }
for (int i = 0; i < PQntuples(res); i++)
{
@@ -1618,13 +1576,10 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
+dumpDatabases(PGconn *conn)
{
PGresult *res;
int i;
- char db_subdir[MAXPGPATH];
- char dbfilepath[MAXPGPATH];
- FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1638,42 +1593,18 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname, oid "
+ "SELECT datname "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
- if (archDumpFormat == archNull && PQntuples(res) > 0)
+ if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
- /*
- * If directory/tar/custom format is specified, create a subdirectory
- * under the main directory and each database dump file or subdirectory
- * will be created in that subdirectory by pg_dump.
- */
- if (archDumpFormat != archNull)
- {
- char map_file_path[MAXPGPATH];
-
- snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
-
- /* Create a subdirectory with 'databases' name under main directory. */
- if (mkdir(db_subdir, pg_dir_create_mode) != 0)
- pg_fatal("could not create directory \"%s\": %m", db_subdir);
-
- snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
-
- /* Create a map file (to store dboid and dbname) */
- map_file = fopen(map_file_path, PG_BINARY_W);
- if (!map_file)
- pg_fatal("could not open file \"%s\": %m", map_file_path);
- }
-
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
- char *oid = PQgetvalue(res, i, 1);
- const char *create_opts = "";
+ const char *create_opts;
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1687,27 +1618,9 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
continue;
}
- /*
- * If this is not a plain format dump, then append dboid and dbname to
- * the map.dat file.
- */
- if (archDumpFormat != archNull)
- {
- if (archDumpFormat == archCustom)
- snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
- else if (archDumpFormat == archTar)
- snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
- else
- snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
-
- /* Put one line entry for dboid and dbname in map file. */
- fprintf(map_file, "%s %s\n", oid, dbname);
- }
-
pg_log_info("dumping database \"%s\"", dbname);
- if (archDumpFormat == archNull)
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
/*
* We assume that "template1" and "postgres" already exist in the
@@ -1721,9 +1634,12 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
if (output_clean)
create_opts = "--clean --create";
- /* Since pg_dump won't emit a \connect command, we must */
- else if (archDumpFormat == archNull)
+ else
+ {
+ create_opts = "";
+ /* Since pg_dump won't emit a \connect command, we must */
fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
}
else
create_opts = "--create";
@@ -1731,30 +1647,19 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
+ ret = runPgDump(dbname, create_opts);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- char global_path[MAXPGPATH];
-
- if (archDumpFormat != archNull)
- snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
- else
- snprintf(global_path, MAXPGPATH, "%s", filename);
-
- OPF = fopen(global_path, PG_BINARY_A);
+ OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- global_path);
+ filename);
}
}
- /* Close map file */
- if (archDumpFormat != archNull)
- fclose(map_file);
-
PQclear(res);
}
@@ -1764,8 +1669,7 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts, char *dbfile,
- ArchiveFormat archDumpFormat)
+runPgDump(const char *dbname, const char *create_opts)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1774,36 +1678,17 @@ runPgDump(const char *dbname, const char *create_opts, char *dbfile,
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
/*
- * If this is not a plain format dump, then append file name and dump
- * format to the pg_dump command to get archive dump.
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
*/
- if (archDumpFormat != archNull)
- {
- printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
- dbfile, create_opts);
-
- if (archDumpFormat == archDirectory)
- appendPQExpBufferStr(&cmd, " --format=directory ");
- else if (archDumpFormat == archCustom)
- appendPQExpBufferStr(&cmd, " --format=custom ");
- else if (archDumpFormat == archTar)
- appendPQExpBufferStr(&cmd, " --format=tar ");
- }
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
else
- {
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
- /*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
- */
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
- else
- appendPQExpBufferStr(&cmd, " -Fp ");
- }
+ appendPQExpBufferStr(&cmd, " -Fp ");
/*
* Append the database name to the already-constructed stem of connection
@@ -1948,36 +1833,3 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
-
-/*
- * parseDumpFormat
- *
- * This will validate dump formats.
- */
-static ArchiveFormat
-parseDumpFormat(const char *format)
-{
- ArchiveFormat archDumpFormat;
-
- if (pg_strcasecmp(format, "c") == 0)
- archDumpFormat = archCustom;
- else if (pg_strcasecmp(format, "custom") == 0)
- archDumpFormat = archCustom;
- else if (pg_strcasecmp(format, "d") == 0)
- archDumpFormat = archDirectory;
- else if (pg_strcasecmp(format, "directory") == 0)
- archDumpFormat = archDirectory;
- else if (pg_strcasecmp(format, "p") == 0)
- archDumpFormat = archNull;
- else if (pg_strcasecmp(format, "plain") == 0)
- archDumpFormat = archNull;
- else if (pg_strcasecmp(format, "t") == 0)
- archDumpFormat = archTar;
- else if (pg_strcasecmp(format, "tar") == 0)
- archDumpFormat = archTar;
- else
- pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
- format);
-
- return archDumpFormat;
-}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 6ef789cb06d..b4e1acdb63f 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump/pg_dumpall using the archiver
+ * from a backup archive created by pg_dump using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,15 +41,11 @@
#include "postgres_fe.h"
#include <ctype.h>
-#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
-#include "common/string.h"
-#include "connectdb.h"
#include "fe_utils/option_utils.h"
-#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -57,43 +53,18 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
-static bool file_exists_in_directory(const char *dir, const char *filename);
-static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
- int numWorkers, bool append_data, int num);
-static int read_one_statement(StringInfo inBuf, FILE *pfile);
-static int restore_all_databases(PGconn *conn, const char *dumpdirpath,
- SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
-static int process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
- const char *outfile);
-static void copy_or_print_global_file(const char *outfile, FILE *pfile);
-static int get_dbnames_list_to_restore(PGconn *conn,
- SimplePtrList *dbname_oid_list,
- SimpleStringList db_exclude_patterns);
-static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
- SimplePtrList *dbname_oid_list);
-
-/*
- * Stores a database OID and the corresponding name.
- */
-typedef struct DbOidName
-{
- Oid oid;
- char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
-} DbOidName;
-
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
+ int exit_code;
int numWorkers = 1;
+ Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
- int n_errors = 0;
- bool globals_only = false;
- SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -119,7 +90,6 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
- {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -174,7 +144,6 @@ main(int argc, char **argv)
{"with-statistics", no_argument, &with_statistics, 1},
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
- {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -203,7 +172,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -230,14 +199,11 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
- case 'g':
- /* restore only global.dat file from directory */
- globals_only = true;
- break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
+
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -352,9 +318,6 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
- case 6: /* database patterns to skip */
- simple_string_list_append(&db_exclude_patterns, optarg);
- break;
default:
/* getopt_long already emitted a complaint */
@@ -382,13 +345,6 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
- if (db_exclude_patterns.head != NULL && globals_only)
- {
- pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
- pg_log_error_hint("Try \"%s --help\" for more information.", progname);
- exit_nicely(1);
- }
-
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -496,114 +452,6 @@ main(int argc, char **argv)
opts->formatName);
}
- /*
- * If toc.dat file is not present in the current path, then check for
- * global.dat. If global.dat file is present, then restore all the
- * databases from map.dat (if it exists), but skip restoring those
- * matching --exclude-database patterns.
- */
- if (inputFileSpec != NULL && !file_exists_in_directory(inputFileSpec, "toc.dat") &&
- file_exists_in_directory(inputFileSpec, "global.dat"))
- {
- PGconn *conn = NULL; /* Connection to restore global sql
- * commands. */
-
- /*
- * Can only use --list or --use-list options with a single database
- * dump.
- */
- if (opts->tocSummary)
- pg_fatal("option -l/--list cannot be used when restoring an archive created by pg_dumpall");
- else if (opts->tocFile)
- pg_fatal("option -L/--use-list cannot be used when restoring an archive created by pg_dumpall");
-
- /*
- * To restore from a pg_dumpall archive, -C (create database) option
- * must be specified unless we are only restoring globals.
- */
- if (!globals_only && opts->createDB != 1)
- {
- pg_log_error("option -C/--create must be specified when restoring an archive created by pg_dumpall");
- pg_log_error_hint("Try \"%s --help\" for more information.", progname);
- pg_log_error_hint("Individual databases can be restored using their specific archives.");
- exit_nicely(1);
- }
-
- /*
- * Connect to the database to execute global sql commands from
- * global.dat file.
- */
- if (opts->cparams.dbname)
- {
- conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
- opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
- false, progname, NULL, NULL, NULL, NULL);
-
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
- }
-
- /* If globals-only, then return from here. */
- if (globals_only)
- {
- /*
- * Open global.dat file and execute/append all the global sql
- * commands.
- */
- n_errors = process_global_sql_commands(conn, inputFileSpec,
- opts->filename);
-
- if (conn)
- PQfinish(conn);
-
- pg_log_info("database restoring skipped because option -g/--globals-only was specified");
- }
- else
- {
- /* Now restore all the databases from map.dat */
- n_errors = restore_all_databases(conn, inputFileSpec, db_exclude_patterns,
- opts, numWorkers);
- }
-
- /* Free db pattern list. */
- simple_string_list_destroy(&db_exclude_patterns);
- }
- else /* process if global.dat file does not exist. */
- {
- if (db_exclude_patterns.head != NULL)
- pg_fatal("option --exclude-database can be used only when restoring an archive created by pg_dumpall");
-
- if (globals_only)
- pg_fatal("option -g/--globals-only can be used only when restoring an archive created by pg_dumpall");
-
- n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, 0);
- }
-
- /* Done, print a summary of ignored errors during restore. */
- if (n_errors)
- {
- pg_log_warning("errors ignored on restore: %d", n_errors);
- return 1;
- }
-
- return 0;
-}
-
-/*
- * restore_one_database
- *
- * This will restore one database using toc.dat file.
- *
- * returns the number of errors while doing restore.
- */
-static int
-restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
- int numWorkers, bool append_data, int num)
-{
- Archive *AH;
- int n_errors;
-
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -611,15 +459,9 @@ restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op. If we are
- * restoring multiple databases, then only update AX handle for cleanup as
- * the previous entry was already in the array and we had closed previous
- * connection, so we can use the same array slot.
+ * it's still NULL, the cleanup function will just be a no-op.
*/
- if (!append_data || num == 0)
- on_exit_close_archive(AH);
- else
- replace_on_exit_close_archive(AH);
+ on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -639,21 +481,25 @@ restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH, append_data);
+ RestoreArchive(AH);
}
- n_errors = AH->n_errors;
+ /* done, print a summary of ignored errors */
+ if (AH->n_errors)
+ pg_log_warning("errors ignored on restore: %d", AH->n_errors);
/* AH may be freed in CloseArchive? */
+ exit_code = AH->n_errors ? 1 : 0;
+
CloseArchive(AH);
- return n_errors;
+ return exit_code;
}
static void
usage(const char *progname)
{
- printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
+ printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -671,7 +517,6 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
- printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -688,7 +533,6 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
- printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -725,8 +569,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
- "combined and specified multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
+ "multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -831,585 +675,3 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
-
-/*
- * file_exists_in_directory
- *
- * Returns true if the file exists in the given directory.
- */
-static bool
-file_exists_in_directory(const char *dir, const char *filename)
-{
- struct stat st;
- char buf[MAXPGPATH];
-
- if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
- pg_fatal("directory name too long: \"%s\"", dir);
-
- return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
-}
-
-/*
- * read_one_statement
- *
- * This will start reading from passed file pointer using fgetc and read till
- * semicolon(sql statement terminator for global.dat file)
- *
- * EOF is returned if end-of-file input is seen; time to shut down.
- */
-
-static int
-read_one_statement(StringInfo inBuf, FILE *pfile)
-{
- int c; /* character read from getc() */
- int m;
-
- StringInfoData q;
-
- initStringInfo(&q);
-
- resetStringInfo(inBuf);
-
- /*
- * Read characters until EOF or the appropriate delimiter is seen.
- */
- while ((c = fgetc(pfile)) != EOF)
- {
- if (c != '\'' && c != '"' && c != '\n' && c != ';')
- {
- appendStringInfoChar(inBuf, (char) c);
- while ((c = fgetc(pfile)) != EOF)
- {
- if (c != '\'' && c != '"' && c != ';' && c != '\n')
- appendStringInfoChar(inBuf, (char) c);
- else
- break;
- }
- }
-
- if (c == '\'' || c == '"')
- {
- appendStringInfoChar(&q, (char) c);
- m = c;
-
- while ((c = fgetc(pfile)) != EOF)
- {
- appendStringInfoChar(&q, (char) c);
-
- if (c == m)
- {
- appendStringInfoString(inBuf, q.data);
- resetStringInfo(&q);
- break;
- }
- }
- }
-
- if (c == ';')
- {
- appendStringInfoChar(inBuf, (char) ';');
- break;
- }
-
- if (c == '\n')
- appendStringInfoChar(inBuf, (char) '\n');
- }
-
- pg_free(q.data);
-
- /* No input before EOF signal means time to quit. */
- if (c == EOF && inBuf->len == 0)
- return EOF;
-
- /* return something that's not EOF */
- return 'Q';
-}
-
-/*
- * get_dbnames_list_to_restore
- *
- * This will mark for skipping any entries from dbname_oid_list that pattern match an
- * entry in the db_exclude_patterns list.
- *
- * Returns the number of database to be restored.
- *
- */
-static int
-get_dbnames_list_to_restore(PGconn *conn,
- SimplePtrList *dbname_oid_list,
- SimpleStringList db_exclude_patterns)
-{
- int count_db = 0;
- PQExpBuffer query;
- PGresult *res;
-
- query = createPQExpBuffer();
-
- if (!conn)
- pg_log_info("considering PATTERN as NAME for --exclude-database option as no database connection while doing pg_restore");
-
- /*
- * Process one by one all dbnames and if specified to skip restoring, then
- * remove dbname from list.
- */
- for (SimplePtrListCell *db_cell = dbname_oid_list->head;
- db_cell; db_cell = db_cell->next)
- {
- DbOidName *dbidname = (DbOidName *) db_cell->ptr;
- bool skip_db_restore = false;
- PQExpBuffer db_lit = createPQExpBuffer();
-
- appendStringLiteralConn(db_lit, dbidname->str, conn);
-
- for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
- {
- /*
- * If there is an exact match then we don't need to try a pattern
- * match
- */
- if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
- skip_db_restore = true;
- /* Otherwise, try a pattern match if there is a connection */
- else if (conn)
- {
- int dotcnt;
-
- appendPQExpBufferStr(query, "SELECT 1 ");
- processSQLNamePattern(conn, query, pat_cell->val, false,
- false, NULL, db_lit->data,
- NULL, NULL, NULL, &dotcnt);
-
- if (dotcnt > 0)
- {
- pg_log_error("improper qualified name (too many dotted names): %s",
- dbidname->str);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- res = executeQuery(conn, query->data);
-
- if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
- {
- skip_db_restore = true;
- pg_log_info("database name \"%s\" matches exclude pattern \"%s\"", dbidname->str, pat_cell->val);
- }
-
- PQclear(res);
- resetPQExpBuffer(query);
- }
-
- if (skip_db_restore)
- break;
- }
-
- destroyPQExpBuffer(db_lit);
-
- /*
- * Mark db to be skipped or increment the counter of dbs to be
- * restored
- */
- if (skip_db_restore)
- {
- pg_log_info("excluding database \"%s\"", dbidname->str);
- dbidname->oid = InvalidOid;
- }
- else
- {
- count_db++;
- }
- }
-
- destroyPQExpBuffer(query);
-
- return count_db;
-}
-
-/*
- * get_dbname_oid_list_from_mfile
- *
- * Open map.dat file and read line by line and then prepare a list of database
- * names and corresponding db_oid.
- *
- * Returns, total number of database names in map.dat file.
- */
-static int
-get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimplePtrList *dbname_oid_list)
-{
- StringInfoData linebuf;
- FILE *pfile;
- char map_file_path[MAXPGPATH];
- int count = 0;
-
-
- /*
- * If there is only global.dat file in dump, then return from here as
- * there is no database to restore.
- */
- if (!file_exists_in_directory(dumpdirpath, "map.dat"))
- {
- pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
- return 0;
- }
-
- snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
-
- /* Open map.dat file. */
- pfile = fopen(map_file_path, PG_BINARY_R);
-
- if (pfile == NULL)
- pg_fatal("could not open file \"%s\": %m", map_file_path);
-
- initStringInfo(&linebuf);
-
- /* Append all the dbname/db_oid combinations to the list. */
- while (pg_get_line_buf(pfile, &linebuf))
- {
- Oid db_oid = InvalidOid;
- char *dbname;
- DbOidName *dbidname;
- int namelen;
- char *p = linebuf.data;
-
- /* Extract dboid. */
- while (isdigit((unsigned char) *p))
- p++;
- if (p > linebuf.data && *p == ' ')
- {
- sscanf(linebuf.data, "%u", &db_oid);
- p++;
- }
-
- /* dbname is the rest of the line */
- dbname = p;
- namelen = strlen(dbname);
-
- /* Report error and exit if the file has any corrupted data. */
- if (!OidIsValid(db_oid) || namelen <= 1)
- pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
- count + 1);
-
- pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
- dbname, db_oid, map_file_path);
-
- dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
- dbidname->oid = db_oid;
- strlcpy(dbidname->str, dbname, namelen);
-
- simple_ptr_list_append(dbname_oid_list, dbidname);
- count++;
- }
-
- /* Close map.dat file. */
- fclose(pfile);
-
- return count;
-}
-
-/*
- * restore_all_databases
- *
- * This will restore databases those dumps are present in
- * directory based on map.dat file mapping.
- *
- * This will skip restoring for databases that are specified with
- * exclude-database option.
- *
- * returns, number of errors while doing restore.
- */
-static int
-restore_all_databases(PGconn *conn, const char *dumpdirpath,
- SimpleStringList db_exclude_patterns, RestoreOptions *opts,
- int numWorkers)
-{
- SimplePtrList dbname_oid_list = {NULL, NULL};
- int num_db_restore = 0;
- int num_total_db;
- int n_errors_total;
- int count = 0;
- char *connected_db = NULL;
- bool dumpData = opts->dumpData;
- bool dumpSchema = opts->dumpSchema;
- bool dumpStatistics = opts->dumpSchema;
-
- /* Save db name to reuse it for all the database. */
- if (opts->cparams.dbname)
- connected_db = opts->cparams.dbname;
-
- num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
-
- /* If map.dat has no entries, return after processing global.dat */
- if (dbname_oid_list.head == NULL)
- return process_global_sql_commands(conn, dumpdirpath, opts->filename);
-
- pg_log_info(ngettext("found %d database name in \"%s\"",
- "found %d database names in \"%s\"",
- num_total_db),
- num_total_db, "map.dat");
-
- if (!conn)
- {
- pg_log_info("trying to connect to database \"%s\"", "postgres");
-
- conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
- opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
- false, progname, NULL, NULL, NULL, NULL);
-
- /* Try with template1. */
- if (!conn)
- {
- pg_log_info("trying to connect to database \"%s\"", "template1");
-
- conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
- opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
- false, progname, NULL, NULL, NULL, NULL);
- }
- }
-
- /*
- * filter the db list according to the exclude patterns
- */
- num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
- db_exclude_patterns);
-
- /* Open global.dat file and execute/append all the global sql commands. */
- n_errors_total = process_global_sql_commands(conn, dumpdirpath, opts->filename);
-
- /* Close the db connection as we are done with globals and patterns. */
- if (conn)
- PQfinish(conn);
-
- /* Exit if no db needs to be restored. */
- if (dbname_oid_list.head == NULL || num_db_restore == 0)
- {
- pg_log_info(ngettext("no database needs restoring out of %d database",
- "no database needs restoring out of %d databases", num_total_db),
- num_total_db);
- return n_errors_total;
- }
-
- pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
-
- /*
- * We have a list of databases to restore after processing the
- * exclude-database switch(es). Now we can restore them one by one.
- */
- for (SimplePtrListCell *db_cell = dbname_oid_list.head;
- db_cell; db_cell = db_cell->next)
- {
- DbOidName *dbidname = (DbOidName *) db_cell->ptr;
- char subdirpath[MAXPGPATH];
- char subdirdbpath[MAXPGPATH];
- char dbfilename[MAXPGPATH];
- int n_errors;
-
- /* ignore dbs marked for skipping */
- if (dbidname->oid == InvalidOid)
- continue;
-
- /*
- * We need to reset override_dbname so that objects can be restored
- * into an already created database. (used with -d/--dbname option)
- */
- if (opts->cparams.override_dbname)
- {
- pfree(opts->cparams.override_dbname);
- opts->cparams.override_dbname = NULL;
- }
-
- snprintf(subdirdbpath, MAXPGPATH, "%s/databases", dumpdirpath);
-
- /*
- * Look for the database dump file/dir. If there is an {oid}.tar or
- * {oid}.dmp file, use it. Otherwise try to use a directory called
- * {oid}
- */
- snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
- if (file_exists_in_directory(subdirdbpath, dbfilename))
- snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", dumpdirpath, dbidname->oid);
- else
- {
- snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
-
- if (file_exists_in_directory(subdirdbpath, dbfilename))
- snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", dumpdirpath, dbidname->oid);
- else
- snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dbidname->oid);
- }
-
- pg_log_info("restoring database \"%s\"", dbidname->str);
-
- /* If database is already created, then don't set createDB flag. */
- if (opts->cparams.dbname)
- {
- PGconn *test_conn;
-
- test_conn = ConnectDatabase(dbidname->str, NULL, opts->cparams.pghost,
- opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
- false, progname, NULL, NULL, NULL, NULL);
- if (test_conn)
- {
- PQfinish(test_conn);
-
- /* Use already created database for connection. */
- opts->createDB = 0;
- opts->cparams.dbname = dbidname->str;
- }
- else
- {
- /* we'll have to create it */
- opts->createDB = 1;
- opts->cparams.dbname = connected_db;
- }
- }
-
- /*
- * Reset flags - might have been reset in pg_backup_archiver.c by the
- * previous restore.
- */
- opts->dumpData = dumpData;
- opts->dumpSchema = dumpSchema;
- opts->dumpStatistics = dumpStatistics;
-
- /* Restore the single database. */
- n_errors = restore_one_database(subdirpath, opts, numWorkers, true, count);
-
- /* Print a summary of ignored errors during single database restore. */
- if (n_errors)
- {
- n_errors_total += n_errors;
- pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
- }
-
- count++;
- }
-
- /* Log number of processed databases. */
- pg_log_info("number of restored databases is %d", num_db_restore);
-
- /* Free dbname and dboid list. */
- simple_ptr_list_destroy(&dbname_oid_list);
-
- return n_errors_total;
-}
-
-/*
- * process_global_sql_commands
- *
- * Open global.dat and execute or copy the sql commands one by one.
- *
- * If outfile is not NULL, copy all sql commands into outfile rather than
- * executing them.
- *
- * Returns the number of errors while processing global.dat
- */
-static int
-process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
-{
- char global_file_path[MAXPGPATH];
- PGresult *result;
- StringInfoData sqlstatement,
- user_create;
- FILE *pfile;
- int n_errors = 0;
-
- snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
-
- /* Open global.dat file. */
- pfile = fopen(global_file_path, PG_BINARY_R);
-
- if (pfile == NULL)
- pg_fatal("could not open file \"%s\": %m", global_file_path);
-
- /*
- * If outfile is given, then just copy all global.dat file data into
- * outfile.
- */
- if (outfile)
- {
- copy_or_print_global_file(outfile, pfile);
- return 0;
- }
-
- /* Init sqlstatement to append commands. */
- initStringInfo(&sqlstatement);
-
- /* creation statement for our current role */
- initStringInfo(&user_create);
- appendStringInfoString(&user_create, "CREATE ROLE ");
- /* should use fmtId here, but we don't know the encoding */
- appendStringInfoString(&user_create, PQuser(conn));
- appendStringInfoChar(&user_create, ';');
-
- /* Process file till EOF and execute sql statements. */
- while (read_one_statement(&sqlstatement, pfile) != EOF)
- {
- /* don't try to create the role we are connected as */
- if (strstr(sqlstatement.data, user_create.data))
- continue;
-
- pg_log_info("executing query: %s", sqlstatement.data);
- result = PQexec(conn, sqlstatement.data);
-
- switch (PQresultStatus(result))
- {
- case PGRES_COMMAND_OK:
- case PGRES_TUPLES_OK:
- case PGRES_EMPTY_QUERY:
- break;
- default:
- n_errors++;
- pg_log_error("could not execute query: %s", PQerrorMessage(conn));
- pg_log_error_detail("Command was: %s", sqlstatement.data);
- }
- PQclear(result);
- }
-
- /* Print a summary of ignored errors during global.dat. */
- if (n_errors)
- pg_log_warning(ngettext("ignored %d error in file \"%s\"",
- "ignored %d errors in file \"%s\"", n_errors),
- n_errors, global_file_path);
- fclose(pfile);
-
- return n_errors;
-}
-
-/*
- * copy_or_print_global_file
- *
- * Copy global.dat into the output file. If "-" is used as outfile,
- * then print commands to stdout.
- */
-static void
-copy_or_print_global_file(const char *outfile, FILE *pfile)
-{
- char out_file_path[MAXPGPATH];
- FILE *OPF;
- int c;
-
- /* "-" is used for stdout. */
- if (strcmp(outfile, "-") == 0)
- OPF = stdout;
- else
- {
- snprintf(out_file_path, MAXPGPATH, "%s", outfile);
- OPF = fopen(out_file_path, PG_BINARY_W);
-
- if (OPF == NULL)
- {
- fclose(pfile);
- pg_fatal("could not open file: \"%s\"", outfile);
- }
- }
-
- /* Append global.dat into output file or print to stdout. */
- while ((c = fgetc(pfile)) != EOF)
- fputc(c, OPF);
-
- fclose(pfile);
-
- /* Close output file. */
- if (strcmp(outfile, "-") != 0)
- fclose(OPF);
-}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
index c3c5fae11ea..37d893d5e6a 100644
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,24 +237,6 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
-command_fails_like(
- [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
- qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
- 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
-);
-
-command_fails_like(
- [ 'pg_restore', '--exclude-database=foo', '-d', 'xxx', 'dumpdir' ],
- qr/\Qpg_restore: error: option --exclude-database can be used only when restoring an archive created by pg_dumpall\E/,
- 'When option --exclude-database is used in pg_restore with dump of pg_dump'
-);
-
-command_fails_like(
- [ 'pg_restore', '--globals-only', '-d', 'xxx', 'dumpdir' ],
- qr/\Qpg_restore: error: option -g\/--globals-only can be used only when restoring an archive created by pg_dumpall\E/,
- 'When option --globals-only is not used in pg_restore with dump of pg_dump'
-);
-
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -262,8 +244,4 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
-command_fails_like(
- [ 'pg_dumpall', '--format', 'x' ],
- qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
- 'pg_dumpall: unrecognized output format');
done_testing();
diff --git a/src/bin/pg_dump/t/006_pg_dumpall.pl b/src/bin/pg_dump/t/006_pg_dumpall.pl
deleted file mode 100644
index c274b777586..00000000000
--- a/src/bin/pg_dump/t/006_pg_dumpall.pl
+++ /dev/null
@@ -1,400 +0,0 @@
-# Copyright (c) 2021-2025, PostgreSQL Global Development Group
-
-use strict;
-use warnings FATAL => 'all';
-
-use PostgreSQL::Test::Cluster;
-use PostgreSQL::Test::Utils;
-use Test::More;
-
-my $tempdir = PostgreSQL::Test::Utils::tempdir;
-my $run_db = 'postgres';
-my $sep = $windows_os ? "\\" : "/";
-
-# Tablespace locations used by "restore_tablespace" test case.
-my $tablespace1 = "${tempdir}${sep}tbl1";
-my $tablespace2 = "${tempdir}${sep}tbl2";
-mkdir($tablespace1) || die "mkdir $tablespace1 $!";
-mkdir($tablespace2) || die "mkdir $tablespace2 $!";
-
-# Scape tablespace locations on Windows.
-$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
-$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
-
-# Where pg_dumpall will be executed.
-my $node = PostgreSQL::Test::Cluster->new('node');
-$node->init;
-$node->start;
-
-
-###############################################################
-# Definition of the pg_dumpall test cases to run.
-#
-# Each of these test cases are named and those names are used for fail
-# reporting and also to save the dump and restore information needed for the
-# test to assert.
-#
-# The "setup_sql" is a psql valid script that contains SQL commands to execute
-# before of actually execute the tests. The setups are all executed before of
-# any test execution.
-#
-# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
-# "restore_cmd" must have the --file flag to save the restore output so that we
-# can assert on it.
-#
-# The "like" and "unlike" is a regexp that is used to match the pg_restore
-# output. It must have at least one of then filled per test cases but it also
-# can have both. See "excluding_databases" test case for example.
-my %pgdumpall_runs = (
- restore_roles => {
- setup_sql => '
- CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
- CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_roles",
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_roles.sql",
- "$tempdir/restore_roles",
- ],
- like => qr/
- ^\s*\QCREATE ROLE dumpall;\E\s*\n
- \s*\QALTER ROLE dumpall WITH SUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN NOREPLICATION NOBYPASSRLS PASSWORD 'SCRAM-SHA-256\E
- [^']+';\s*\n
- \s*\QCREATE ROLE dumpall2;\E
- \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
- /xm
- },
-
- restore_tablespace => {
- setup_sql => "
- CREATE ROLE tap;
- CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
- CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_tablespace",
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_tablespace.sql",
- "$tempdir/restore_tablespace",
- ],
- # Match "E" as optional since it is added on LOCATION when running on
- # Windows.
- like => qr/^
- \n\QCREATE TABLESPACE tbl1 OWNER tap LOCATION \E(?:E)?\Q'$tablespace1';\E
- \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
- \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
- /xm,
- },
-
- restore_grants => {
- setup_sql => "
- CREATE DATABASE tapgrantsdb;
- CREATE SCHEMA private;
- CREATE SEQUENCE serial START 101;
- CREATE FUNCTION fn() RETURNS void AS \$\$
- BEGIN
- END;
- \$\$ LANGUAGE plpgsql;
- CREATE ROLE super;
- CREATE ROLE grant1;
- CREATE ROLE grant2;
- CREATE ROLE grant3;
- CREATE ROLE grant4;
- CREATE ROLE grant5;
- CREATE ROLE grant6;
- CREATE ROLE grant7;
- CREATE ROLE grant8;
-
- CREATE TABLE t (id int);
- INSERT INTO t VALUES (1), (2), (3), (4);
-
- GRANT SELECT ON TABLE t TO grant1;
- GRANT INSERT ON TABLE t TO grant2;
- GRANT ALL PRIVILEGES ON TABLE t to grant3;
- GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
- GRANT USAGE, CREATE ON SCHEMA private TO grant5;
- GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
- GRANT super TO grant7;
- GRANT EXECUTE ON FUNCTION fn() TO grant8;
- ",
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_grants",
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_grants.sql",
- "$tempdir/restore_grants",
- ],
- like => qr/^
- \n\QGRANT super TO grant7 WITH INHERIT TRUE GRANTED BY\E
- (.*\n)*
- \n\QGRANT ALL ON SCHEMA private TO grant5;\E
- (.*\n)*
- \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
- (.*\n)*
- \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
- (.*\n)*
- \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
- \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
- \n\QGRANT ALL ON TABLE public.t TO grant3;\E
- (.*\n)*
- \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
- /xm,
- },
-
- excluding_databases => {
- setup_sql => 'CREATE DATABASE db1;
- \c db1
- CREATE TABLE t1 (id int);
- INSERT INTO t1 VALUES (1), (2), (3), (4);
- CREATE TABLE t2 (id int);
- INSERT INTO t2 VALUES (1), (2), (3), (4);
-
- CREATE DATABASE db2;
- \c db2
- CREATE TABLE t3 (id int);
- INSERT INTO t3 VALUES (1), (2), (3), (4);
- CREATE TABLE t4 (id int);
- INSERT INTO t4 VALUES (1), (2), (3), (4);
-
- CREATE DATABASE dbex3;
- \c dbex3
- CREATE TABLE t5 (id int);
- INSERT INTO t5 VALUES (1), (2), (3), (4);
- CREATE TABLE t6 (id int);
- INSERT INTO t6 VALUES (1), (2), (3), (4);
-
- CREATE DATABASE dbex4;
- \c dbex4
- CREATE TABLE t7 (id int);
- INSERT INTO t7 VALUES (1), (2), (3), (4);
- CREATE TABLE t8 (id int);
- INSERT INTO t8 VALUES (1), (2), (3), (4);
-
- CREATE DATABASE db5;
- \c db5
- CREATE TABLE t9 (id int);
- INSERT INTO t9 VALUES (1), (2), (3), (4);
- CREATE TABLE t10 (id int);
- INSERT INTO t10 VALUES (1), (2), (3), (4);
- ',
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'directory',
- '--file' => "$tempdir/excluding_databases",
- '--exclude-database' => 'dbex*',
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'directory',
- '--file' => "$tempdir/excluding_databases.sql",
- '--exclude-database' => 'db5',
- "$tempdir/excluding_databases",
- ],
- like => qr/^
- \n\QCREATE DATABASE db1\E
- (.*\n)*
- \n\QCREATE TABLE public.t1 (\E
- (.*\n)*
- \n\QCREATE TABLE public.t2 (\E
- (.*\n)*
- \n\QCREATE DATABASE db2\E
- (.*\n)*
- \n\QCREATE TABLE public.t3 (\E
- (.*\n)*
- \n\QCREATE TABLE public.t4 (/xm,
- unlike => qr/^
- \n\QCREATE DATABASE db3\E
- (.*\n)*
- \n\QCREATE TABLE public.t5 (\E
- (.*\n)*
- \n\QCREATE TABLE public.t6 (\E
- (.*\n)*
- \n\QCREATE DATABASE db4\E
- (.*\n)*
- \n\QCREATE TABLE public.t7 (\E
- (.*\n)*
- \n\QCREATE TABLE public.t8 (\E
- \n\QCREATE DATABASE db5\E
- (.*\n)*
- \n\QCREATE TABLE public.t9 (\E
- (.*\n)*
- \n\QCREATE TABLE public.t10 (\E
- /xm,
- },
-
- format_directory => {
- setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
- INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'directory',
- '--file' => "$tempdir/format_directory",
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'directory',
- '--file' => "$tempdir/format_directory.sql",
- "$tempdir/format_directory",
- ],
- like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
- },
-
- format_tar => {
- setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
- INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'tar',
- '--file' => "$tempdir/format_tar",
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'tar',
- '--file' => "$tempdir/format_tar.sql",
- "$tempdir/format_tar",
- ],
- like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
- },
-
- format_custom => {
- setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
- INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'custom',
- '--file' => "$tempdir/format_custom",
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'custom',
- '--file' => "$tempdir/format_custom.sql",
- "$tempdir/format_custom",
- ],
- like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
- },
-
- dump_globals_only => {
- setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
- INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'directory',
- '--globals-only',
- '--file' => "$tempdir/dump_globals_only",
- ],
- restore_cmd => [
- 'pg_restore', '-C', '--globals-only',
- '--format' => 'directory',
- '--file' => "$tempdir/dump_globals_only.sql",
- "$tempdir/dump_globals_only",
- ],
- like => qr/
- ^\s*\QCREATE ROLE dumpall;\E\s*\n
- /xm
- },);
-
-# First execute the setup_sql
-foreach my $run (sort keys %pgdumpall_runs)
-{
- if ($pgdumpall_runs{$run}->{setup_sql})
- {
- $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
- }
-}
-
-# Execute the tests
-foreach my $run (sort keys %pgdumpall_runs)
-{
- # Create a new target cluster to pg_restore each test case run so that we
- # don't need to take care of the cleanup from the target cluster after each
- # run.
- my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
- $target_node->init;
- $target_node->start;
-
- # Dumpall from node cluster.
- $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
- "$run: pg_dumpall runs");
-
- # Restore the dump on "target_node" cluster.
- my @restore_cmd = (
- @{ $pgdumpall_runs{$run}->{restore_cmd} },
- '--host', $target_node->host, '--port', $target_node->port);
-
- my ($stdout, $stderr) = run_command(\@restore_cmd);
-
- # pg_restore --file output file.
- my $output_file = slurp_file("$tempdir/${run}.sql");
-
- if ( !($pgdumpall_runs{$run}->{like})
- && !($pgdumpall_runs{$run}->{unlike}))
- {
- die "missing \"like\" or \"unlike\" in test \"$run\"";
- }
-
- if ($pgdumpall_runs{$run}->{like})
- {
- like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
- }
-
- if ($pgdumpall_runs{$run}->{unlike})
- {
- unlike(
- $output_file,
- $pgdumpall_runs{$run}->{unlike},
- "should not dump $run");
- }
-}
-
-# Some negative test case with dump of pg_dumpall and restore using pg_restore
-# test case 1: when -C is not used in pg_restore with dump of pg_dumpall
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom",
- '--format' => 'custom',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
- 'When -C is not used in pg_restore with dump of pg_dumpall');
-
-# test case 2: When --list option is used with dump of pg_dumpall
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--list',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
- 'When --list is used in pg_restore with dump of pg_dumpall');
-
-# test case 3: When non-exist database is given with -d option
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '-d' => 'dbpq',
- ],
- qr/\Qpg_restore: error: could not connect to database "dbpq"\E/,
- 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
-);
-
-$node->stop('fast');
-
-done_testing();
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-29 20:34 Noah Misch <[email protected]>
parent: Andrew Dunstan <[email protected]>
1 sibling, 1 reply; 65+ messages in thread
From: Noah Misch @ 2025-07-29 20:34 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Tom Lane <[email protected]>; Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Tue, Jul 29, 2025 at 04:09:13PM -0400, Andrew Dunstan wrote:
> here's a reversion patch for master.
> This reverts parts or all of the following commits:
I briefly looked through this. The biggest non-reverted part is, I think,
c1da728 "Move common pg_dump code related to connections to a new file".
Refraining from a revert of that one is defensible.
> dec6643487b Improve pg_dump/pg_dumpall help synopses and terminology
> @@ -1276,7 +1276,7 @@ main(int argc, char **argv)
> static void
> help(const char *progname)
> {
> - printf(_("%s exports a PostgreSQL database as an SQL script or to other formats.\n\n"), progname);
> + printf(_("%s dumps a database as a text file or to other formats.\n\n"), progname);
> printf(_("Usage:\n"));
> printf(_(" %s [OPTION]... [DBNAME]\n"), progname);
I think commit dec6643487b, which e.g. decided to standardize on the term
"export" for these programs, was independent of $SUBJECT.
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-30 01:07 Andrew Dunstan <[email protected]>
parent: Noah Misch <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Andrew Dunstan @ 2025-07-30 01:07 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Tom Lane <[email protected]>; Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-07-29 Tu 4:34 PM, Noah Misch wrote:
> On Tue, Jul 29, 2025 at 04:09:13PM -0400, Andrew Dunstan wrote:
>> here's a reversion patch for master.
>> This reverts parts or all of the following commits:
> I briefly looked through this. The biggest non-reverted part is, I think,
> c1da728 "Move common pg_dump code related to connections to a new file".
> Refraining from a revert of that one is defensible.
Yes, that was deliberate, since we intend to use it in the same way when
we redo this.
>
>> dec6643487b Improve pg_dump/pg_dumpall help synopses and terminology
>> @@ -1276,7 +1276,7 @@ main(int argc, char **argv)
>> static void
>> help(const char *progname)
>> {
>> - printf(_("%s exports a PostgreSQL database as an SQL script or to other formats.\n\n"), progname);
>> + printf(_("%s dumps a database as a text file or to other formats.\n\n"), progname);
>> printf(_("Usage:\n"));
>> printf(_(" %s [OPTION]... [DBNAME]\n"), progname);
> I think commit dec6643487b, which e.g. decided to standardize on the term
> "export" for these programs, was independent of $SUBJECT.
OK, thanks for looking. Will fix.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-30 18:51 Andrew Dunstan <[email protected]>
parent: Andrew Dunstan <[email protected]>
1 sibling, 3 replies; 65+ messages in thread
From: Andrew Dunstan @ 2025-07-30 18:51 UTC (permalink / raw)
To: Noah Misch <[email protected]>; Tom Lane <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-07-29 Tu 4:09 PM, Andrew Dunstan wrote:
>
> On 2025-07-28 Mo 8:04 AM, Andrew Dunstan wrote:
>>
>> On 2025-07-27 Su 7:56 PM, Noah Misch wrote:
>>> On Fri, Jul 25, 2025 at 04:59:29PM -0400, Tom Lane wrote:
>>>> Andrew Dunstan <[email protected]> writes:
>>>>> Before we throw the baby out with the bathwater, how about this
>>>>> suggestion? pg_dumpall would continue to produce globals.dat, but it
>>>>> wouldn't be processed by pg_restore, which would only restore the
>>>>> individual databases. Or else we just don't produce globals.dat at
>>>>> all.
>>>>> Then we could introduce a structured object that pg_restore could
>>>>> safely
>>>>> use for release 19, and I think we'd still have something useful for
>>>>> release 18.
>>>> I dunno ... that seems like a pretty weird behavior. People would
>>>> have to do a separate text-mode "pg_dumpall -g" and remember to
>>>> restore that too. Admittedly, this could be more convenient than
>>>> "pg_dumpall -g" plus separately pg_dump'ing each database, which is
>>>> what people have to do today if they want anything smarter than a flat
>>>> text dumpfile. But it still seems like a hack --- and it would not be
>>>> compatible with v19, where presumably "pg_dumpall | pg_restore"
>>>> *would* restore globals. I think that the prospect of changing
>>>> dump/restore scripts and then having to change them again in v19
>>>> isn't too appetizing.
>>> +1
>>
>>
>> OK, got it. Will revert.
>>
>>
>>
>
> here's a reversion patch for master. It applies cleanly to release 18
> as well. Thanks to Mahendra Singh Thalor for helping me sanity check
> it (Any issues are of course my responsibility)
>
>
> I'll work on pulling the entry out of the release notes.
>
>
>
OK, now that's reverted we should discuss how to proceed. I had two
thoughts - we could use invent a JSON format for the globals, or we
could just use the existing archive format. I think the archive format
is pretty flexible, and should be able to accommodate this. The downside
is it's not humanly readable. The upside is that we don't need to do
anything special either to write it or parse it.
There might also be other reasonable options. But I think we should stay
out of the business of using custom code to parse text.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-31 09:44 Christoph Berg <[email protected]>
parent: Andrew Dunstan <[email protected]>
2 siblings, 1 reply; 65+ messages in thread
From: Christoph Berg @ 2025-07-31 09:44 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Noah Misch <[email protected]>; Tom Lane <[email protected]>; Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
The 18 branch is broken for apt.pg.o:
00:54:18 # Failed test 'dump_globals_only: pg_dumpall runs'
00:54:18 # at t/006_pg_dumpall.pl line 329.
00:54:18 # Tests were run but no plan was declared and done_testing() was not seen.
00:54:18 # Looks like your test exited with 2 just after 1.
00:54:18 t/006_pg_dumpall.pl ...........
00:54:18 # initializing database system by copying initdb template
00:54:18 # initializing database system by copying initdb template
00:54:18 not ok 1 - dump_globals_only: pg_dumpall runs
00:54:18 Dubious, test returned 2 (wstat 512, 0x200)
00:54:18 Failed 1/1 subtests
Devel is ok.
Christoph
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-31 13:40 Andrew Dunstan <[email protected]>
parent: Christoph Berg <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Andrew Dunstan @ 2025-07-31 13:40 UTC (permalink / raw)
To: Christoph Berg <[email protected]>; +Cc: Noah Misch <[email protected]>; Tom Lane <[email protected]>; Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-07-31 Th 5:44 AM, Christoph Berg wrote:
> The 18 branch is broken for apt.pg.o:
>
> 00:54:18 # Failed test 'dump_globals_only: pg_dumpall runs'
> 00:54:18 # at t/006_pg_dumpall.pl line 329.
> 00:54:18 # Tests were run but no plan was declared and done_testing() was not seen.
> 00:54:18 # Looks like your test exited with 2 just after 1.
> 00:54:18 t/006_pg_dumpall.pl ...........
> 00:54:18 # initializing database system by copying initdb template
> 00:54:18 # initializing database system by copying initdb template
> 00:54:18 not ok 1 - dump_globals_only: pg_dumpall runs
> 00:54:18 Dubious, test returned 2 (wstat 512, 0x200)
> 00:54:18 Failed 1/1 subtests
>
> Devel is ok.
>
That file was deleted by the revert. Maybe you have a cache problem?
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-31 13:52 Christoph Berg <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Christoph Berg @ 2025-07-31 13:52 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Noah Misch <[email protected]>; Tom Lane <[email protected]>; Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Re: Andrew Dunstan
> That file was deleted by the revert. Maybe you have a cache problem?
Oh right. This was caused by our snapshot builds using the latest
tarball (if available) and putting a patch on top that. I've now
bumped the upstream version to 18~beta3, this should avoid the
problem.
Sorry for the noise, and thanks for the pointer!
Christoph
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-31 18:22 Nathan Bossart <[email protected]>
parent: Andrew Dunstan <[email protected]>
2 siblings, 1 reply; 65+ messages in thread
From: Nathan Bossart @ 2025-07-31 18:22 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Noah Misch <[email protected]>; Tom Lane <[email protected]>; Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
> OK, now that's reverted...
Can we close the open item for this one now? Or is there something else
remaining?
--
nathan
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-07-31 19:06 Andrew Dunstan <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Andrew Dunstan @ 2025-07-31 19:06 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Noah Misch <[email protected]>; Tom Lane <[email protected]>; Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-07-31 Th 2:22 PM, Nathan Bossart wrote:
> On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
>> OK, now that's reverted...
> Can we close the open item for this one now? Or is there something else
> remaining?
>
Thanks for the reminder. I have marked the item as fixed.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-08-24 01:08 Noah Misch <[email protected]>
parent: Andrew Dunstan <[email protected]>
2 siblings, 1 reply; 65+ messages in thread
From: Noah Misch @ 2025-08-24 01:08 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Tom Lane <[email protected]>; Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
> OK, now that's reverted we should discuss how to proceed. I had two thoughts
> - we could use invent a JSON format for the globals, or we could just use
> the existing archive format. I think the archive format is pretty flexible,
> and should be able to accommodate this. The downside is it's not humanly
> readable. The upside is that we don't need to do anything special either to
> write it or parse it.
I would first try to use the existing archiver API, because that makes it
harder to miss bugs. Any tension between that API and pg_dumpall is likely to
have corresponding tension on the pg_restore side. Resolving that tension
will reveal much of the project's scope that remained hidden during the v18
attempt. Perhaps more important than that, using the archiver API means
future pg_dump and pg_restore options are more likely to cooperate properly
with $SUBJECT. In other words, I want it to be hard to add pg_dump/pg_restore
features that malfunction only for $SUBJECT archives. The strength of the
archiver architecture shows in how rarely new features need format-specific
logic and how rarely format-specific bugs get reported. We've had little or
no trouble with e.g. bugs that appear in -Fd but not in -Fc.
If pg_backup_json.c emerged as a new backend to the archiver API, I'd not have
concerns about that. But a JSON format specific to $SUBJECT sounds like a
recipe for bugs.
> There might also be other reasonable options. But I think we should stay out
> of the business of using custom code to parse text.
Agreed.
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-08-24 16:42 Andrew Dunstan <[email protected]>
parent: Noah Misch <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Andrew Dunstan @ 2025-08-24 16:42 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: Tom Lane <[email protected]>; Mahendra Singh Thalor <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-08-23 Sa 9:08 PM, Noah Misch wrote:
> On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
>> OK, now that's reverted we should discuss how to proceed. I had two thoughts
>> - we could use invent a JSON format for the globals, or we could just use
>> the existing archive format. I think the archive format is pretty flexible,
>> and should be able to accommodate this. The downside is it's not humanly
>> readable. The upside is that we don't need to do anything special either to
>> write it or parse it.
> I would first try to use the existing archiver API, because that makes it
> harder to miss bugs. Any tension between that API and pg_dumpall is likely to
> have corresponding tension on the pg_restore side. Resolving that tension
> will reveal much of the project's scope that remained hidden during the v18
> attempt. Perhaps more important than that, using the archiver API means
> future pg_dump and pg_restore options are more likely to cooperate properly
> with $SUBJECT. In other words, I want it to be hard to add pg_dump/pg_restore
> features that malfunction only for $SUBJECT archives. The strength of the
> archiver architecture shows in how rarely new features need format-specific
> logic and how rarely format-specific bugs get reported. We've had little or
> no trouble with e.g. bugs that appear in -Fd but not in -Fc.
Yeah, that's what we're going to try.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-10-15 17:35 Mahendra Singh Thalor <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-10-15 17:35 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; tushar <[email protected]>; +Cc: Noah Misch <[email protected]>; Tom Lane <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Sun, 24 Aug 2025 at 22:12, Andrew Dunstan <[email protected]> wrote:
>
>
> On 2025-08-23 Sa 9:08 PM, Noah Misch wrote:
>
> On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
>
> OK, now that's reverted we should discuss how to proceed. I had two thoughts
> - we could use invent a JSON format for the globals, or we could just use
> the existing archive format. I think the archive format is pretty flexible,
> and should be able to accommodate this. The downside is it's not humanly
> readable. The upside is that we don't need to do anything special either to
> write it or parse it.
>
> I would first try to use the existing archiver API, because that makes it
> harder to miss bugs. Any tension between that API and pg_dumpall is likely to
> have corresponding tension on the pg_restore side. Resolving that tension
> will reveal much of the project's scope that remained hidden during the v18
> attempt. Perhaps more important than that, using the archiver API means
> future pg_dump and pg_restore options are more likely to cooperate properly
> with $SUBJECT. In other words, I want it to be hard to add pg_dump/pg_restore
> features that malfunction only for $SUBJECT archives. The strength of the
> archiver architecture shows in how rarely new features need format-specific
> logic and how rarely format-specific bugs get reported. We've had little or
> no trouble with e.g. bugs that appear in -Fd but not in -Fc.
>
>
> Yeah, that's what we're going to try.
>
>
> cheers
>
>
> andrew
>
> --
> Andrew Dunstan
> EDB: https://www.enterprisedb.com
Thanks Andrew, Noah and all others for feedback.
Based on the above suggestions and discussions, I removed sql commands
from the global.dat file. For global commands, now we are making
toc.dat/toc.dmp/toc.tar file based on format specified and based on
format specified, we are making archive entries for these global
commands. By this approach, we removed the hard-coded parsing part of
the global.dat file and we are able to skip DROP DATABASE with the
globals-only option.
Here, I am attaching a patch for review, testing and feedback. This is
a WIP patch. I will do some more code cleanup and will add some more
comments also. Please review this and let me know design level
feedback. Thanks Tushar Ahuja for some internal testing and feedback.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v01-15102025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (85.1K, 2-v01-15102025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
From 09cc2e11a4b8639ee81528363a9f134cb7fc78c5 Mon Sep 17 00:00:00 2001
From: ThalorMahendra <[email protected]>
Date: Wed, 15 Oct 2025 22:38:14 +0530
Subject: [PATCH] Non text modes for pg_dumpall, correspondingly change
pg_restore
pg_dumpall acquires a new -F/--format option, with the same meanings as
pg_dump. The default is p, meaning plain text. For any other value, a
directory is created containing two files, toc.dat/.dmp/.tar and map.dat. The
first contains commands restoring the global data based on -F, and the second
contains a map from oids to database names. It will also contain a
subdirectory called databases, inside which it will create archives in
the specified format, named using the database oids.
In these casess the -f argument is required.
If pg_restore encounters a directory containing map.dat,
it restores the global settings from toc.dat/.dmp/.tar if exist, and then
restores each database.
pg_restore acquires two new options: -g/--globals-only which suppresses
restoration of any databases, and --exclude-database which inhibits
restoration of particualr database(s) in the same way the same option
works in pg_dumpall.
---
doc/src/sgml/ref/pg_dumpall.sgml | 89 +++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/connectdb.c | 1 -
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 10 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 23 +-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 599 ++++++++++++++++++++++-----
src/bin/pg_dump/pg_restore.c | 599 ++++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 22 +
src/bin/pg_dump/t/006_pg_dumpall.pl | 400 ++++++++++++++++++
14 files changed, 1671 insertions(+), 146 deletions(-)
create mode 100644 src/bin/pg_dump/t/006_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 9f639f61db0..73e166062b9 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,10 +139,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>global.dat</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index a468a38361a..7497b527ae6 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -591,6 +615,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
index d55d53dbeea..f44a8a45fca 100644
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -287,7 +287,6 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
exit_nicely(1);
}
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index a2233b0a1b4..4a4ebbd8ec9 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -102,6 +102,7 @@ tests += {
't/003_pg_dump_with_server.pl',
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
+ 't/006_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..5974d6706fd 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index d9041dad720..f631d945472 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -312,7 +312,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, bool globals_only);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 59eaecb4ed7..d378c7b601e 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, bool globals_only)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,9 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ if (globals_only && te && te->tag && (strcmp(te->tag, "DROP_DATABASE") == 0))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1324,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1695,7 +1703,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1724,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..818b80a9369 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 641bece12c7..857f1cc7948 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index bb451c1bae1..c9e8c15b721 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -30,6 +30,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -65,9 +66,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,6 +78,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpIdLocal(void);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -123,6 +127,13 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static const CatalogId nilCatalogId = {0, 0};
+static ArchiveMode archiveMode = archModeWrite;
+static DataDirSyncMethod sync_method = DATA_DIR_SYNC_METHOD_FSYNC;
+static ArchiveFormat archDumpFormat = archNull;
int
main(int argc, char *argv[])
@@ -148,6 +159,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +209,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -208,6 +221,8 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
+ char global_path[MAXPGPATH];
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -246,7 +261,9 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ InitDumpOptions(&dopt);
+
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -257,6 +274,7 @@ main(int argc, char *argv[])
case 'c':
output_clean = true;
+ dopt.outputClean = 1;
break;
case 'd':
@@ -274,7 +292,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -314,6 +334,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -429,6 +450,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -489,6 +525,33 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (filename && archDumpFormat != archNull)
+ {
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ /* set file path for global sql commands. */
+ if (archDumpFormat == archCustom)
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", filename);
+ else if (archDumpFormat == archTar)
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", filename);
+ else if (archDumpFormat == archDirectory)
+ snprintf(global_path, MAXPGPATH, "%s", filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -538,19 +601,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -585,37 +635,123 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
if (verbose)
dumpTimestamp("Started on");
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ /* create a archive file for global commands. */
+ if (filename && archDumpFormat != archNull)
+ {
+ /* Open the output file */
+ fout = CreateArchive(global_path, archDumpFormat, compression_spec,
+ dosync, archiveMode, NULL, sync_method);
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+ ((ArchiveHandle * )fout) ->connection = conn;
+ ((ArchiveHandle * ) fout) -> public.numWorkers = 1;
- /* Replicate encoding and std_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
- if (strcmp(std_strings, "off") == 0)
- fprintf(OPF, "SET escape_string_warning = off;\n");
- fprintf(OPF, "\n");
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
+
+ /* dumpEncoding: put the correct encoding into the archive */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ const char *encname = pg_encoding_to_char(encoding);
+
+ pg_log_info("saving encoding = %s", encname);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+
+ ArchiveEntry(fout, nilCatalogId, createDumpIdLocal(),
+ ARCHIVE_OPTS(.tag = "ENCODING",
+ .description = "ENCODING",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpStdStrings: put the correct escape string behavior into the archive */
+ {
+ const char *stdstrings = std_strings ? "on" : "off";
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving \"standard_conforming_strings = %s\"", stdstrings);
+
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = '%s';\n",
+ stdstrings);
+
+ ArchiveEntry(fout, nilCatalogId, createDumpIdLocal(),
+ ARCHIVE_OPTS(.tag = "STDSTRINGS",
+ .description = "STDSTRINGS",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+
+ destroyPQExpBuffer(qry);
+ }
+
+ /* default_transaction_read_only = off */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving default_transaction_read_only = off");
+
+ appendPQExpBuffer(qry, "SET default_transaction_read_only = off;\n");
+
+ ArchiveEntry(fout, nilCatalogId, createDumpIdLocal(),
+ ARCHIVE_OPTS(.tag = "DEFAULT_TRANSACTION_READ_ONLY",
+ .description = "DEFAULT_TRANSACTION_READ_ONLY",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+
+ destroyPQExpBuffer(qry);
+ }
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and std_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
+ if (strcmp(std_strings, "off") == 0)
+ fprintf(OPF, "SET escape_string_warning = off;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -659,27 +795,41 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump will
+ * handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
-
- PQfinish(conn);
+ dumpDatabases(conn, archDumpFormat);
if (verbose)
dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (filename)
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+
+ if (filename && archDumpFormat != archNull)
+ {
+ RestoreOptions *ropt;
+
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
+
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
+ }
+ else if (filename)
{
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -690,12 +840,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -770,6 +922,7 @@ static void
dropRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer delQry = createPQExpBuffer();
PGresult *res;
int i_rolname;
int i;
@@ -790,7 +943,7 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Drop roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -799,15 +952,31 @@ dropRoles(PGconn *conn)
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(rolename));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dropRoles",
+ //.owner = dba,
+ .description = "dropRoles_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ //.dropStmt = delQry->data));
+ }
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -888,7 +1057,7 @@ dumpRoles(PGconn *conn)
i_rolcomment = PQfnumber(res, "rolcomment");
i_is_current_user = PQfnumber(res, "is_current_user");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -993,7 +1162,25 @@ dumpRoles(PGconn *conn)
"ROLE", rolename,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ {
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
+ if_exists ? "IF EXISTS " : "", fmtId(rolename));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpRoles",
+ //.owner = dba,
+ .description = "dumpRoles_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ //.dropStmt = delQry->data));
+ }
}
/*
@@ -1001,15 +1188,13 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
}
@@ -1087,7 +1272,7 @@ dumpRoleMembership(PGconn *conn)
i_inherit_option = PQfnumber(res, "inherit_option");
i_set_option = PQfnumber(res, "set_option");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Role memberships\n--\n\n");
/*
@@ -1167,6 +1352,7 @@ dumpRoleMembership(PGconn *conn)
char *grantor;
char *set_option = "true";
bool found;
+ PQExpBuffer creaQry = createPQExpBuffer();
/* If we already did this grant, don't do it again. */
if (done[i - start])
@@ -1223,8 +1409,8 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ appendPQExpBuffer(creaQry, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(creaQry, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1245,10 +1431,24 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(creaQry, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(creaQry, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(creaQry, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", creaQry->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpRoleMembership",
+ //.owner = dba,
+ .description = "dumpRoleMembership_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = creaQry->data));
+ }
}
}
@@ -1260,7 +1460,8 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1287,7 +1488,7 @@ dumpRoleGUCPrivs(PGconn *conn)
"FROM pg_catalog.pg_parameter_acl "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1312,14 +1513,28 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpRoleGUCPrivs",
+ //.owner = dba,
+ .description = "dumpRoleGUCPrivs_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ }
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1331,6 +1546,7 @@ dropTablespaces(PGconn *conn)
{
PGresult *res;
int i;
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Get all tablespaces except built-in ones (which we assume are named
@@ -1341,21 +1557,37 @@ dropTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(spcname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dropTablespaces",
+ //.owner = dba,
+ .description = "dropTablespaces_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ // .dropStmt = delQry->data));
+ }
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1381,7 +1613,7 @@ dumpTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1451,7 +1683,25 @@ dumpTablespaces(PGconn *conn)
"TABLESPACE", spcname,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ {
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
+ if_exists ? "IF EXISTS " : "", fmtId(fspcname));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpTablespaces",
+ //.owner = dba,
+ .description = "dumpTablespaces_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ //.dropStmt = delQry->data));
+ }
free(fspcname);
destroyPQExpBuffer(buf);
@@ -1481,7 +1731,7 @@ dropDBs(PGconn *conn)
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY datname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1497,9 +1747,26 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(dbname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(),
+ ARCHIVE_OPTS(.tag = "DROP_DATABASE",
+ //.owner = dba,
+ .description = "DROP_DATABASE_COMMANDS",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ //.dropStmt = delQry->data));
+ }
}
}
@@ -1517,6 +1784,7 @@ dumpUserConfig(PGconn *conn, const char *username)
{
PQExpBuffer buf = createPQExpBuffer();
PGresult *res;
+ static bool header_done = false;
printfPQExpBuffer(buf, "SELECT unnest(setconfig) FROM pg_db_role_setting "
"WHERE setdatabase = 0 AND setrole = "
@@ -1532,7 +1800,9 @@ dumpUserConfig(PGconn *conn, const char *username)
char *sanitized;
sanitized = sanitize_line(username, true);
- fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ if (!header_done && (archDumpFormat == archNull))
+ fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ header_done = true;
free(sanitized);
}
@@ -1542,7 +1812,19 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpUserConfig",
+ //.owner = dba,
+ .description = "dumpUserConfig_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ }
}
PQclear(res);
@@ -1608,10 +1890,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1625,19 +1910,43 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
- if (PQntuples(res) > 0)
+ if (archDumpFormat == archNull && PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1651,12 +1960,15 @@ dumpDatabases(PGconn *conn)
continue;
}
- pg_log_info("dumping database \"%s\"", dbname);
+ if (archDumpFormat == archNull)
+ pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
free(sanitized);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
/*
* We assume that "template1" and "postgres" already exist in the
* target installation. dropDBs() won't have removed them, for fear
@@ -1669,24 +1981,38 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
- else
- {
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
fprintf(OPF, "\\connect %s\n\n", dbname);
- }
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1695,6 +2021,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1704,7 +2034,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1713,17 +2044,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1868,3 +2218,42 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+static int
+createDumpIdLocal(void)
+{
+ return ++dumpIdVal;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c9776306c5c..1334d4fdc1e 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,31 +41,60 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num,
+ bool globals_only);
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only);
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +118,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +172,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +201,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +228,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -316,6 +350,9 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
case 6:
opts->restrict_key = pg_strdup(optarg);
@@ -347,6 +384,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -472,6 +516,111 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If map.dat file is present, then restore all the
+ * databases from map.dat , but skip restoring those matching
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "map.dat") ||
+ file_exists_in_directory(inputFileSpec, "toc.tar") ||
+ file_exists_in_directory(inputFileSpec, "toc.dmp")))
+ {
+ char global_path[MAXPGPATH];
+
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring an archive created by pg_dumpall");
+ else if (opts->tocFile)
+ pg_fatal("option -L/--use-list cannot be used when restoring an archive created by pg_dumpall");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option -C/--create must be specified when restoring an archive created by pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ n_errors = restore_global_objects(global_path, opts, numWorkers, false, 0, globals_only);
+
+ pg_log_info("database restoring skipped because option -g/--globals-only was specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else /* process if map.dat file does not exist. */
+ {
+ if (db_exclude_patterns.head != NULL)
+ pg_fatal("option --exclude-database can be used only when restoring an archive created by pg_dumpall");
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring an archive created by pg_dumpall");
+
+ n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, 0, globals_only);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ *
+ * If globals_only is set, then skip DROP DATABASE commands from restore.
+ */
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ return restore_one_database(inputFileSpec, opts, numWorkers, append_data, num, globals_only);
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -479,9 +628,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -501,25 +656,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, globals_only);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -537,6 +688,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -553,6 +705,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -588,8 +741,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -694,3 +847,415 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn && db_exclude_patterns.head != NULL)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no database connection while doing pg_restore");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+ PQExpBuffer db_lit = createPQExpBuffer();
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches exclude pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ destroyPQExpBuffer(db_lit);
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ destroyPQExpBuffer(query);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+
+
+ /*
+ * If there is only global.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen <= 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbname, db_oid, map_file_path);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+ PGconn *conn = NULL;
+ char global_path[MAXPGPATH];
+
+ /* Based on file, set path. */
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
+
+ /* If map.dat has no entries, return after processing global commands. */
+ if (dbname_oid_list.head == NULL)
+ return restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Open toc.dat file and execute/append all the global sql commands. */
+ n_errors_total = restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ count++;
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
index 37d893d5e6a..c3c5fae11ea 100644
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,24 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option --exclude-database can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --exclude-database is used in pg_restore with dump of pg_dump'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--globals-only', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option -g\/--globals-only can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --globals-only is not used in pg_restore with dump of pg_dump'
+);
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +262,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
done_testing();
diff --git a/src/bin/pg_dump/t/006_pg_dumpall.pl b/src/bin/pg_dump/t/006_pg_dumpall.pl
new file mode 100644
index 00000000000..c274b777586
--- /dev/null
+++ b/src/bin/pg_dump/t/006_pg_dumpall.pl
@@ -0,0 +1,400 @@
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# Scape tablespace locations on Windows.
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ \s*\QALTER ROLE dumpall WITH SUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN NOREPLICATION NOBYPASSRLS PASSWORD 'SCRAM-SHA-256\E
+ [^']+';\s*\n
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl1 OWNER tap LOCATION \E(?:E)?\Q'$tablespace1';\E
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT super TO grant7 WITH INHERIT TRUE GRANTED BY\E
+ (.*\n)*
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: When --list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: When non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\Qpg_restore: error: could not connect to database "dbpq"\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+$node->stop('fast');
+
+done_testing();
--
2.39.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-10-16 10:54 Mahendra Singh Thalor <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-10-16 10:54 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; tushar <[email protected]>; +Cc: Noah Misch <[email protected]>; Tom Lane <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Wed, 15 Oct 2025 at 23:05, Mahendra Singh Thalor <[email protected]> wrote:
>
> On Sun, 24 Aug 2025 at 22:12, Andrew Dunstan <[email protected]> wrote:
> >
> >
> > On 2025-08-23 Sa 9:08 PM, Noah Misch wrote:
> >
> > On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
> >
> > OK, now that's reverted we should discuss how to proceed. I had two thoughts
> > - we could use invent a JSON format for the globals, or we could just use
> > the existing archive format. I think the archive format is pretty flexible,
> > and should be able to accommodate this. The downside is it's not humanly
> > readable. The upside is that we don't need to do anything special either to
> > write it or parse it.
> >
> > I would first try to use the existing archiver API, because that makes it
> > harder to miss bugs. Any tension between that API and pg_dumpall is likely to
> > have corresponding tension on the pg_restore side. Resolving that tension
> > will reveal much of the project's scope that remained hidden during the v18
> > attempt. Perhaps more important than that, using the archiver API means
> > future pg_dump and pg_restore options are more likely to cooperate properly
> > with $SUBJECT. In other words, I want it to be hard to add pg_dump/pg_restore
> > features that malfunction only for $SUBJECT archives. The strength of the
> > archiver architecture shows in how rarely new features need format-specific
> > logic and how rarely format-specific bugs get reported. We've had little or
> > no trouble with e.g. bugs that appear in -Fd but not in -Fc.
> >
> >
> > Yeah, that's what we're going to try.
> >
> >
> > cheers
> >
> >
> > andrew
> >
> > --
> > Andrew Dunstan
> > EDB: https://www.enterprisedb.com
>
> Thanks Andrew, Noah and all others for feedback.
>
> Based on the above suggestions and discussions, I removed sql commands
> from the global.dat file. For global commands, now we are making
> toc.dat/toc.dmp/toc.tar file based on format specified and based on
> format specified, we are making archive entries for these global
> commands. By this approach, we removed the hard-coded parsing part of
> the global.dat file and we are able to skip DROP DATABASE with the
> globals-only option.
>
> Here, I am attaching a patch for review, testing and feedback. This is
> a WIP patch. I will do some more code cleanup and will add some more
> comments also. Please review this and let me know design level
> feedback. Thanks Tushar Ahuja for some internal testing and feedback.
>
Hi,
Here, I am attaching an updated patch. In offline discussion, Andrew
reported some test-case failures(Thanks Andrew). I fixed those.
Please let me know feedback for the patch.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v02-16102025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (83.9K, 2-v02-16102025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
From feeeb56d7c3e943cda2e608a5cb85cca8dc32edb Mon Sep 17 00:00:00 2001
From: ThalorMahendra <[email protected]>
Date: Thu, 16 Oct 2025 16:13:50 +0530
Subject: [PATCH] Non text modes for pg_dumpall, correspondingly change
pg_restore
pg_dumpall acquires a new -F/--format option, with the same meanings as
pg_dump. The default is p, meaning plain text. For any other value, a
directory is created containing two files, toc.dat/.dmp/.tar and map.dat. The
first contains commands restoring the global data based on -F, and the second
contains a map from oids to database names. It will also contain a
subdirectory called databases, inside which it will create archives in
the specified format, named using the database oids.
In these casess the -f argument is required.
If pg_restore encounters a directory containing map.dat,
it restores the global settings from toc.dat/.dmp/.tar if exist, and then
restores each database.
pg_restore acquires two new options: -g/--globals-only which suppresses
restoration of any databases, and --exclude-database which inhibits
restoration of particualr database(s) in the same way the same option
works in pg_dumpall.
---
doc/src/sgml/ref/pg_dumpall.sgml | 89 +++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/connectdb.c | 1 -
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 10 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 23 +-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 600 ++++++++++++++++++++++-----
src/bin/pg_dump/pg_restore.c | 593 +++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 10 +
src/bin/pg_dump/t/006_pg_dumpall.pl | 396 ++++++++++++++++++
14 files changed, 1650 insertions(+), 146 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
create mode 100755 src/bin/pg_dump/t/006_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 9f639f61db0..4063e88d388 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,10 +139,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>toc.dat/toc.dmp/toc.tar</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index a468a38361a..7497b527ae6 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -591,6 +615,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
index d55d53dbeea..f44a8a45fca 100644
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -287,7 +287,6 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
exit_nicely(1);
}
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index a2233b0a1b4..4a4ebbd8ec9 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -102,6 +102,7 @@ tests += {
't/003_pg_dump_with_server.pl',
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
+ 't/006_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..5974d6706fd 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index d9041dad720..f631d945472 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -312,7 +312,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, bool globals_only);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 59eaecb4ed7..d378c7b601e 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, bool globals_only)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,9 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ if (globals_only && te && te->tag && (strcmp(te->tag, "DROP_DATABASE") == 0))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1324,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1695,7 +1703,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1724,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..818b80a9369 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 890db7b08c2..4501802d805 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index bb451c1bae1..668e55e415c 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -30,6 +30,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -65,9 +66,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,6 +78,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpIdLocal(void);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -123,6 +127,13 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static const CatalogId nilCatalogId = {0, 0};
+static ArchiveMode archiveMode = archModeWrite;
+static DataDirSyncMethod sync_method = DATA_DIR_SYNC_METHOD_FSYNC;
+static ArchiveFormat archDumpFormat = archNull;
int
main(int argc, char *argv[])
@@ -148,6 +159,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +209,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -208,6 +221,8 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
+ char global_path[MAXPGPATH];
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -246,7 +261,9 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ InitDumpOptions(&dopt);
+
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -257,6 +274,7 @@ main(int argc, char *argv[])
case 'c':
output_clean = true;
+ dopt.outputClean = 1;
break;
case 'd':
@@ -274,7 +292,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -314,6 +334,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -429,6 +450,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -489,6 +525,33 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (filename && archDumpFormat != archNull)
+ {
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ /* set file path for global sql commands. */
+ if (archDumpFormat == archCustom)
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", filename);
+ else if (archDumpFormat == archTar)
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", filename);
+ else if (archDumpFormat == archDirectory)
+ snprintf(global_path, MAXPGPATH, "%s", filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -538,19 +601,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -585,37 +635,123 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
if (verbose)
dumpTimestamp("Started on");
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ /* create a archive file for global commands. */
+ if (filename && archDumpFormat != archNull)
+ {
+ /* Open the output file */
+ fout = CreateArchive(global_path, archDumpFormat, compression_spec,
+ dosync, archiveMode, NULL, sync_method);
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+ ((ArchiveHandle * )fout) ->connection = conn;
+ ((ArchiveHandle * ) fout) -> public.numWorkers = 1;
- /* Replicate encoding and std_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
- if (strcmp(std_strings, "off") == 0)
- fprintf(OPF, "SET escape_string_warning = off;\n");
- fprintf(OPF, "\n");
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
+
+ /* dumpEncoding: put the correct encoding into the archive */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ const char *encname = pg_encoding_to_char(encoding);
+
+ pg_log_info("saving encoding = %s", encname);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+
+ ArchiveEntry(fout, nilCatalogId, createDumpIdLocal(),
+ ARCHIVE_OPTS(.tag = "ENCODING",
+ .description = "ENCODING",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpStdStrings: put the correct escape string behavior into the archive */
+ {
+ const char *stdstrings = std_strings ? "on" : "off";
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving \"standard_conforming_strings = %s\"", stdstrings);
+
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = '%s';\n",
+ stdstrings);
+
+ ArchiveEntry(fout, nilCatalogId, createDumpIdLocal(),
+ ARCHIVE_OPTS(.tag = "STDSTRINGS",
+ .description = "STDSTRINGS",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+
+ destroyPQExpBuffer(qry);
+ }
+
+ /* default_transaction_read_only = off */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving default_transaction_read_only = off");
+
+ appendPQExpBuffer(qry, "SET default_transaction_read_only = off;\n");
+
+ ArchiveEntry(fout, nilCatalogId, createDumpIdLocal(),
+ ARCHIVE_OPTS(.tag = "DEFAULT_TRANSACTION_READ_ONLY",
+ .description = "DEFAULT_TRANSACTION_READ_ONLY",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+
+ destroyPQExpBuffer(qry);
+ }
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and std_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
+ if (strcmp(std_strings, "off") == 0)
+ fprintf(OPF, "SET escape_string_warning = off;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -659,27 +795,41 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump will
+ * handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
-
- PQfinish(conn);
+ dumpDatabases(conn, archDumpFormat);
if (verbose)
dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (filename)
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+
+ if (filename && archDumpFormat != archNull)
+ {
+ RestoreOptions *ropt;
+
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
+
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
+ }
+ else if (filename)
{
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -690,12 +840,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -770,6 +922,7 @@ static void
dropRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer delQry = createPQExpBuffer();
PGresult *res;
int i_rolname;
int i;
@@ -790,7 +943,7 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Drop roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -799,15 +952,31 @@ dropRoles(PGconn *conn)
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(rolename));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dropRoles",
+ //.owner = dba,
+ .description = "dropRoles_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ //.dropStmt = delQry->data));
+ }
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -888,7 +1057,7 @@ dumpRoles(PGconn *conn)
i_rolcomment = PQfnumber(res, "rolcomment");
i_is_current_user = PQfnumber(res, "is_current_user");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -993,7 +1162,25 @@ dumpRoles(PGconn *conn)
"ROLE", rolename,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ {
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
+ if_exists ? "IF EXISTS " : "", fmtId(rolename));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpRoles",
+ //.owner = dba,
+ .description = "dumpRoles_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ //.dropStmt = delQry->data));
+ }
}
/*
@@ -1001,15 +1188,13 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
}
@@ -1087,7 +1272,7 @@ dumpRoleMembership(PGconn *conn)
i_inherit_option = PQfnumber(res, "inherit_option");
i_set_option = PQfnumber(res, "set_option");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Role memberships\n--\n\n");
/*
@@ -1167,6 +1352,7 @@ dumpRoleMembership(PGconn *conn)
char *grantor;
char *set_option = "true";
bool found;
+ PQExpBuffer creaQry = createPQExpBuffer();
/* If we already did this grant, don't do it again. */
if (done[i - start])
@@ -1223,8 +1409,8 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ appendPQExpBuffer(creaQry, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(creaQry, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1245,10 +1431,24 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(creaQry, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(creaQry, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(creaQry, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", creaQry->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpRoleMembership",
+ //.owner = dba,
+ .description = "dumpRoleMembership_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = creaQry->data));
+ }
}
}
@@ -1260,7 +1460,8 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1287,7 +1488,7 @@ dumpRoleGUCPrivs(PGconn *conn)
"FROM pg_catalog.pg_parameter_acl "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1312,14 +1513,28 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpRoleGUCPrivs",
+ //.owner = dba,
+ .description = "dumpRoleGUCPrivs_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ }
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1331,6 +1546,7 @@ dropTablespaces(PGconn *conn)
{
PGresult *res;
int i;
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Get all tablespaces except built-in ones (which we assume are named
@@ -1341,21 +1557,37 @@ dropTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(spcname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dropTablespaces",
+ //.owner = dba,
+ .description = "dropTablespaces_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ // .dropStmt = delQry->data));
+ }
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1381,7 +1613,7 @@ dumpTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1451,7 +1683,25 @@ dumpTablespaces(PGconn *conn)
"TABLESPACE", spcname,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ {
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
+ if_exists ? "IF EXISTS " : "", fmtId(fspcname));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpTablespaces",
+ //.owner = dba,
+ .description = "dumpTablespaces_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ //.dropStmt = delQry->data));
+ }
free(fspcname);
destroyPQExpBuffer(buf);
@@ -1481,7 +1731,7 @@ dropDBs(PGconn *conn)
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY datname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1497,9 +1747,26 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(dbname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(),
+ ARCHIVE_OPTS(.tag = "DROP_DATABASE",
+ //.owner = dba,
+ .description = "DROP_DATABASE_COMMANDS",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ //.dropStmt = delQry->data));
+ }
}
}
@@ -1517,6 +1784,7 @@ dumpUserConfig(PGconn *conn, const char *username)
{
PQExpBuffer buf = createPQExpBuffer();
PGresult *res;
+ static bool header_done = false;
printfPQExpBuffer(buf, "SELECT unnest(setconfig) FROM pg_db_role_setting "
"WHERE setdatabase = 0 AND setrole = "
@@ -1532,7 +1800,9 @@ dumpUserConfig(PGconn *conn, const char *username)
char *sanitized;
sanitized = sanitize_line(username, true);
- fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ if (!header_done && (archDumpFormat == archNull))
+ fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ header_done = true;
free(sanitized);
}
@@ -1542,7 +1812,19 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpUserConfig",
+ //.owner = dba,
+ .description = "dumpUserConfig_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ }
}
PQclear(res);
@@ -1608,10 +1890,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1625,19 +1910,43 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
- if (PQntuples(res) > 0)
+ if (archDumpFormat == archNull && PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1651,10 +1960,14 @@ dumpDatabases(PGconn *conn)
continue;
}
- pg_log_info("dumping database \"%s\"", dbname);
+ if (archDumpFormat == archNull)
+ pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
free(sanitized);
/*
@@ -1669,24 +1982,38 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
- else
- {
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
fprintf(OPF, "\\connect %s\n\n", dbname);
- }
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1695,6 +2022,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1704,7 +2035,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1713,17 +2045,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1868,3 +2219,42 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+static int
+createDumpIdLocal(void)
+{
+ return ++dumpIdVal;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c9776306c5c..02176a77bd7 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,31 +41,60 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num,
+ bool globals_only);
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only);
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +118,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +172,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +201,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +228,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -316,6 +350,9 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
case 6:
opts->restrict_key = pg_strdup(optarg);
@@ -347,6 +384,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -472,6 +516,105 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If map.dat file is present, then restore all the
+ * databases from map.dat , but skip restoring those matching
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "map.dat") ||
+ file_exists_in_directory(inputFileSpec, "toc.tar") ||
+ file_exists_in_directory(inputFileSpec, "toc.dmp")))
+ {
+ char global_path[MAXPGPATH];
+
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring an archive created by pg_dumpall");
+ else if (opts->tocFile)
+ pg_fatal("option -L/--use-list cannot be used when restoring an archive created by pg_dumpall");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option -C/--create must be specified when restoring an archive created by pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ n_errors = restore_global_objects(global_path, opts, numWorkers, false, 0, globals_only);
+
+ pg_log_info("database restoring skipped because option -g/--globals-only was specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else /* process if map.dat file does not exist. */
+ {
+ n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, 0, globals_only);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ *
+ * If globals_only is set, then skip DROP DATABASE commands from restore.
+ */
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ return restore_one_database(inputFileSpec, opts, numWorkers, append_data, num, globals_only);
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -479,9 +622,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -501,25 +650,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, globals_only);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -537,6 +682,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -553,6 +699,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -588,8 +735,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -694,3 +841,415 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn && db_exclude_patterns.head != NULL)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no database connection while doing pg_restore");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+ PQExpBuffer db_lit = createPQExpBuffer();
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches exclude pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ destroyPQExpBuffer(db_lit);
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ destroyPQExpBuffer(query);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+
+
+ /*
+ * If there is only global.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen <= 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbname, db_oid, map_file_path);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+ PGconn *conn = NULL;
+ char global_path[MAXPGPATH];
+
+ /* Based on file, set path. */
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
+
+ /* If map.dat has no entries, return after processing global commands. */
+ if (dbname_oid_list.head == NULL)
+ return restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Open toc.dat file and execute/append all the global sql commands. */
+ n_errors_total = restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ count++;
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..56e89da1e5e
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,12 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +250,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
done_testing();
diff --git a/src/bin/pg_dump/t/006_pg_dumpall.pl b/src/bin/pg_dump/t/006_pg_dumpall.pl
new file mode 100755
index 00000000000..3c7d2ad7c53
--- /dev/null
+++ b/src/bin/pg_dump/t/006_pg_dumpall.pl
@@ -0,0 +1,396 @@
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# Scape tablespace locations on Windows.
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT super TO grant7 WITH INHERIT TRUE GRANTED BY\E
+ (.*\n)*
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: When --list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: When non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+$node->stop('fast');
+
+done_testing();
--
2.39.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-10-28 06:02 Mahendra Singh Thalor <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-10-28 06:02 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; tushar <[email protected]>; +Cc: Noah Misch <[email protected]>; Tom Lane <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Thu, 16 Oct 2025 at 16:24, Mahendra Singh Thalor <[email protected]> wrote:
>
> On Wed, 15 Oct 2025 at 23:05, Mahendra Singh Thalor <[email protected]> wrote:
> >
> > On Sun, 24 Aug 2025 at 22:12, Andrew Dunstan <[email protected]> wrote:
> > >
> > >
> > > On 2025-08-23 Sa 9:08 PM, Noah Misch wrote:
> > >
> > > On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
> > >
> > > OK, now that's reverted we should discuss how to proceed. I had two thoughts
> > > - we could use invent a JSON format for the globals, or we could just use
> > > the existing archive format. I think the archive format is pretty flexible,
> > > and should be able to accommodate this. The downside is it's not humanly
> > > readable. The upside is that we don't need to do anything special either to
> > > write it or parse it.
> > >
> > > I would first try to use the existing archiver API, because that makes it
> > > harder to miss bugs. Any tension between that API and pg_dumpall is likely to
> > > have corresponding tension on the pg_restore side. Resolving that tension
> > > will reveal much of the project's scope that remained hidden during the v18
> > > attempt. Perhaps more important than that, using the archiver API means
> > > future pg_dump and pg_restore options are more likely to cooperate properly
> > > with $SUBJECT. In other words, I want it to be hard to add pg_dump/pg_restore
> > > features that malfunction only for $SUBJECT archives. The strength of the
> > > archiver architecture shows in how rarely new features need format-specific
> > > logic and how rarely format-specific bugs get reported. We've had little or
> > > no trouble with e.g. bugs that appear in -Fd but not in -Fc.
> > >
> > >
> > > Yeah, that's what we're going to try.
> > >
> > >
> > > cheers
> > >
> > >
> > > andrew
> > >
> > > --
> > > Andrew Dunstan
> > > EDB: https://www.enterprisedb.com
> >
> > Thanks Andrew, Noah and all others for feedback.
> >
> > Based on the above suggestions and discussions, I removed sql commands
> > from the global.dat file. For global commands, now we are making
> > toc.dat/toc.dmp/toc.tar file based on format specified and based on
> > format specified, we are making archive entries for these global
> > commands. By this approach, we removed the hard-coded parsing part of
> > the global.dat file and we are able to skip DROP DATABASE with the
> > globals-only option.
> >
> > Here, I am attaching a patch for review, testing and feedback. This is
> > a WIP patch. I will do some more code cleanup and will add some more
> > comments also. Please review this and let me know design level
> > feedback. Thanks Tushar Ahuja for some internal testing and feedback.
> >
>
> Hi,
> Here, I am attaching an updated patch. In offline discussion, Andrew
> reported some test-case failures(Thanks Andrew). I fixed those.
> Please let me know feedback for the patch.
>
Hi,
Here I am attaching a re-based patch as v02 was failing on head.
Thanks Tushar for the testing.
Please review this and let me know feedback.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v03-28102025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (83.9K, 2-v03-28102025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
From 474880773c7fc4679ea9054bde58c912ce279266 Mon Sep 17 00:00:00 2001
From: ThalorMahendra <[email protected]>
Date: Tue, 28 Oct 2025 11:27:43 +0530
Subject: [PATCH] Non text modes for pg_dumpall, correspondingly change
pg_restore
pg_dumpall acquires a new -F/--format option, with the same meanings as
pg_dump. The default is p, meaning plain text. For any other value, a
directory is created containing two files, toc.dat/.dmp/.tar and map.dat. The
first contains commands restoring the global data based on -F, and the second
contains a map from oids to database names. It will also contain a
subdirectory called databases, inside which it will create archives in
the specified format, named using the database oids.
In these casess the -f argument is required.
If pg_restore encounters a directory containing map.dat,
it restores the global settings from toc.dat/.dmp/.tar if exist, and then
restores each database.
pg_restore acquires two new options: -g/--globals-only which suppresses
restoration of any databases, and --exclude-database which inhibits
restoration of particualr database(s) in the same way the same option
works in pg_dumpall.
---
---
doc/src/sgml/ref/pg_dumpall.sgml | 89 +++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/connectdb.c | 1 -
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 10 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 23 +-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 600 ++++++++++++++++++++++-----
src/bin/pg_dump/pg_restore.c | 593 +++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 10 +
src/bin/pg_dump/t/007_pg_dumpall.pl | 396 ++++++++++++++++++
14 files changed, 1650 insertions(+), 146 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
create mode 100755 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 9f639f61db0..4063e88d388 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,10 +139,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>toc.dat/toc.dmp/toc.tar</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index a468a38361a..7497b527ae6 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -591,6 +615,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
index d55d53dbeea..f44a8a45fca 100644
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -287,7 +287,6 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
exit_nicely(1);
}
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index f3c669f484e..3e21aaf5780 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..5974d6706fd 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index d9041dad720..f631d945472 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -312,7 +312,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, bool globals_only);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 59eaecb4ed7..d378c7b601e 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, bool globals_only)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,9 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ if (globals_only && te && te->tag && (strcmp(te->tag, "DROP_DATABASE") == 0))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1324,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1695,7 +1703,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1724,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..818b80a9369 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 47913178a93..00ce946aab1 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index bb451c1bae1..668e55e415c 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -30,6 +30,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -65,9 +66,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,6 +78,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpIdLocal(void);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -123,6 +127,13 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static const CatalogId nilCatalogId = {0, 0};
+static ArchiveMode archiveMode = archModeWrite;
+static DataDirSyncMethod sync_method = DATA_DIR_SYNC_METHOD_FSYNC;
+static ArchiveFormat archDumpFormat = archNull;
int
main(int argc, char *argv[])
@@ -148,6 +159,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +209,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -208,6 +221,8 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
+ char global_path[MAXPGPATH];
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -246,7 +261,9 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ InitDumpOptions(&dopt);
+
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -257,6 +274,7 @@ main(int argc, char *argv[])
case 'c':
output_clean = true;
+ dopt.outputClean = 1;
break;
case 'd':
@@ -274,7 +292,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -314,6 +334,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -429,6 +450,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -489,6 +525,33 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (filename && archDumpFormat != archNull)
+ {
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ /* set file path for global sql commands. */
+ if (archDumpFormat == archCustom)
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", filename);
+ else if (archDumpFormat == archTar)
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", filename);
+ else if (archDumpFormat == archDirectory)
+ snprintf(global_path, MAXPGPATH, "%s", filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -538,19 +601,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -585,37 +635,123 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
if (verbose)
dumpTimestamp("Started on");
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ /* create a archive file for global commands. */
+ if (filename && archDumpFormat != archNull)
+ {
+ /* Open the output file */
+ fout = CreateArchive(global_path, archDumpFormat, compression_spec,
+ dosync, archiveMode, NULL, sync_method);
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+ ((ArchiveHandle * )fout) ->connection = conn;
+ ((ArchiveHandle * ) fout) -> public.numWorkers = 1;
- /* Replicate encoding and std_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
- if (strcmp(std_strings, "off") == 0)
- fprintf(OPF, "SET escape_string_warning = off;\n");
- fprintf(OPF, "\n");
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
+
+ /* dumpEncoding: put the correct encoding into the archive */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ const char *encname = pg_encoding_to_char(encoding);
+
+ pg_log_info("saving encoding = %s", encname);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+
+ ArchiveEntry(fout, nilCatalogId, createDumpIdLocal(),
+ ARCHIVE_OPTS(.tag = "ENCODING",
+ .description = "ENCODING",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpStdStrings: put the correct escape string behavior into the archive */
+ {
+ const char *stdstrings = std_strings ? "on" : "off";
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving \"standard_conforming_strings = %s\"", stdstrings);
+
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = '%s';\n",
+ stdstrings);
+
+ ArchiveEntry(fout, nilCatalogId, createDumpIdLocal(),
+ ARCHIVE_OPTS(.tag = "STDSTRINGS",
+ .description = "STDSTRINGS",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+
+ destroyPQExpBuffer(qry);
+ }
+
+ /* default_transaction_read_only = off */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving default_transaction_read_only = off");
+
+ appendPQExpBuffer(qry, "SET default_transaction_read_only = off;\n");
+
+ ArchiveEntry(fout, nilCatalogId, createDumpIdLocal(),
+ ARCHIVE_OPTS(.tag = "DEFAULT_TRANSACTION_READ_ONLY",
+ .description = "DEFAULT_TRANSACTION_READ_ONLY",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+
+ destroyPQExpBuffer(qry);
+ }
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and std_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
+ if (strcmp(std_strings, "off") == 0)
+ fprintf(OPF, "SET escape_string_warning = off;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -659,27 +795,41 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump will
+ * handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
-
- PQfinish(conn);
+ dumpDatabases(conn, archDumpFormat);
if (verbose)
dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (filename)
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+
+ if (filename && archDumpFormat != archNull)
+ {
+ RestoreOptions *ropt;
+
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
+
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
+ }
+ else if (filename)
{
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -690,12 +840,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -770,6 +922,7 @@ static void
dropRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer delQry = createPQExpBuffer();
PGresult *res;
int i_rolname;
int i;
@@ -790,7 +943,7 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Drop roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -799,15 +952,31 @@ dropRoles(PGconn *conn)
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(rolename));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dropRoles",
+ //.owner = dba,
+ .description = "dropRoles_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ //.dropStmt = delQry->data));
+ }
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -888,7 +1057,7 @@ dumpRoles(PGconn *conn)
i_rolcomment = PQfnumber(res, "rolcomment");
i_is_current_user = PQfnumber(res, "is_current_user");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -993,7 +1162,25 @@ dumpRoles(PGconn *conn)
"ROLE", rolename,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ {
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
+ if_exists ? "IF EXISTS " : "", fmtId(rolename));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpRoles",
+ //.owner = dba,
+ .description = "dumpRoles_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ //.dropStmt = delQry->data));
+ }
}
/*
@@ -1001,15 +1188,13 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
}
@@ -1087,7 +1272,7 @@ dumpRoleMembership(PGconn *conn)
i_inherit_option = PQfnumber(res, "inherit_option");
i_set_option = PQfnumber(res, "set_option");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Role memberships\n--\n\n");
/*
@@ -1167,6 +1352,7 @@ dumpRoleMembership(PGconn *conn)
char *grantor;
char *set_option = "true";
bool found;
+ PQExpBuffer creaQry = createPQExpBuffer();
/* If we already did this grant, don't do it again. */
if (done[i - start])
@@ -1223,8 +1409,8 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ appendPQExpBuffer(creaQry, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(creaQry, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1245,10 +1431,24 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(creaQry, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(creaQry, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(creaQry, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", creaQry->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpRoleMembership",
+ //.owner = dba,
+ .description = "dumpRoleMembership_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = creaQry->data));
+ }
}
}
@@ -1260,7 +1460,8 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1287,7 +1488,7 @@ dumpRoleGUCPrivs(PGconn *conn)
"FROM pg_catalog.pg_parameter_acl "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1312,14 +1513,28 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpRoleGUCPrivs",
+ //.owner = dba,
+ .description = "dumpRoleGUCPrivs_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ }
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1331,6 +1546,7 @@ dropTablespaces(PGconn *conn)
{
PGresult *res;
int i;
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Get all tablespaces except built-in ones (which we assume are named
@@ -1341,21 +1557,37 @@ dropTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(spcname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dropTablespaces",
+ //.owner = dba,
+ .description = "dropTablespaces_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ // .dropStmt = delQry->data));
+ }
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1381,7 +1613,7 @@ dumpTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && (archDumpFormat == archNull))
fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1451,7 +1683,25 @@ dumpTablespaces(PGconn *conn)
"TABLESPACE", spcname,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ {
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
+ if_exists ? "IF EXISTS " : "", fmtId(fspcname));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpTablespaces",
+ //.owner = dba,
+ .description = "dumpTablespaces_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ //.dropStmt = delQry->data));
+ }
free(fspcname);
destroyPQExpBuffer(buf);
@@ -1481,7 +1731,7 @@ dropDBs(PGconn *conn)
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY datname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1497,9 +1747,26 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(dbname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(),
+ ARCHIVE_OPTS(.tag = "DROP_DATABASE",
+ //.owner = dba,
+ .description = "DROP_DATABASE_COMMANDS",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ //.dropStmt = delQry->data));
+ }
}
}
@@ -1517,6 +1784,7 @@ dumpUserConfig(PGconn *conn, const char *username)
{
PQExpBuffer buf = createPQExpBuffer();
PGresult *res;
+ static bool header_done = false;
printfPQExpBuffer(buf, "SELECT unnest(setconfig) FROM pg_db_role_setting "
"WHERE setdatabase = 0 AND setrole = "
@@ -1532,7 +1800,9 @@ dumpUserConfig(PGconn *conn, const char *username)
char *sanitized;
sanitized = sanitize_line(username, true);
- fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ if (!header_done && (archDumpFormat == archNull))
+ fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ header_done = true;
free(sanitized);
}
@@ -1542,7 +1812,19 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ {
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpIdLocal(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "dumpUserConfig",
+ //.owner = dba,
+ .description = "dumpUserConfig_des",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ }
}
PQclear(res);
@@ -1608,10 +1890,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1625,19 +1910,43 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
- if (PQntuples(res) > 0)
+ if (archDumpFormat == archNull && PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1651,10 +1960,14 @@ dumpDatabases(PGconn *conn)
continue;
}
- pg_log_info("dumping database \"%s\"", dbname);
+ if (archDumpFormat == archNull)
+ pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
free(sanitized);
/*
@@ -1669,24 +1982,38 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
- else
- {
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
fprintf(OPF, "\\connect %s\n\n", dbname);
- }
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1695,6 +2022,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1704,7 +2035,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1713,17 +2045,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1868,3 +2219,42 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+static int
+createDumpIdLocal(void)
+{
+ return ++dumpIdVal;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c9776306c5c..02176a77bd7 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,31 +41,60 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num,
+ bool globals_only);
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only);
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +118,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +172,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +201,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +228,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -316,6 +350,9 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
case 6:
opts->restrict_key = pg_strdup(optarg);
@@ -347,6 +384,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -472,6 +516,105 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If map.dat file is present, then restore all the
+ * databases from map.dat , but skip restoring those matching
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "map.dat") ||
+ file_exists_in_directory(inputFileSpec, "toc.tar") ||
+ file_exists_in_directory(inputFileSpec, "toc.dmp")))
+ {
+ char global_path[MAXPGPATH];
+
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring an archive created by pg_dumpall");
+ else if (opts->tocFile)
+ pg_fatal("option -L/--use-list cannot be used when restoring an archive created by pg_dumpall");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option -C/--create must be specified when restoring an archive created by pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ n_errors = restore_global_objects(global_path, opts, numWorkers, false, 0, globals_only);
+
+ pg_log_info("database restoring skipped because option -g/--globals-only was specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else /* process if map.dat file does not exist. */
+ {
+ n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, 0, globals_only);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ *
+ * If globals_only is set, then skip DROP DATABASE commands from restore.
+ */
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ return restore_one_database(inputFileSpec, opts, numWorkers, append_data, num, globals_only);
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -479,9 +622,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -501,25 +650,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, globals_only);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -537,6 +682,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -553,6 +699,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -588,8 +735,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -694,3 +841,415 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn && db_exclude_patterns.head != NULL)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no database connection while doing pg_restore");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+ PQExpBuffer db_lit = createPQExpBuffer();
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches exclude pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ destroyPQExpBuffer(db_lit);
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ destroyPQExpBuffer(query);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+
+
+ /*
+ * If there is only global.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen <= 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbname, db_oid, map_file_path);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+ PGconn *conn = NULL;
+ char global_path[MAXPGPATH];
+
+ /* Based on file, set path. */
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
+
+ /* If map.dat has no entries, return after processing global commands. */
+ if (dbname_oid_list.head == NULL)
+ return restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Open toc.dat file and execute/append all the global sql commands. */
+ n_errors_total = restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ count++;
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..56e89da1e5e
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,12 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +250,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100755
index 00000000000..3c7d2ad7c53
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,396 @@
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# Scape tablespace locations on Windows.
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT super TO grant7 WITH INHERIT TRUE GRANTED BY\E
+ (.*\n)*
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: When --list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: When non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+$node->stop('fast');
+
+done_testing();
--
2.39.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-10-31 09:20 Mahendra Singh Thalor <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-10-31 09:20 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; tushar <[email protected]>; +Cc: Noah Misch <[email protected]>; Tom Lane <[email protected]>; Álvaro Herrera <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Tue, 28 Oct 2025 at 11:32, Mahendra Singh Thalor <[email protected]> wrote:
>
> On Thu, 16 Oct 2025 at 16:24, Mahendra Singh Thalor <[email protected]> wrote:
> >
> > On Wed, 15 Oct 2025 at 23:05, Mahendra Singh Thalor <[email protected]> wrote:
> > >
> > > On Sun, 24 Aug 2025 at 22:12, Andrew Dunstan <[email protected]> wrote:
> > > >
> > > >
> > > > On 2025-08-23 Sa 9:08 PM, Noah Misch wrote:
> > > >
> > > > On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
> > > >
> > > > OK, now that's reverted we should discuss how to proceed. I had two thoughts
> > > > - we could use invent a JSON format for the globals, or we could just use
> > > > the existing archive format. I think the archive format is pretty flexible,
> > > > and should be able to accommodate this. The downside is it's not humanly
> > > > readable. The upside is that we don't need to do anything special either to
> > > > write it or parse it.
> > > >
> > > > I would first try to use the existing archiver API, because that makes it
> > > > harder to miss bugs. Any tension between that API and pg_dumpall is likely to
> > > > have corresponding tension on the pg_restore side. Resolving that tension
> > > > will reveal much of the project's scope that remained hidden during the v18
> > > > attempt. Perhaps more important than that, using the archiver API means
> > > > future pg_dump and pg_restore options are more likely to cooperate properly
> > > > with $SUBJECT. In other words, I want it to be hard to add pg_dump/pg_restore
> > > > features that malfunction only for $SUBJECT archives. The strength of the
> > > > archiver architecture shows in how rarely new features need format-specific
> > > > logic and how rarely format-specific bugs get reported. We've had little or
> > > > no trouble with e.g. bugs that appear in -Fd but not in -Fc.
> > > >
> > > >
> > > > Yeah, that's what we're going to try.
> > > >
> > > >
> > > > cheers
> > > >
> > > >
> > > > andrew
> > > >
> > > > --
> > > > Andrew Dunstan
> > > > EDB: https://www.enterprisedb.com
> > >
> > > Thanks Andrew, Noah and all others for feedback.
> > >
> > > Based on the above suggestions and discussions, I removed sql commands
> > > from the global.dat file. For global commands, now we are making
> > > toc.dat/toc.dmp/toc.tar file based on format specified and based on
> > > format specified, we are making archive entries for these global
> > > commands. By this approach, we removed the hard-coded parsing part of
> > > the global.dat file and we are able to skip DROP DATABASE with the
> > > globals-only option.
> > >
> > > Here, I am attaching a patch for review, testing and feedback. This is
> > > a WIP patch. I will do some more code cleanup and will add some more
> > > comments also. Please review this and let me know design level
> > > feedback. Thanks Tushar Ahuja for some internal testing and feedback.
> > >
> >
> > Hi,
> > Here, I am attaching an updated patch. In offline discussion, Andrew
> > reported some test-case failures(Thanks Andrew). I fixed those.
> > Please let me know feedback for the patch.
> >
>
> Hi,
> Here I am attaching a re-based patch as v02 was failing on head.
> Thanks Tushar for the testing.
> Please review this and let me know feedback.
>
Hi all,
Here I am attaching an updated patch for review and testing. Based on
some offline comments by Andrew, I did some code cleanup.
Please consider this patch for feedback.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v04-31102025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (83.9K, 2-v04-31102025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
From a29957e76b98f6b544ca989d4d8098a5968534fa Mon Sep 17 00:00:00 2001
From: ThalorMahendra <[email protected]>
Date: Fri, 31 Oct 2025 14:45:48 +0530
Subject: [PATCH] Non text modes for pg_dumpall, correspondingly change
pg_restore
pg_dumpall acquires a new -F/--format option, with the same meanings as
pg_dump. The default is p, meaning plain text. For any other value, a
directory is created containing two files, toc.dat/.dmp/.tar and map.dat. The
first contains commands restoring the global data based on -F, and the second
contains a map from oids to database names. It will also contain a
subdirectory called databases, inside which it will create archives in
the specified format, named using the database oids.
In these casess the -f argument is required.
If pg_restore encounters a directory containing map.dat,
it restores the global settings from toc.dat/.dmp/.tar if exist, and then
restores each database.
pg_restore acquires two new options: -g/--globals-only which suppresses
restoration of any databases, and --exclude-database which inhibits
restoration of particualr database(s) in the same way the same option
works in pg_dumpall.
---
doc/src/sgml/ref/pg_dumpall.sgml | 89 +++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/connectdb.c | 1 -
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 10 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 23 +-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 608 ++++++++++++++++++++++-----
src/bin/pg_dump/pg_restore.c | 593 +++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 10 +
src/bin/pg_dump/t/007_pg_dumpall.pl | 396 +++++++++++++++++
14 files changed, 1658 insertions(+), 146 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
create mode 100755 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 9f639f61db0..4063e88d388 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,10 +139,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>toc.dat/toc.dmp/toc.tar</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index a468a38361a..7497b527ae6 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -591,6 +615,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
index d55d53dbeea..f44a8a45fca 100644
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -287,7 +287,6 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
exit_nicely(1);
}
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index f3c669f484e..3e21aaf5780 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..5974d6706fd 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index d9041dad720..f631d945472 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -312,7 +312,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, bool globals_only);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 59eaecb4ed7..d378c7b601e 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, bool globals_only)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,9 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ if (globals_only && te && te->tag && (strcmp(te->tag, "DROP_DATABASE") == 0))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1324,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1695,7 +1703,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1724,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..818b80a9369 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 47913178a93..00ce946aab1 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index bb451c1bae1..601b9f9738e 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -30,6 +30,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -65,9 +66,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,6 +78,9 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
+static void createOneArchiveEntry(const char *query, const char *tag);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -123,6 +128,13 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static const CatalogId nilCatalogId = {0, 0};
+static ArchiveMode archiveMode = archModeWrite;
+static DataDirSyncMethod sync_method = DATA_DIR_SYNC_METHOD_FSYNC;
+static ArchiveFormat archDumpFormat = archNull;
int
main(int argc, char *argv[])
@@ -148,6 +160,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +210,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -208,6 +222,8 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
+ char global_path[MAXPGPATH];
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -246,7 +262,9 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ InitDumpOptions(&dopt);
+
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -257,6 +275,7 @@ main(int argc, char *argv[])
case 'c':
output_clean = true;
+ dopt.outputClean = 1;
break;
case 'd':
@@ -274,7 +293,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -314,6 +335,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -429,6 +451,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -489,6 +526,35 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ /* set file path for global sql commands. */
+ if (archDumpFormat == archCustom)
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", filename);
+ else if (archDumpFormat == archTar)
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", filename);
+ else if (archDumpFormat == archDirectory)
+ snprintf(global_path, MAXPGPATH, "%s", filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -538,19 +604,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -585,37 +638,115 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
if (verbose)
dumpTimestamp("Started on");
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ /* create a archive file for global commands. */
+ if (filename && archDumpFormat != archNull)
+ {
+ /* Open the output file */
+ fout = CreateArchive(global_path, archDumpFormat, compression_spec,
+ dosync, archiveMode, NULL, sync_method);
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
+
+ ((ArchiveHandle * )fout) ->connection = conn;
+ ((ArchiveHandle * ) fout) -> public.numWorkers = 1;
+
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
+
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump\n--\n\n", "COMMENT");
+
+ /* create entry for restrict */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+ appendPQExpBuffer(qry, "\\restrict %s\n\n", restrict_key);
+ createOneArchiveEntry(qry->data, "RESTRICT");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* default_transaction_read_only = off */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving default_transaction_read_only = off");
+ appendPQExpBuffer(qry, "SET default_transaction_read_only = off;\n");
+ createOneArchiveEntry(qry->data, "DEFAULT_TRANSACTION_READ_ONLY");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpEncoding: put the correct encoding into the archive */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ const char *encname = pg_encoding_to_char(encoding);
- /* Replicate encoding and std_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
- if (strcmp(std_strings, "off") == 0)
- fprintf(OPF, "SET escape_string_warning = off;\n");
- fprintf(OPF, "\n");
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+
+ pg_log_info("saving encoding = %s", encname);
+ createOneArchiveEntry(qry->data, "ENCODING");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpStdStrings: put the correct escape string behavior into the archive */
+ {
+ const char *stdstrings = std_strings ? "on" : "off";
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving \"standard_conforming_strings = %s\"", stdstrings);
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = '%s';\n",
+ stdstrings);
+ createOneArchiveEntry(qry->data, "STDSTRINGS");
+ destroyPQExpBuffer(qry);
+ }
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and std_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
+ if (strcmp(std_strings, "off") == 0)
+ fprintf(OPF, "SET escape_string_warning = off;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -659,27 +790,51 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump will
+ * handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
+ else
+ {
+ /* create entry for unrestrict */
+ PQExpBuffer qry = createPQExpBuffer();
- if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ appendPQExpBuffer(qry, "\\unrestrict %s\n\n", restrict_key);
+ createOneArchiveEntry(qry->data, "UNRESTRICT");
+ destroyPQExpBuffer(qry);
+ }
- PQfinish(conn);
+ if (!globals_only && !roles_only && !tablespaces_only)
+ dumpDatabases(conn, archDumpFormat);
if (verbose)
dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (filename)
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+
+ if (archDumpFormat != archNull)
+ {
+ RestoreOptions *ropt;
+
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump complete\n--\n\n", "COMMENT");
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
+
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
+ }
+ else if (filename)
{
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -690,12 +845,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -770,6 +927,7 @@ static void
dropRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer delQry = createPQExpBuffer();
PGresult *res;
int i_rolname;
int i;
@@ -791,7 +949,12 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -799,15 +962,21 @@ dropRoles(PGconn *conn)
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(rolename));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropRoles");
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -889,7 +1058,12 @@ dumpRoles(PGconn *conn)
i_is_current_user = PQfnumber(res, "is_current_user");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -993,7 +1167,10 @@ dumpRoles(PGconn *conn)
"ROLE", rolename,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoles");
}
/*
@@ -1001,15 +1178,13 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
}
@@ -1088,7 +1263,12 @@ dumpRoleMembership(PGconn *conn)
i_set_option = PQfnumber(res, "set_option");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role memberships\n--\n\n", "COMMENT");
+ }
/*
* We can't dump these GRANT commands in arbitrary order, because a role
@@ -1167,6 +1347,7 @@ dumpRoleMembership(PGconn *conn)
char *grantor;
char *set_option = "true";
bool found;
+ PQExpBuffer creaQry = createPQExpBuffer();
/* If we already did this grant, don't do it again. */
if (done[i - start])
@@ -1223,8 +1404,8 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ appendPQExpBuffer(creaQry, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(creaQry, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1245,10 +1426,15 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(creaQry, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(creaQry, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(creaQry, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", creaQry->data);
+ else
+ createOneArchiveEntry(creaQry->data, "dumpRoleMembership");
}
}
@@ -1260,7 +1446,8 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1288,7 +1475,12 @@ dumpRoleGUCPrivs(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role privileges on configuration parameters\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1312,14 +1504,19 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoleGUCPrivs");
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1331,6 +1528,7 @@ dropTablespaces(PGconn *conn)
{
PGresult *res;
int i;
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Get all tablespaces except built-in ones (which we assume are named
@@ -1342,20 +1540,31 @@ dropTablespaces(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(spcname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropTablespaces");
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1382,7 +1591,12 @@ dumpTablespaces(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1451,14 +1665,19 @@ dumpTablespaces(PGconn *conn)
"TABLESPACE", spcname,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpTablespaces");
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1482,7 +1701,12 @@ dropDBs(PGconn *conn)
"ORDER BY datname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop databases (except postgres and template1)\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1497,15 +1721,23 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(dbname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "DROP_DATABASE");
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1532,7 +1764,18 @@ dumpUserConfig(PGconn *conn, const char *username)
char *sanitized;
sanitized = sanitize_line(username, true);
- fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
}
@@ -1542,7 +1785,11 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpUserConfig");
}
PQclear(res);
@@ -1608,10 +1855,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1625,19 +1875,48 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Databases\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Databases\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Databases\n--\n\n", "COMMENT");
+ }
+
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1654,7 +1933,18 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
/*
@@ -1669,24 +1959,46 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
+ fprintf(OPF, "\\connect %s\n\n", dbname);
else
{
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\\connect %s\n\n", dbname);
+ createOneArchiveEntry(qry->data, "CONNECT");
+ destroyPQExpBuffer(qry);
}
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1695,6 +2007,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1704,7 +2020,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1713,17 +2030,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1807,7 +2143,18 @@ dumpTimestamp(const char *msg)
time_t now = time(NULL);
if (strftime(buf, sizeof(buf), PGDUMP_STRFTIME_FMT, localtime(&now)) != 0)
- fprintf(OPF, "-- %s %s\n\n", msg, buf);
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "-- %s %s\n\n", msg, buf);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "-- %s %s\n\n", msg, buf);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+ }
}
/*
@@ -1868,3 +2215,54 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
+
+static void
+createOneArchiveEntry(const char *query, const char *tag)
+{
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = tag,
+ .section = SECTION_PRE_DATA,
+ .createStmt = query));
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c9776306c5c..02176a77bd7 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,31 +41,60 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num,
+ bool globals_only);
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only);
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +118,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +172,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +201,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +228,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -316,6 +350,9 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
case 6:
opts->restrict_key = pg_strdup(optarg);
@@ -347,6 +384,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -472,6 +516,105 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If map.dat file is present, then restore all the
+ * databases from map.dat , but skip restoring those matching
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "map.dat") ||
+ file_exists_in_directory(inputFileSpec, "toc.tar") ||
+ file_exists_in_directory(inputFileSpec, "toc.dmp")))
+ {
+ char global_path[MAXPGPATH];
+
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring an archive created by pg_dumpall");
+ else if (opts->tocFile)
+ pg_fatal("option -L/--use-list cannot be used when restoring an archive created by pg_dumpall");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option -C/--create must be specified when restoring an archive created by pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ n_errors = restore_global_objects(global_path, opts, numWorkers, false, 0, globals_only);
+
+ pg_log_info("database restoring skipped because option -g/--globals-only was specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else /* process if map.dat file does not exist. */
+ {
+ n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, 0, globals_only);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ *
+ * If globals_only is set, then skip DROP DATABASE commands from restore.
+ */
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ return restore_one_database(inputFileSpec, opts, numWorkers, append_data, num, globals_only);
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -479,9 +622,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -501,25 +650,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, globals_only);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -537,6 +682,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -553,6 +699,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -588,8 +735,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -694,3 +841,415 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn && db_exclude_patterns.head != NULL)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no database connection while doing pg_restore");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+ PQExpBuffer db_lit = createPQExpBuffer();
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches exclude pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ destroyPQExpBuffer(db_lit);
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ destroyPQExpBuffer(query);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+
+
+ /*
+ * If there is only global.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen <= 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbname, db_oid, map_file_path);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+ PGconn *conn = NULL;
+ char global_path[MAXPGPATH];
+
+ /* Based on file, set path. */
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
+
+ /* If map.dat has no entries, return after processing global commands. */
+ if (dbname_oid_list.head == NULL)
+ return restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Open toc.dat file and execute/append all the global sql commands. */
+ n_errors_total = restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ count++;
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..56e89da1e5e
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,12 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +250,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100755
index 00000000000..3c7d2ad7c53
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,396 @@
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# Scape tablespace locations on Windows.
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT super TO grant7 WITH INHERIT TRUE GRANTED BY\E
+ (.*\n)*
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: When --list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: When non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+$node->stop('fast');
+
+done_testing();
--
2.39.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-03 06:35 Vaibhav Dalvi <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 2 replies; 65+ messages in thread
From: Vaibhav Dalvi @ 2025-11-03 06:35 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: [email protected]; Vaibhav Dalvi <[email protected]>
Hi Mahendra,
Thank you for your work on this feature.
I have just begun reviewing the latest patch and
encountered the following errors during the initial setup:
```
$ ./db/bin/pg_restore testdump_dir -C -d postgres -F d -p 5556
pg_restore: error: could not execute query: ERROR: syntax error at or near
"\\"
LINE 1: \restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj...
^
Command was: \restrict
aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
pg_restore: error: could not execute query: ERROR: syntax error at or near
"\\"
LINE 1: \unrestrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCj...
^
Command was: \unrestrict
aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
pg_restore: error: could not execute query: ERROR: syntax error at or near
"\\"
LINE 1: \connect template1
^
Command was: \connect template1
pg_restore: error: could not execute query: ERROR: syntax error at or near
"\\"
LINE 1: \connect postgres
^
Command was: \connect postgres
```
To cross-check tried with plain dump(with pg_dumpall) and
restored(SQL file restore) without patch and didn't get above
connection errors.
It appears there might be an issue with the dump file itself.
Please note that this is my first observation as I have just
started the review. I will continue with my assessment.
Regards,
Vaibhav Dalvi
EnterpriseDB
On Fri, Oct 31, 2025 at 2:51 PM Mahendra Singh Thalor <[email protected]>
wrote:
> On Tue, 28 Oct 2025 at 11:32, Mahendra Singh Thalor <[email protected]>
> wrote:
> >
> > On Thu, 16 Oct 2025 at 16:24, Mahendra Singh Thalor <[email protected]>
> wrote:
> > >
> > > On Wed, 15 Oct 2025 at 23:05, Mahendra Singh Thalor <
> [email protected]> wrote:
> > > >
> > > > On Sun, 24 Aug 2025 at 22:12, Andrew Dunstan <[email protected]>
> wrote:
> > > > >
> > > > >
> > > > > On 2025-08-23 Sa 9:08 PM, Noah Misch wrote:
> > > > >
> > > > > On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
> > > > >
> > > > > OK, now that's reverted we should discuss how to proceed. I had
> two thoughts
> > > > > - we could use invent a JSON format for the globals, or we could
> just use
> > > > > the existing archive format. I think the archive format is pretty
> flexible,
> > > > > and should be able to accommodate this. The downside is it's not
> humanly
> > > > > readable. The upside is that we don't need to do anything special
> either to
> > > > > write it or parse it.
> > > > >
> > > > > I would first try to use the existing archiver API, because that
> makes it
> > > > > harder to miss bugs. Any tension between that API and pg_dumpall
> is likely to
> > > > > have corresponding tension on the pg_restore side. Resolving that
> tension
> > > > > will reveal much of the project's scope that remained hidden
> during the v18
> > > > > attempt. Perhaps more important than that, using the archiver API
> means
> > > > > future pg_dump and pg_restore options are more likely to cooperate
> properly
> > > > > with $SUBJECT. In other words, I want it to be hard to add
> pg_dump/pg_restore
> > > > > features that malfunction only for $SUBJECT archives. The
> strength of the
> > > > > archiver architecture shows in how rarely new features need
> format-specific
> > > > > logic and how rarely format-specific bugs get reported. We've had
> little or
> > > > > no trouble with e.g. bugs that appear in -Fd but not in -Fc.
> > > > >
> > > > >
> > > > > Yeah, that's what we're going to try.
> > > > >
> > > > >
> > > > > cheers
> > > > >
> > > > >
> > > > > andrew
> > > > >
> > > > > --
> > > > > Andrew Dunstan
> > > > > EDB: https://www.enterprisedb.com
> > > >
> > > > Thanks Andrew, Noah and all others for feedback.
> > > >
> > > > Based on the above suggestions and discussions, I removed sql
> commands
> > > > from the global.dat file. For global commands, now we are making
> > > > toc.dat/toc.dmp/toc.tar file based on format specified and based on
> > > > format specified, we are making archive entries for these global
> > > > commands. By this approach, we removed the hard-coded parsing part of
> > > > the global.dat file and we are able to skip DROP DATABASE with the
> > > > globals-only option.
> > > >
> > > > Here, I am attaching a patch for review, testing and feedback. This
> is
> > > > a WIP patch. I will do some more code cleanup and will add some more
> > > > comments also. Please review this and let me know design level
> > > > feedback. Thanks Tushar Ahuja for some internal testing and feedback.
> > > >
> > >
> > > Hi,
> > > Here, I am attaching an updated patch. In offline discussion, Andrew
> > > reported some test-case failures(Thanks Andrew). I fixed those.
> > > Please let me know feedback for the patch.
> > >
> >
> > Hi,
> > Here I am attaching a re-based patch as v02 was failing on head.
> > Thanks Tushar for the testing.
> > Please review this and let me know feedback.
> >
>
> Hi all,
> Here I am attaching an updated patch for review and testing. Based on
> some offline comments by Andrew, I did some code cleanup.
> Please consider this patch for feedback.
>
> --
> Thanks and Regards
> Mahendra Singh Thalor
> EnterpriseDB: http://www.enterprisedb.com
>
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-03 11:54 Mahendra Singh Thalor <[email protected]>
parent: Vaibhav Dalvi <[email protected]>
1 sibling, 2 replies; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-11-03 11:54 UTC (permalink / raw)
To: Vaibhav Dalvi <[email protected]>; +Cc: [email protected]
On Mon, 3 Nov 2025 at 12:06, Vaibhav Dalvi <[email protected]>
wrote:
>
> Hi Mahendra,
>
> Thank you for your work on this feature.
> I have just begun reviewing the latest patch and
> encountered the following errors during the initial setup:
>
> ```
> $ ./db/bin/pg_restore testdump_dir -C -d postgres -F d -p 5556
> pg_restore: error: could not execute query: ERROR: syntax error at or
near "\\"
> LINE 1: \restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj...
> ^
> Command was: \restrict
aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
>
> pg_restore: error: could not execute query: ERROR: syntax error at or
near "\\"
> LINE 1: \unrestrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCj...
> ^
> Command was: \unrestrict
aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
>
> pg_restore: error: could not execute query: ERROR: syntax error at or
near "\\"
> LINE 1: \connect template1
> ^
> Command was: \connect template1
>
> pg_restore: error: could not execute query: ERROR: syntax error at or
near "\\"
> LINE 1: \connect postgres
> ^
> Command was: \connect postgres
> ```
> To cross-check tried with plain dump(with pg_dumpall) and
> restored(SQL file restore) without patch and didn't get above
> connection errors.
>
> It appears there might be an issue with the dump file itself.
> Please note that this is my first observation as I have just
> started the review. I will continue with my assessment.
>
> Regards,
> Vaibhav Dalvi
> EnterpriseDB
Thanks Vaibhav for the review.
This change was added by me in v04. Only in the case of a file, we should
restore these commands. Attached patch is fixing the same.
If we dump and restore the same file with the same user, then we will get
an error of ROLE CREATE as the same role is already created. I think,
either we can ignore this error, or we can keep it as a restore can be done
with different users.
> mst@localhost bin]$ ./pg_restore d1 -C -d postgres
> pg_restore: error: could not execute query: ERROR: role "mst" already
> exists
> Command was: CREATE ROLE mst;
> ALTER ROLE mst WITH SUPERUSER INHERIT CREATEROLE CREATEDB LOGIN
> REPLICATION BYPASSRLS;
>
>
> pg_restore: warning: errors ignored on restore: 1
>
> On Fri, Oct 31, 2025 at 2:51 PM Mahendra Singh Thalor <[email protected]>
wrote:
>>
>> On Tue, 28 Oct 2025 at 11:32, Mahendra Singh Thalor <[email protected]>
wrote:
>> >
>> > On Thu, 16 Oct 2025 at 16:24, Mahendra Singh Thalor <[email protected]>
wrote:
>> > >
>> > > On Wed, 15 Oct 2025 at 23:05, Mahendra Singh Thalor <
[email protected]> wrote:
>> > > >
>> > > > On Sun, 24 Aug 2025 at 22:12, Andrew Dunstan <[email protected]>
wrote:
>> > > > >
>> > > > >
>> > > > > On 2025-08-23 Sa 9:08 PM, Noah Misch wrote:
>> > > > >
>> > > > > On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
>> > > > >
>> > > > > OK, now that's reverted we should discuss how to proceed. I had
two thoughts
>> > > > > - we could use invent a JSON format for the globals, or we could
just use
>> > > > > the existing archive format. I think the archive format is
pretty flexible,
>> > > > > and should be able to accommodate this. The downside is it's not
humanly
>> > > > > readable. The upside is that we don't need to do anything
special either to
>> > > > > write it or parse it.
>> > > > >
>> > > > > I would first try to use the existing archiver API, because that
makes it
>> > > > > harder to miss bugs. Any tension between that API and
pg_dumpall is likely to
>> > > > > have corresponding tension on the pg_restore side. Resolving
that tension
>> > > > > will reveal much of the project's scope that remained hidden
during the v18
>> > > > > attempt. Perhaps more important than that, using the archiver
API means
>> > > > > future pg_dump and pg_restore options are more likely to
cooperate properly
>> > > > > with $SUBJECT. In other words, I want it to be hard to add
pg_dump/pg_restore
>> > > > > features that malfunction only for $SUBJECT archives. The
strength of the
>> > > > > archiver architecture shows in how rarely new features need
format-specific
>> > > > > logic and how rarely format-specific bugs get reported. We've
had little or
>> > > > > no trouble with e.g. bugs that appear in -Fd but not in -Fc.
>> > > > >
>> > > > >
>> > > > > Yeah, that's what we're going to try.
>> > > > >
>> > > > >
>> > > > > cheers
>> > > > >
>> > > > >
>> > > > > andrew
>> > > > >
>> > > > > --
>> > > > > Andrew Dunstan
>> > > > > EDB: https://www.enterprisedb.com
>> > > >
>> > > > Thanks Andrew, Noah and all others for feedback.
>> > > >
>> > > > Based on the above suggestions and discussions, I removed sql
commands
>> > > > from the global.dat file. For global commands, now we are making
>> > > > toc.dat/toc.dmp/toc.tar file based on format specified and based on
>> > > > format specified, we are making archive entries for these global
>> > > > commands. By this approach, we removed the hard-coded parsing part
of
>> > > > the global.dat file and we are able to skip DROP DATABASE with the
>> > > > globals-only option.
>> > > >
>> > > > Here, I am attaching a patch for review, testing and feedback.
This is
>> > > > a WIP patch. I will do some more code cleanup and will add some
more
>> > > > comments also. Please review this and let me know design level
>> > > > feedback. Thanks Tushar Ahuja for some internal testing and
feedback.
>> > > >
>> > >
>> > > Hi,
>> > > Here, I am attaching an updated patch. In offline discussion, Andrew
>> > > reported some test-case failures(Thanks Andrew). I fixed those.
>> > > Please let me know feedback for the patch.
>> > >
>> >
>> > Hi,
>> > Here I am attaching a re-based patch as v02 was failing on head.
>> > Thanks Tushar for the testing.
>> > Please review this and let me know feedback.
>> >
>>
>> Hi all,
>> Here I am attaching an updated patch for review and testing. Based on
>> some offline comments by Andrew, I did some code cleanup.
>> Please consider this patch for feedback.
>>
>> --
>> Thanks and Regards
>> Mahendra Singh Thalor
>> EnterpriseDB: http://www.enterprisedb.com
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v05_03112025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (84.2K, 3-v05_03112025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
From 1bec0089809f9ba04b95b993caeefff068326c2d Mon Sep 17 00:00:00 2001
From: ThalorMahendra <[email protected]>
Date: Mon, 3 Nov 2025 17:17:09 +0530
Subject: [PATCH] Non text modes for pg_dumpall, correspondingly change
pg_restore
pg_dumpall acquires a new -F/--format option, with the same meanings as
pg_dump. The default is p, meaning plain text. For any other value, a
directory is created containing two files, toc.dat/.dmp/.tar and map.dat. The
first contains commands restoring the global data based on -F, and the second
contains a map from oids to database names. It will also contain a
subdirectory called databases, inside which it will create archives in
the specified format, named using the database oids.
In these casess the -f argument is required.
If pg_restore encounters a directory containing map.dat,
it restores the global settings from toc.dat/.dmp/.tar if exist, and then
restores each database.
pg_restore acquires two new options: -g/--globals-only which suppresses
restoration of any databases, and --exclude-database which inhibits
restoration of particualr database(s) in the same way the same option
works in pg_dumpall.
v05
---
doc/src/sgml/ref/pg_dumpall.sgml | 89 +++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/connectdb.c | 1 -
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 10 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 31 +-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 608 ++++++++++++++++++++++-----
src/bin/pg_dump/pg_restore.c | 593 +++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 10 +
src/bin/pg_dump/t/007_pg_dumpall.pl | 396 +++++++++++++++++
14 files changed, 1666 insertions(+), 146 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
create mode 100755 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 9f639f61db0..4063e88d388 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,10 +139,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>toc.dat/toc.dmp/toc.tar</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index a468a38361a..7497b527ae6 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -591,6 +615,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
index d55d53dbeea..f44a8a45fca 100644
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -287,7 +287,6 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
exit_nicely(1);
}
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index f3c669f484e..3e21aaf5780 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..5974d6706fd 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index d9041dad720..f631d945472 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -312,7 +312,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, bool globals_only);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 59eaecb4ed7..e4cfa9a963a 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, bool globals_only)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,17 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ /* Skip DROP DATABASE if globals_only. */
+ if (globals_only && te && te->tag && (strcmp(te->tag, "DROP_DATABASE") == 0))
+ continue;
+
+ /* Skip for RESTRICT, UNRESTRICT, CONNECT. */
+ if (!ropt->filename && te && te->tag &&
+ ((strcmp(te->tag, "RESTRICT") == 0) ||
+ (strcmp(te->tag, "UNRESTRICT") == 0) ||
+ (strcmp(te->tag, "CONNECT") == 0)))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1332,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1695,7 +1711,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1732,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..818b80a9369 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 47913178a93..00ce946aab1 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index bb451c1bae1..601b9f9738e 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -30,6 +30,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -65,9 +66,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,6 +78,9 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
+static void createOneArchiveEntry(const char *query, const char *tag);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -123,6 +128,13 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static const CatalogId nilCatalogId = {0, 0};
+static ArchiveMode archiveMode = archModeWrite;
+static DataDirSyncMethod sync_method = DATA_DIR_SYNC_METHOD_FSYNC;
+static ArchiveFormat archDumpFormat = archNull;
int
main(int argc, char *argv[])
@@ -148,6 +160,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +210,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -208,6 +222,8 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
+ char global_path[MAXPGPATH];
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -246,7 +262,9 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ InitDumpOptions(&dopt);
+
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -257,6 +275,7 @@ main(int argc, char *argv[])
case 'c':
output_clean = true;
+ dopt.outputClean = 1;
break;
case 'd':
@@ -274,7 +293,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -314,6 +335,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -429,6 +451,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -489,6 +526,35 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ /* set file path for global sql commands. */
+ if (archDumpFormat == archCustom)
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", filename);
+ else if (archDumpFormat == archTar)
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", filename);
+ else if (archDumpFormat == archDirectory)
+ snprintf(global_path, MAXPGPATH, "%s", filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -538,19 +604,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -585,37 +638,115 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
if (verbose)
dumpTimestamp("Started on");
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ /* create a archive file for global commands. */
+ if (filename && archDumpFormat != archNull)
+ {
+ /* Open the output file */
+ fout = CreateArchive(global_path, archDumpFormat, compression_spec,
+ dosync, archiveMode, NULL, sync_method);
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
+
+ ((ArchiveHandle * )fout) ->connection = conn;
+ ((ArchiveHandle * ) fout) -> public.numWorkers = 1;
+
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
+
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump\n--\n\n", "COMMENT");
+
+ /* create entry for restrict */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+ appendPQExpBuffer(qry, "\\restrict %s\n\n", restrict_key);
+ createOneArchiveEntry(qry->data, "RESTRICT");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* default_transaction_read_only = off */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving default_transaction_read_only = off");
+ appendPQExpBuffer(qry, "SET default_transaction_read_only = off;\n");
+ createOneArchiveEntry(qry->data, "DEFAULT_TRANSACTION_READ_ONLY");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpEncoding: put the correct encoding into the archive */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ const char *encname = pg_encoding_to_char(encoding);
- /* Replicate encoding and std_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
- if (strcmp(std_strings, "off") == 0)
- fprintf(OPF, "SET escape_string_warning = off;\n");
- fprintf(OPF, "\n");
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+
+ pg_log_info("saving encoding = %s", encname);
+ createOneArchiveEntry(qry->data, "ENCODING");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpStdStrings: put the correct escape string behavior into the archive */
+ {
+ const char *stdstrings = std_strings ? "on" : "off";
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving \"standard_conforming_strings = %s\"", stdstrings);
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = '%s';\n",
+ stdstrings);
+ createOneArchiveEntry(qry->data, "STDSTRINGS");
+ destroyPQExpBuffer(qry);
+ }
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and std_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
+ if (strcmp(std_strings, "off") == 0)
+ fprintf(OPF, "SET escape_string_warning = off;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -659,27 +790,51 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump will
+ * handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
+ else
+ {
+ /* create entry for unrestrict */
+ PQExpBuffer qry = createPQExpBuffer();
- if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ appendPQExpBuffer(qry, "\\unrestrict %s\n\n", restrict_key);
+ createOneArchiveEntry(qry->data, "UNRESTRICT");
+ destroyPQExpBuffer(qry);
+ }
- PQfinish(conn);
+ if (!globals_only && !roles_only && !tablespaces_only)
+ dumpDatabases(conn, archDumpFormat);
if (verbose)
dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (filename)
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+
+ if (archDumpFormat != archNull)
+ {
+ RestoreOptions *ropt;
+
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump complete\n--\n\n", "COMMENT");
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
+
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
+ }
+ else if (filename)
{
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -690,12 +845,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -770,6 +927,7 @@ static void
dropRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer delQry = createPQExpBuffer();
PGresult *res;
int i_rolname;
int i;
@@ -791,7 +949,12 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -799,15 +962,21 @@ dropRoles(PGconn *conn)
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(rolename));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropRoles");
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -889,7 +1058,12 @@ dumpRoles(PGconn *conn)
i_is_current_user = PQfnumber(res, "is_current_user");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -993,7 +1167,10 @@ dumpRoles(PGconn *conn)
"ROLE", rolename,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoles");
}
/*
@@ -1001,15 +1178,13 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
}
@@ -1088,7 +1263,12 @@ dumpRoleMembership(PGconn *conn)
i_set_option = PQfnumber(res, "set_option");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role memberships\n--\n\n", "COMMENT");
+ }
/*
* We can't dump these GRANT commands in arbitrary order, because a role
@@ -1167,6 +1347,7 @@ dumpRoleMembership(PGconn *conn)
char *grantor;
char *set_option = "true";
bool found;
+ PQExpBuffer creaQry = createPQExpBuffer();
/* If we already did this grant, don't do it again. */
if (done[i - start])
@@ -1223,8 +1404,8 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ appendPQExpBuffer(creaQry, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(creaQry, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1245,10 +1426,15 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(creaQry, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(creaQry, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(creaQry, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", creaQry->data);
+ else
+ createOneArchiveEntry(creaQry->data, "dumpRoleMembership");
}
}
@@ -1260,7 +1446,8 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1288,7 +1475,12 @@ dumpRoleGUCPrivs(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role privileges on configuration parameters\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1312,14 +1504,19 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoleGUCPrivs");
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1331,6 +1528,7 @@ dropTablespaces(PGconn *conn)
{
PGresult *res;
int i;
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Get all tablespaces except built-in ones (which we assume are named
@@ -1342,20 +1540,31 @@ dropTablespaces(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(spcname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropTablespaces");
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1382,7 +1591,12 @@ dumpTablespaces(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1451,14 +1665,19 @@ dumpTablespaces(PGconn *conn)
"TABLESPACE", spcname,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpTablespaces");
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1482,7 +1701,12 @@ dropDBs(PGconn *conn)
"ORDER BY datname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop databases (except postgres and template1)\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1497,15 +1721,23 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(dbname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "DROP_DATABASE");
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1532,7 +1764,18 @@ dumpUserConfig(PGconn *conn, const char *username)
char *sanitized;
sanitized = sanitize_line(username, true);
- fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
}
@@ -1542,7 +1785,11 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpUserConfig");
}
PQclear(res);
@@ -1608,10 +1855,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1625,19 +1875,48 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Databases\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Databases\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Databases\n--\n\n", "COMMENT");
+ }
+
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1654,7 +1933,18 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
/*
@@ -1669,24 +1959,46 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
+ fprintf(OPF, "\\connect %s\n\n", dbname);
else
{
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\\connect %s\n\n", dbname);
+ createOneArchiveEntry(qry->data, "CONNECT");
+ destroyPQExpBuffer(qry);
}
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1695,6 +2007,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1704,7 +2020,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1713,17 +2030,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1807,7 +2143,18 @@ dumpTimestamp(const char *msg)
time_t now = time(NULL);
if (strftime(buf, sizeof(buf), PGDUMP_STRFTIME_FMT, localtime(&now)) != 0)
- fprintf(OPF, "-- %s %s\n\n", msg, buf);
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "-- %s %s\n\n", msg, buf);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "-- %s %s\n\n", msg, buf);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+ }
}
/*
@@ -1868,3 +2215,54 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
+
+static void
+createOneArchiveEntry(const char *query, const char *tag)
+{
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = tag,
+ .section = SECTION_PRE_DATA,
+ .createStmt = query));
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c9776306c5c..02176a77bd7 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,31 +41,60 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num,
+ bool globals_only);
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only);
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +118,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +172,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +201,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +228,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -316,6 +350,9 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
case 6:
opts->restrict_key = pg_strdup(optarg);
@@ -347,6 +384,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -472,6 +516,105 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If map.dat file is present, then restore all the
+ * databases from map.dat , but skip restoring those matching
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "map.dat") ||
+ file_exists_in_directory(inputFileSpec, "toc.tar") ||
+ file_exists_in_directory(inputFileSpec, "toc.dmp")))
+ {
+ char global_path[MAXPGPATH];
+
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring an archive created by pg_dumpall");
+ else if (opts->tocFile)
+ pg_fatal("option -L/--use-list cannot be used when restoring an archive created by pg_dumpall");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option -C/--create must be specified when restoring an archive created by pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ n_errors = restore_global_objects(global_path, opts, numWorkers, false, 0, globals_only);
+
+ pg_log_info("database restoring skipped because option -g/--globals-only was specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else /* process if map.dat file does not exist. */
+ {
+ n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, 0, globals_only);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ *
+ * If globals_only is set, then skip DROP DATABASE commands from restore.
+ */
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ return restore_one_database(inputFileSpec, opts, numWorkers, append_data, num, globals_only);
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -479,9 +622,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -501,25 +650,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, globals_only);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -537,6 +682,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -553,6 +699,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -588,8 +735,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -694,3 +841,415 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn && db_exclude_patterns.head != NULL)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no database connection while doing pg_restore");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+ PQExpBuffer db_lit = createPQExpBuffer();
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches exclude pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ destroyPQExpBuffer(db_lit);
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ destroyPQExpBuffer(query);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+
+
+ /*
+ * If there is only global.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen <= 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbname, db_oid, map_file_path);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+ PGconn *conn = NULL;
+ char global_path[MAXPGPATH];
+
+ /* Based on file, set path. */
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
+
+ /* If map.dat has no entries, return after processing global commands. */
+ if (dbname_oid_list.head == NULL)
+ return restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Open toc.dat file and execute/append all the global sql commands. */
+ n_errors_total = restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ count++;
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..56e89da1e5e
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,12 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +250,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100755
index 00000000000..3c7d2ad7c53
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,396 @@
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# Scape tablespace locations on Windows.
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT super TO grant7 WITH INHERIT TRUE GRANTED BY\E
+ (.*\n)*
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: When --list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: When non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+$node->stop('fast');
+
+done_testing();
--
2.39.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-03 11:59 Vaibhav Dalvi <[email protected]>
parent: Vaibhav Dalvi <[email protected]>
1 sibling, 0 replies; 65+ messages in thread
From: Vaibhav Dalvi @ 2025-11-03 11:59 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: [email protected]
Hi Mahendra,
I have a few more review comments regarding the patch:
1. Is the following change in `src/bin/pg_dump/connectdb.c` intentional?
```
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -287,7 +287,6 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
exit_nicely(1);
}
```
When I re-added `PQfinish(conn);`, the regression tests passed successfully.
The `git diff` shows:
```
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
index f44a8a45fca..d55d53dbeea 100644
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -287,6 +287,7 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
exit_nicely(1);
}
```
If this change is intentional, could you please add a test case to justify
or demonstrate the need for it?
2. Please remove the extra blank line before `static void usage(const char
*progname);`.
```
+
static void usage(const char *progname);
```
3. There is an unnecessary line deletion that does not appear to be related
to this feature:
```
opts->cparams.pghost = pg_strdup(optarg);
break;
-
```
Could this deletion be part of a separate cleanup?
Regards,
Vaibhav Dalvi
On Mon, Nov 3, 2025 at 12:05 PM Vaibhav Dalvi <
[email protected]> wrote:
> Hi Mahendra,
>
> Thank you for your work on this feature.
> I have just begun reviewing the latest patch and
> encountered the following errors during the initial setup:
>
> ```
> $ ./db/bin/pg_restore testdump_dir -C -d postgres -F d -p 5556
> pg_restore: error: could not execute query: ERROR: syntax error at or near
> "\\"
> LINE 1: \restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj...
> ^
> Command was: \restrict
> aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
>
> pg_restore: error: could not execute query: ERROR: syntax error at or near
> "\\"
> LINE 1: \unrestrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCj...
> ^
> Command was: \unrestrict
> aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
>
> pg_restore: error: could not execute query: ERROR: syntax error at or near
> "\\"
> LINE 1: \connect template1
> ^
> Command was: \connect template1
>
> pg_restore: error: could not execute query: ERROR: syntax error at or near
> "\\"
> LINE 1: \connect postgres
> ^
> Command was: \connect postgres
> ```
> To cross-check tried with plain dump(with pg_dumpall) and
> restored(SQL file restore) without patch and didn't get above
> connection errors.
>
> It appears there might be an issue with the dump file itself.
> Please note that this is my first observation as I have just
> started the review. I will continue with my assessment.
>
> Regards,
> Vaibhav Dalvi
> EnterpriseDB
>
> On Fri, Oct 31, 2025 at 2:51 PM Mahendra Singh Thalor <[email protected]>
> wrote:
>
>> On Tue, 28 Oct 2025 at 11:32, Mahendra Singh Thalor <[email protected]>
>> wrote:
>> >
>> > On Thu, 16 Oct 2025 at 16:24, Mahendra Singh Thalor <[email protected]>
>> wrote:
>> > >
>> > > On Wed, 15 Oct 2025 at 23:05, Mahendra Singh Thalor <
>> [email protected]> wrote:
>> > > >
>> > > > On Sun, 24 Aug 2025 at 22:12, Andrew Dunstan <[email protected]>
>> wrote:
>> > > > >
>> > > > >
>> > > > > On 2025-08-23 Sa 9:08 PM, Noah Misch wrote:
>> > > > >
>> > > > > On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
>> > > > >
>> > > > > OK, now that's reverted we should discuss how to proceed. I had
>> two thoughts
>> > > > > - we could use invent a JSON format for the globals, or we could
>> just use
>> > > > > the existing archive format. I think the archive format is pretty
>> flexible,
>> > > > > and should be able to accommodate this. The downside is it's not
>> humanly
>> > > > > readable. The upside is that we don't need to do anything special
>> either to
>> > > > > write it or parse it.
>> > > > >
>> > > > > I would first try to use the existing archiver API, because that
>> makes it
>> > > > > harder to miss bugs. Any tension between that API and pg_dumpall
>> is likely to
>> > > > > have corresponding tension on the pg_restore side. Resolving
>> that tension
>> > > > > will reveal much of the project's scope that remained hidden
>> during the v18
>> > > > > attempt. Perhaps more important than that, using the archiver
>> API means
>> > > > > future pg_dump and pg_restore options are more likely to
>> cooperate properly
>> > > > > with $SUBJECT. In other words, I want it to be hard to add
>> pg_dump/pg_restore
>> > > > > features that malfunction only for $SUBJECT archives. The
>> strength of the
>> > > > > archiver architecture shows in how rarely new features need
>> format-specific
>> > > > > logic and how rarely format-specific bugs get reported. We've
>> had little or
>> > > > > no trouble with e.g. bugs that appear in -Fd but not in -Fc.
>> > > > >
>> > > > >
>> > > > > Yeah, that's what we're going to try.
>> > > > >
>> > > > >
>> > > > > cheers
>> > > > >
>> > > > >
>> > > > > andrew
>> > > > >
>> > > > > --
>> > > > > Andrew Dunstan
>> > > > > EDB: https://www.enterprisedb.com
>> > > >
>> > > > Thanks Andrew, Noah and all others for feedback.
>> > > >
>> > > > Based on the above suggestions and discussions, I removed sql
>> commands
>> > > > from the global.dat file. For global commands, now we are making
>> > > > toc.dat/toc.dmp/toc.tar file based on format specified and based on
>> > > > format specified, we are making archive entries for these global
>> > > > commands. By this approach, we removed the hard-coded parsing part
>> of
>> > > > the global.dat file and we are able to skip DROP DATABASE with the
>> > > > globals-only option.
>> > > >
>> > > > Here, I am attaching a patch for review, testing and feedback. This
>> is
>> > > > a WIP patch. I will do some more code cleanup and will add some more
>> > > > comments also. Please review this and let me know design level
>> > > > feedback. Thanks Tushar Ahuja for some internal testing and
>> feedback.
>> > > >
>> > >
>> > > Hi,
>> > > Here, I am attaching an updated patch. In offline discussion, Andrew
>> > > reported some test-case failures(Thanks Andrew). I fixed those.
>> > > Please let me know feedback for the patch.
>> > >
>> >
>> > Hi,
>> > Here I am attaching a re-based patch as v02 was failing on head.
>> > Thanks Tushar for the testing.
>> > Please review this and let me know feedback.
>> >
>>
>> Hi all,
>> Here I am attaching an updated patch for review and testing. Based on
>> some offline comments by Andrew, I did some code cleanup.
>> Please consider this patch for feedback.
>>
>> --
>> Thanks and Regards
>> Mahendra Singh Thalor
>> EnterpriseDB: http://www.enterprisedb.com
>>
>
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-04 12:53 tushar <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
1 sibling, 1 reply; 65+ messages in thread
From: tushar @ 2025-11-04 12:53 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
On Mon, Nov 3, 2025 at 5:25 PM Mahendra Singh Thalor <[email protected]>
wrote:
> On Mon, 3 Nov 2025 at 12:06, Vaibhav Dalvi <[email protected]>
> wrote:
> >
> > Hi Mahendra,
> >
> > Thank you for your work on this feature.
> > I have just begun reviewing the latest patch and
> > encountered the following errors during the initial setup:
> >
> > ```
> > $ ./db/bin/pg_restore testdump_dir -C -d postgres -F d -p 5556
> > pg_restore: error: could not execute query: ERROR: syntax error at or
> near "\\"
> > LINE 1: \restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj...
> > ^
> > Command was: \restrict
> aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
> >
> > pg_restore: error: could not execute query: ERROR: syntax error at or
> near "\\"
> > LINE 1: \unrestrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCj...
> > ^
> > Command was: \unrestrict
> aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
> >
> > pg_restore: error: could not execute query: ERROR: syntax error at or
> near "\\"
> > LINE 1: \connect template1
> > ^
> > Command was: \connect template1
> >
> > pg_restore: error: could not execute query: ERROR: syntax error at or
> near "\\"
> > LINE 1: \connect postgres
> > ^
> > Command was: \connect postgres
> > ```
> > To cross-check tried with plain dump(with pg_dumpall) and
> > restored(SQL file restore) without patch and didn't get above
> > connection errors.
> >
> > It appears there might be an issue with the dump file itself.
> > Please note that this is my first observation as I have just
> > started the review. I will continue with my assessment.
> >
> > Regards,
> > Vaibhav Dalvi
> > EnterpriseDB
>
> Thanks Vaibhav for the review.
> This change was added by me in v04. Only in the case of a file, we should
> restore these commands. Attached patch is fixing the same.
>
> Thanks Mahendra, I am getting a segmentation fault against v05 patch.
[edb@1a1c15437e7c bin]$ ./pg_dumpall -Ft --file a.3 -v
pg_dumpall: executing SELECT pg_catalog.set_config('search_path', '',
false);
Segmentation fault
Issue is coming with all output file formats -F[t/c/d] except plain
regards,
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-04 16:55 Andrew Dunstan <[email protected]>
parent: tushar <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Andrew Dunstan @ 2025-11-04 16:55 UTC (permalink / raw)
To: tushar <[email protected]>; Mahendra Singh Thalor <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
On 2025-11-04 Tu 7:53 AM, tushar wrote:
>
>
> On Mon, Nov 3, 2025 at 5:25 PM Mahendra Singh Thalor
> <[email protected]> wrote:
>
> On Mon, 3 Nov 2025 at 12:06, Vaibhav Dalvi
> <[email protected]> wrote:
> >
> > Hi Mahendra,
> >
> > Thank you for your work on this feature.
> > I have just begun reviewing the latest patch and
> > encountered the following errors during the initial setup:
> >
> > ```
> > $ ./db/bin/pg_restore testdump_dir -C -d postgres -F d -p 5556
> > pg_restore: error: could not execute query: ERROR: syntax error
> at or near "\\"
> > LINE 1: \restrict
> aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj...
> > ^
> > Command was: \restrict
> aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
> >
> > pg_restore: error: could not execute query: ERROR: syntax error
> at or near "\\"
> > LINE 1: \unrestrict
> aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCj...
> > ^
> > Command was: \unrestrict
> aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
> >
> > pg_restore: error: could not execute query: ERROR: syntax error
> at or near "\\"
> > LINE 1: \connect template1
> > ^
> > Command was: \connect template1
> >
> > pg_restore: error: could not execute query: ERROR: syntax error
> at or near "\\"
> > LINE 1: \connect postgres
> > ^
> > Command was: \connect postgres
> > ```
> > To cross-check tried with plain dump(with pg_dumpall) and
> > restored(SQL file restore) without patch and didn't get above
> > connection errors.
> >
> > It appears there might be an issue with the dump file itself.
> > Please note that this is my first observation as I have just
> > started the review. I will continue with my assessment.
> >
> > Regards,
> > Vaibhav Dalvi
> > EnterpriseDB
>
> Thanks Vaibhav for the review.
> This change was added by me in v04. Only in the case of a file, we
> should restore these commands. Attached patch is fixing the same.
>
> Thanks Mahendra, I am getting a segmentation fault against v05 patch.
>
> [edb@1a1c15437e7c bin]$ ./pg_dumpall -Ft --file a.3 -v
> pg_dumpall: executing SELECT pg_catalog.set_config('search_path', '',
> false);
> Segmentation fault
>
> Issue is coming with all output file formats -F[t/c/d] except plain
>
>
Yeah, I don't think we need to dump the timestamp in non-text modes.
This fix worked for me:
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 601b9f9738e..f66cc26d9a2 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -638,7 +638,7 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- if (verbose)
+ if (verbose && archDumpFormat == archNull)
dumpTimestamp("Started on");
/* create a archive file for global commands. */
@@ -2258,6 +2258,7 @@ createDumpId(void)
static void
createOneArchiveEntry(const char *query, const char *tag)
{
+ Assert(fout != NULL);
ArchiveEntry(fout,
nilCatalogId, /* catalog ID */
createDumpId(), /* dump ID */
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-05 06:59 Vaibhav Dalvi <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
1 sibling, 1 reply; 65+ messages in thread
From: Vaibhav Dalvi @ 2025-11-05 06:59 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: [email protected]
Hi Mahendra,
Thank you for the fix. Please find my further review comments below.
### Restrict-Key Option
The `--restrict-key` option is currently being accepted by
`pg_dumpall` even when non-plain formats are specified,
which contradicts its intended use only with the plain format.
For example:
```
$ ./db/bin/pg_dump --format=d -f testdump_dir --restrict-key=RESTRICT_KEY
pg_dump: error: option --restrict-key can only be used with --format=plain
$ ./db/bin/pg_dumpall --format=d -f testdump_dir --restrict-key=RESTRICT_KEY
pg_dumpall: error: invalid restrict key
```
I have attached a delta patch that addresses the issue with the
`--restrict-key` option. It would be beneficial to include a dedicated
test case for this check.
### Use of Dump Options Structure (dopt)
Please ensure consistency by utilizing the main dump options
structure (`dopt`) instead of declaring and using individual variables
where the structure already provides fields. For example, the
`output_clean` variable seems redundant here:
```c
case 'c':
output_clean = true;
dopt.outputClean = 1;
break;
```
In my attached delta file, I have replaced the unnecessary
`restrict_key` variable with `dopt.restrict_key`.
### Cosmetic Issues
1. Please review the spacing around the pointer:
```c
+ ((ArchiveHandle * )fout) ->connection = conn;
+ ((ArchiveHandle * ) fout) -> public.numWorkers = 1;
```
2. Please be consistent with the punctuation of single-line comments;
some end with a full stop (`.`) and others do not.
3. In the SGML documentation changes, some new statements start
with one space, and others start with two. Please adhere to a single
standard for indentation across the patch.
Regards,
Vaibhav
EnterpriseDB
On Mon, Nov 3, 2025 at 5:24 PM Mahendra Singh Thalor <[email protected]>
wrote:
> On Mon, 3 Nov 2025 at 12:06, Vaibhav Dalvi <[email protected]>
> wrote:
> >
> > Hi Mahendra,
> >
> > Thank you for your work on this feature.
> > I have just begun reviewing the latest patch and
> > encountered the following errors during the initial setup:
> >
> > ```
> > $ ./db/bin/pg_restore testdump_dir -C -d postgres -F d -p 5556
> > pg_restore: error: could not execute query: ERROR: syntax error at or
> near "\\"
> > LINE 1: \restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj...
> > ^
> > Command was: \restrict
> aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
> >
> > pg_restore: error: could not execute query: ERROR: syntax error at or
> near "\\"
> > LINE 1: \unrestrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCj...
> > ^
> > Command was: \unrestrict
> aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
> >
> > pg_restore: error: could not execute query: ERROR: syntax error at or
> near "\\"
> > LINE 1: \connect template1
> > ^
> > Command was: \connect template1
> >
> > pg_restore: error: could not execute query: ERROR: syntax error at or
> near "\\"
> > LINE 1: \connect postgres
> > ^
> > Command was: \connect postgres
> > ```
> > To cross-check tried with plain dump(with pg_dumpall) and
> > restored(SQL file restore) without patch and didn't get above
> > connection errors.
> >
> > It appears there might be an issue with the dump file itself.
> > Please note that this is my first observation as I have just
> > started the review. I will continue with my assessment.
> >
> > Regards,
> > Vaibhav Dalvi
> > EnterpriseDB
>
> Thanks Vaibhav for the review.
> This change was added by me in v04. Only in the case of a file, we should
> restore these commands. Attached patch is fixing the same.
>
> If we dump and restore the same file with the same user, then we will get
> an error of ROLE CREATE as the same role is already created. I think,
> either we can ignore this error, or we can keep it as a restore can be done
> with different users.
>
>> mst@localhost bin]$ ./pg_restore d1 -C -d postgres
>> pg_restore: error: could not execute query: ERROR: role "mst" already
>> exists
>> Command was: CREATE ROLE mst;
>> ALTER ROLE mst WITH SUPERUSER INHERIT CREATEROLE CREATEDB LOGIN
>> REPLICATION BYPASSRLS;
>>
>>
>> pg_restore: warning: errors ignored on restore: 1
>
>
>
> >
> > On Fri, Oct 31, 2025 at 2:51 PM Mahendra Singh Thalor <
> [email protected]> wrote:
> >>
> >> On Tue, 28 Oct 2025 at 11:32, Mahendra Singh Thalor <[email protected]>
> wrote:
> >> >
> >> > On Thu, 16 Oct 2025 at 16:24, Mahendra Singh Thalor <
> [email protected]> wrote:
> >> > >
> >> > > On Wed, 15 Oct 2025 at 23:05, Mahendra Singh Thalor <
> [email protected]> wrote:
> >> > > >
> >> > > > On Sun, 24 Aug 2025 at 22:12, Andrew Dunstan <[email protected]>
> wrote:
> >> > > > >
> >> > > > >
> >> > > > > On 2025-08-23 Sa 9:08 PM, Noah Misch wrote:
> >> > > > >
> >> > > > > On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
> >> > > > >
> >> > > > > OK, now that's reverted we should discuss how to proceed. I had
> two thoughts
> >> > > > > - we could use invent a JSON format for the globals, or we
> could just use
> >> > > > > the existing archive format. I think the archive format is
> pretty flexible,
> >> > > > > and should be able to accommodate this. The downside is it's
> not humanly
> >> > > > > readable. The upside is that we don't need to do anything
> special either to
> >> > > > > write it or parse it.
> >> > > > >
> >> > > > > I would first try to use the existing archiver API, because
> that makes it
> >> > > > > harder to miss bugs. Any tension between that API and
> pg_dumpall is likely to
> >> > > > > have corresponding tension on the pg_restore side. Resolving
> that tension
> >> > > > > will reveal much of the project's scope that remained hidden
> during the v18
> >> > > > > attempt. Perhaps more important than that, using the archiver
> API means
> >> > > > > future pg_dump and pg_restore options are more likely to
> cooperate properly
> >> > > > > with $SUBJECT. In other words, I want it to be hard to add
> pg_dump/pg_restore
> >> > > > > features that malfunction only for $SUBJECT archives. The
> strength of the
> >> > > > > archiver architecture shows in how rarely new features need
> format-specific
> >> > > > > logic and how rarely format-specific bugs get reported. We've
> had little or
> >> > > > > no trouble with e.g. bugs that appear in -Fd but not in -Fc.
> >> > > > >
> >> > > > >
> >> > > > > Yeah, that's what we're going to try.
> >> > > > >
> >> > > > >
> >> > > > > cheers
> >> > > > >
> >> > > > >
> >> > > > > andrew
> >> > > > >
> >> > > > > --
> >> > > > > Andrew Dunstan
> >> > > > > EDB: https://www.enterprisedb.com
> >> > > >
> >> > > > Thanks Andrew, Noah and all others for feedback.
> >> > > >
> >> > > > Based on the above suggestions and discussions, I removed sql
> commands
> >> > > > from the global.dat file. For global commands, now we are making
> >> > > > toc.dat/toc.dmp/toc.tar file based on format specified and based
> on
> >> > > > format specified, we are making archive entries for these global
> >> > > > commands. By this approach, we removed the hard-coded parsing
> part of
> >> > > > the global.dat file and we are able to skip DROP DATABASE with the
> >> > > > globals-only option.
> >> > > >
> >> > > > Here, I am attaching a patch for review, testing and feedback.
> This is
> >> > > > a WIP patch. I will do some more code cleanup and will add some
> more
> >> > > > comments also. Please review this and let me know design level
> >> > > > feedback. Thanks Tushar Ahuja for some internal testing and
> feedback.
> >> > > >
> >> > >
> >> > > Hi,
> >> > > Here, I am attaching an updated patch. In offline discussion, Andrew
> >> > > reported some test-case failures(Thanks Andrew). I fixed those.
> >> > > Please let me know feedback for the patch.
> >> > >
> >> >
> >> > Hi,
> >> > Here I am attaching a re-based patch as v02 was failing on head.
> >> > Thanks Tushar for the testing.
> >> > Please review this and let me know feedback.
> >> >
> >>
> >> Hi all,
> >> Here I am attaching an updated patch for review and testing. Based on
> >> some offline comments by Andrew, I did some code cleanup.
> >> Please consider this patch for feedback.
> >>
> >> --
> >> Thanks and Regards
> >> Mahendra Singh Thalor
> >> EnterpriseDB: http://www.enterprisedb.com
>
>
>
> --
> Thanks and Regards
> Mahendra Singh Thalor
> EnterpriseDB: http://www.enterprisedb.com
>
Attachments:
[application/octet-stream] delta-v05-non-text-modes-for-pg_dumpall.patch (3.7K, 3-delta-v05-non-text-modes-for-pg_dumpall.patch)
download | inline diff:
From 4debd839b5dbbfd188ad2422112f8e303c5d7a71 Mon Sep 17 00:00:00 2001
From: Vaibhav Dalvi <[email protected]>
Date: Wed, 5 Nov 2025 06:22:00 +0000
Subject: [PATCH v1 1/1] delta v05 non-text modes for pg_dumpall
This delta patch is to fix --restrict-key
with non-text dump format.
Vaibhav Dalvi
---
src/bin/pg_dump/pg_dumpall.c | 52 +++++++++++++-----------------------
1 file changed, 19 insertions(+), 33 deletions(-)
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 601b9f9738e..9e447dc9738 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -127,7 +127,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-static char *restrict_key;
static Archive *fout = NULL;
static pg_compress_specification compression_spec = {0};
static int dumpIdVal = 0;
@@ -397,7 +396,7 @@ main(int argc, char *argv[])
break;
case 9:
- restrict_key = pg_strdup(optarg);
+ dopt.restrict_key = pg_strdup(optarg);
appendPQExpBufferStr(pgdumpopts, " --restrict-key ");
appendShellString(pgdumpopts, optarg);
break;
@@ -555,15 +554,20 @@ main(int argc, char *argv[])
else
OPF = stdout;
- /*
- * If you don't provide a restrict key, one will be appointed for you.
- */
- if (!restrict_key)
- restrict_key = generate_restrict_key();
- if (!restrict_key)
- pg_fatal("could not generate restrict key");
- if (!valid_restrict_key(restrict_key))
- pg_fatal("invalid restrict key");
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * If you don't provide a restrict key, one will be appointed for you.
+ */
+ if (!dopt.restrict_key)
+ dopt.restrict_key = generate_restrict_key();
+ if (!dopt.restrict_key)
+ pg_fatal("could not generate restrict key");
+ if (!valid_restrict_key(dopt.restrict_key))
+ pg_fatal("invalid restrict key");
+ }
+ else if (dopt.restrict_key)
+ pg_fatal("option --restrict-key can only be used with --format=plain");
/*
* If there was a database specified on the command line, use that,
@@ -670,15 +674,6 @@ main(int argc, char *argv[])
createOneArchiveEntry("--\n-- PostgreSQL database cluster dump\n--\n\n", "COMMENT");
- /* create entry for restrict */
- {
- PQExpBuffer qry = createPQExpBuffer();
-
- appendPQExpBuffer(qry, "\\restrict %s\n\n", restrict_key);
- createOneArchiveEntry(qry->data, "RESTRICT");
- destroyPQExpBuffer(qry);
- }
-
/* default_transaction_read_only = off */
{
PQExpBuffer qry = createPQExpBuffer();
@@ -727,7 +722,7 @@ main(int argc, char *argv[])
* meta-commands so that the client machine that runs psql with the dump
* output remains unaffected.
*/
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ fprintf(OPF, "\\restrict %s\n\n", dopt.restrict_key);
/*
* We used to emit \connect postgres here, but that served no purpose
@@ -793,19 +788,10 @@ main(int argc, char *argv[])
if (archDumpFormat == archNull)
{
/*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
+ * Exit restricted mode just before dumping the databases. pg_dump
+ * will handle entering restricted mode again as appropriate.
*/
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
- }
- else
- {
- /* create entry for unrestrict */
- PQExpBuffer qry = createPQExpBuffer();
-
- appendPQExpBuffer(qry, "\\unrestrict %s\n\n", restrict_key);
- createOneArchiveEntry(qry->data, "UNRESTRICT");
- destroyPQExpBuffer(qry);
+ fprintf(OPF, "\\unrestrict %s\n\n", dopt.restrict_key);
}
if (!globals_only && !roles_only && !tablespaces_only)
--
2.43.0
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-05 13:16 Vaibhav Dalvi <[email protected]>
parent: Vaibhav Dalvi <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Vaibhav Dalvi @ 2025-11-05 13:16 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: [email protected]
Hi Mahendra,
Here are a few more comments following my review of the patch:
### 1\. Incorrect Comment for `-g` (globals-only) Option
The comment for the `-g` case in the code states that it restores the
`global.dat` file. However, in the non-text dump output, I only see the
following files: `databases`, `map.dat`, and `toc.dat`.
```c
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
```
Please update this comment to accurately reflect the file being restored
(e.g., `toc.dat` or the global objects within the archive).
### 2\. Incorrect Order of `case` Statements in `pg_restore.c`
The new `case 7` statement in `pg_restore.c` appears to be
inserted before `case 6`, disrupting the numerical order.
```c
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
case 6:
opts->restrict_key = pg_strdup(optarg);
```
Please re-order the `case` statements so they follow ascending
numerical order.
### 3\. Missing Example in SGML Documentation
The SGML documentation for `pg_dumpall` is missing an explicit
example demonstrating its use with non-text formats (e.g., directory
format).
It would be beneficial to include a clear example for this new feature.
### 4\. Cosmetic Issues
Please address the following minor stylistic points:
Please ensure the function signatures
adhere to standard coding style, particularly for line wrapping.
The following lines seem to have inconsistent indentation:
```c
static int restore_global_objects(const char *inputFileSpec, RestoreOptions
*opts,
int numWorkers, bool append_data, int num, bool globals_only);
static int restore_all_databases(const char *inputFileSpec,
SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
```
Please fix instances where the 80-character line limit is
crossed, such as in the example below:
```c
n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1,
false);
```
I believe this concludes my formal review.
Thanks,
Vaibhav Dalvi
On Wed, Nov 5, 2025 at 12:29 PM Vaibhav Dalvi <
[email protected]> wrote:
> Hi Mahendra,
>
> Thank you for the fix. Please find my further review comments below.
>
> ### Restrict-Key Option
>
> The `--restrict-key` option is currently being accepted by
> `pg_dumpall` even when non-plain formats are specified,
> which contradicts its intended use only with the plain format.
>
> For example:
>
> ```
> $ ./db/bin/pg_dump --format=d -f testdump_dir --restrict-key=RESTRICT_KEY
> pg_dump: error: option --restrict-key can only be used with --format=plain
> $ ./db/bin/pg_dumpall --format=d -f testdump_dir
> --restrict-key=RESTRICT_KEY
> pg_dumpall: error: invalid restrict key
> ```
>
> I have attached a delta patch that addresses the issue with the
> `--restrict-key` option. It would be beneficial to include a dedicated
> test case for this check.
>
> ### Use of Dump Options Structure (dopt)
>
> Please ensure consistency by utilizing the main dump options
> structure (`dopt`) instead of declaring and using individual variables
> where the structure already provides fields. For example, the
> `output_clean` variable seems redundant here:
>
> ```c
> case 'c':
> output_clean = true;
> dopt.outputClean = 1;
> break;
> ```
>
> In my attached delta file, I have replaced the unnecessary
> `restrict_key` variable with `dopt.restrict_key`.
>
> ### Cosmetic Issues
>
> 1. Please review the spacing around the pointer:
> ```c
> + ((ArchiveHandle * )fout) ->connection = conn;
> + ((ArchiveHandle * ) fout) -> public.numWorkers = 1;
> ```
> 2. Please be consistent with the punctuation of single-line comments;
> some end with a full stop (`.`) and others do not.
> 3. In the SGML documentation changes, some new statements start
> with one space, and others start with two. Please adhere to a single
> standard for indentation across the patch.
>
> Regards,
> Vaibhav
> EnterpriseDB
>
> On Mon, Nov 3, 2025 at 5:24 PM Mahendra Singh Thalor <[email protected]>
> wrote:
>
>> On Mon, 3 Nov 2025 at 12:06, Vaibhav Dalvi <
>> [email protected]> wrote:
>> >
>> > Hi Mahendra,
>> >
>> > Thank you for your work on this feature.
>> > I have just begun reviewing the latest patch and
>> > encountered the following errors during the initial setup:
>> >
>> > ```
>> > $ ./db/bin/pg_restore testdump_dir -C -d postgres -F d -p 5556
>> > pg_restore: error: could not execute query: ERROR: syntax error at or
>> near "\\"
>> > LINE 1: \restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj...
>> > ^
>> > Command was: \restrict
>> aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
>> >
>> > pg_restore: error: could not execute query: ERROR: syntax error at or
>> near "\\"
>> > LINE 1: \unrestrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCj...
>> > ^
>> > Command was: \unrestrict
>> aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
>> >
>> > pg_restore: error: could not execute query: ERROR: syntax error at or
>> near "\\"
>> > LINE 1: \connect template1
>> > ^
>> > Command was: \connect template1
>> >
>> > pg_restore: error: could not execute query: ERROR: syntax error at or
>> near "\\"
>> > LINE 1: \connect postgres
>> > ^
>> > Command was: \connect postgres
>> > ```
>> > To cross-check tried with plain dump(with pg_dumpall) and
>> > restored(SQL file restore) without patch and didn't get above
>> > connection errors.
>> >
>> > It appears there might be an issue with the dump file itself.
>> > Please note that this is my first observation as I have just
>> > started the review. I will continue with my assessment.
>> >
>> > Regards,
>> > Vaibhav Dalvi
>> > EnterpriseDB
>>
>> Thanks Vaibhav for the review.
>> This change was added by me in v04. Only in the case of a file, we should
>> restore these commands. Attached patch is fixing the same.
>>
>> If we dump and restore the same file with the same user, then we will get
>> an error of ROLE CREATE as the same role is already created. I think,
>> either we can ignore this error, or we can keep it as a restore can be done
>> with different users.
>>
>>> mst@localhost bin]$ ./pg_restore d1 -C -d postgres
>>> pg_restore: error: could not execute query: ERROR: role "mst" already
>>> exists
>>> Command was: CREATE ROLE mst;
>>> ALTER ROLE mst WITH SUPERUSER INHERIT CREATEROLE CREATEDB LOGIN
>>> REPLICATION BYPASSRLS;
>>>
>>>
>>> pg_restore: warning: errors ignored on restore: 1
>>
>>
>>
>> >
>> > On Fri, Oct 31, 2025 at 2:51 PM Mahendra Singh Thalor <
>> [email protected]> wrote:
>> >>
>> >> On Tue, 28 Oct 2025 at 11:32, Mahendra Singh Thalor <
>> [email protected]> wrote:
>> >> >
>> >> > On Thu, 16 Oct 2025 at 16:24, Mahendra Singh Thalor <
>> [email protected]> wrote:
>> >> > >
>> >> > > On Wed, 15 Oct 2025 at 23:05, Mahendra Singh Thalor <
>> [email protected]> wrote:
>> >> > > >
>> >> > > > On Sun, 24 Aug 2025 at 22:12, Andrew Dunstan <
>> [email protected]> wrote:
>> >> > > > >
>> >> > > > >
>> >> > > > > On 2025-08-23 Sa 9:08 PM, Noah Misch wrote:
>> >> > > > >
>> >> > > > > On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
>> >> > > > >
>> >> > > > > OK, now that's reverted we should discuss how to proceed. I
>> had two thoughts
>> >> > > > > - we could use invent a JSON format for the globals, or we
>> could just use
>> >> > > > > the existing archive format. I think the archive format is
>> pretty flexible,
>> >> > > > > and should be able to accommodate this. The downside is it's
>> not humanly
>> >> > > > > readable. The upside is that we don't need to do anything
>> special either to
>> >> > > > > write it or parse it.
>> >> > > > >
>> >> > > > > I would first try to use the existing archiver API, because
>> that makes it
>> >> > > > > harder to miss bugs. Any tension between that API and
>> pg_dumpall is likely to
>> >> > > > > have corresponding tension on the pg_restore side. Resolving
>> that tension
>> >> > > > > will reveal much of the project's scope that remained hidden
>> during the v18
>> >> > > > > attempt. Perhaps more important than that, using the archiver
>> API means
>> >> > > > > future pg_dump and pg_restore options are more likely to
>> cooperate properly
>> >> > > > > with $SUBJECT. In other words, I want it to be hard to add
>> pg_dump/pg_restore
>> >> > > > > features that malfunction only for $SUBJECT archives. The
>> strength of the
>> >> > > > > archiver architecture shows in how rarely new features need
>> format-specific
>> >> > > > > logic and how rarely format-specific bugs get reported. We've
>> had little or
>> >> > > > > no trouble with e.g. bugs that appear in -Fd but not in -Fc.
>> >> > > > >
>> >> > > > >
>> >> > > > > Yeah, that's what we're going to try.
>> >> > > > >
>> >> > > > >
>> >> > > > > cheers
>> >> > > > >
>> >> > > > >
>> >> > > > > andrew
>> >> > > > >
>> >> > > > > --
>> >> > > > > Andrew Dunstan
>> >> > > > > EDB: https://www.enterprisedb.com
>> >> > > >
>> >> > > > Thanks Andrew, Noah and all others for feedback.
>> >> > > >
>> >> > > > Based on the above suggestions and discussions, I removed sql
>> commands
>> >> > > > from the global.dat file. For global commands, now we are making
>> >> > > > toc.dat/toc.dmp/toc.tar file based on format specified and based
>> on
>> >> > > > format specified, we are making archive entries for these global
>> >> > > > commands. By this approach, we removed the hard-coded parsing
>> part of
>> >> > > > the global.dat file and we are able to skip DROP DATABASE with
>> the
>> >> > > > globals-only option.
>> >> > > >
>> >> > > > Here, I am attaching a patch for review, testing and feedback.
>> This is
>> >> > > > a WIP patch. I will do some more code cleanup and will add some
>> more
>> >> > > > comments also. Please review this and let me know design level
>> >> > > > feedback. Thanks Tushar Ahuja for some internal testing and
>> feedback.
>> >> > > >
>> >> > >
>> >> > > Hi,
>> >> > > Here, I am attaching an updated patch. In offline discussion,
>> Andrew
>> >> > > reported some test-case failures(Thanks Andrew). I fixed those.
>> >> > > Please let me know feedback for the patch.
>> >> > >
>> >> >
>> >> > Hi,
>> >> > Here I am attaching a re-based patch as v02 was failing on head.
>> >> > Thanks Tushar for the testing.
>> >> > Please review this and let me know feedback.
>> >> >
>> >>
>> >> Hi all,
>> >> Here I am attaching an updated patch for review and testing. Based on
>> >> some offline comments by Andrew, I did some code cleanup.
>> >> Please consider this patch for feedback.
>> >>
>> >> --
>> >> Thanks and Regards
>> >> Mahendra Singh Thalor
>> >> EnterpriseDB: http://www.enterprisedb.com
>>
>>
>>
>> --
>> Thanks and Regards
>> Mahendra Singh Thalor
>> EnterpriseDB: http://www.enterprisedb.com
>>
>
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-06 05:33 Mahendra Singh Thalor <[email protected]>
parent: Vaibhav Dalvi <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-11-06 05:33 UTC (permalink / raw)
To: Vaibhav Dalvi <[email protected]>; +Cc: [email protected]
Thanks Vaibhav, Tushar and Andrew for the review and testing.
On Mon, 3 Nov 2025 at 17:30, Vaibhav Dalvi
<[email protected]> wrote:
>
> Hi Mahendra,
>
> I have a few more review comments regarding the patch:
>
> 1. Is the following change in `src/bin/pg_dump/connectdb.c` intentional?
>
> ```
> --- a/src/bin/pg_dump/connectdb.c
> +++ b/src/bin/pg_dump/connectdb.c
Yes, we need this. If there is any error, then we were trying to
disconnect the database in 2 places so we were getting a crash. I will
try to reproduce crashe without this patch and will respond.
On Tue, 4 Nov 2025 at 18:23, tushar <[email protected]> wrote:
> Thanks Mahendra, I am getting a segmentation fault against v05 patch.
>
> [edb@1a1c15437e7c bin]$ ./pg_dumpall -Ft --file a.3 -v
> pg_dumpall: executing SELECT pg_catalog.set_config('search_path', '', false);
> Segmentation fault
>
> Issue is coming with all output file formats -F[t/c/d] except plain
>
> regards,
Thanks for the report. Fixed,
On Tue, 4 Nov 2025 at 22:25, Andrew Dunstan <[email protected]> wrote:
> Yeah, I don't think we need to dump the timestamp in non-text modes. This fix worked for me:
>
>
> diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
> index 601b9f9738e..f66cc26d9a2 100644
> --- a/src/bin/pg_dump/pg_dumpall.c
> +++ b/src/bin/pg_dump/pg_dumpall.c
> @@ -638,7 +638,7 @@ main(int argc, char *argv[])
> if (quote_all_identifiers)
> executeCommand(conn, "SET quote_all_identifiers = true");
>
> - if (verbose)
> + if (verbose && archDumpFormat == archNull)
> dumpTimestamp("Started on");
Thanks Andrew. Yes, we should not dump timestamp in non-text modes.
On Wed, 5 Nov 2025 at 18:47, Vaibhav Dalvi
<[email protected]> wrote:
>
> Hi Mahendra,
>
> Here are a few more comments following my review of the patch:
>
> ### 1\. Incorrect Comment for `-g` (globals-only) Option
>
> The comment for the `-g` case in the code states that it restores the
> `global.dat` file. However, in the non-text dump output, I only see the
> following files: `databases`, `map.dat`, and `toc.dat`.
Fixed.
>
> ```c
> + case 'g':
> + /* restore only global.dat file from directory */
> + globals_only = true;
> + break;
Fixed.
> ```
>
> Please update this comment to accurately reflect the file being restored
> (e.g., `toc.dat` or the global objects within the archive).
Fixed.
>
> ### 2\. Incorrect Order of `case` Statements in `pg_restore.c`
>
> The new `case 7` statement in `pg_restore.c` appears to be
> inserted before `case 6`, disrupting the numerical order.
>
> ```c
> + case 7: /* database patterns to skip */
> + simple_string_list_append(&db_exclude_patterns, optarg);
> + break;
>
> case 6:
> opts->restrict_key = pg_strdup(optarg);
> ```
>
> Please re-order the `case` statements so they follow ascending
> numerical order.
Fixed.
>
> ### 3\. Missing Example in SGML Documentation
>
> The SGML documentation for `pg_dumpall` is missing an explicit
> example demonstrating its use with non-text formats (e.g., directory format).
> It would be beneficial to include a clear example for this new feature.
I think we don't add such cases in doc. We already added test cases in
code. If others also feel that we should add a test case in SGML, then
I will update the doc with the test case.
>
> ### 4\. Cosmetic Issues
>
> Please address the following minor stylistic points:
>
> Please ensure the function signatures
> adhere to standard coding style, particularly for line wrapping.
> The following lines seem to have inconsistent indentation:
>
> ```c
> static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
> int numWorkers, bool append_data, int num, bool globals_only);
> static int restore_all_databases(const char *inputFileSpec,
> SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
> ```
>
> Please fix instances where the 80-character line limit is
> crossed, such as in the example below:
Fixed.
>
> ```c
> n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
> ```
>
> I believe this concludes my formal review.
>
> Thanks,
> Vaibhav Dalvi
>
> On Wed, Nov 5, 2025 at 12:29 PM Vaibhav Dalvi <[email protected]> wrote:
>>
>> Hi Mahendra,
>>
>> Thank you for the fix. Please find my further review comments below.
>>
>> ### Restrict-Key Option
>>
>> The `--restrict-key` option is currently being accepted by
>> `pg_dumpall` even when non-plain formats are specified,
>> which contradicts its intended use only with the plain format.
>>
>> For example:
>>
>> ```
>> $ ./db/bin/pg_dump --format=d -f testdump_dir --restrict-key=RESTRICT_KEY
>> pg_dump: error: option --restrict-key can only be used with --format=plain
>> $ ./db/bin/pg_dumpall --format=d -f testdump_dir --restrict-key=RESTRICT_KEY
>> pg_dumpall: error: invalid restrict key
>> ```
>>
>> I have attached a delta patch that addresses the issue with the
>> `--restrict-key` option. It would be beneficial to include a dedicated
>> test case for this check.
We should dump restrict-key with all modes as we need to restore with
the "-f file" option in text mode.
Ex: pg_dumpall --format=d -f testdump_dir
and restore::: pg_restore testdump_dir -d dabasename -C -f testdumpfile
(In testdumpfile, we will generate commands from archive dump)
So I am not merging this delat patch.
>>
>> ### Use of Dump Options Structure (dopt)
>>
>> Please ensure consistency by utilizing the main dump options
>> structure (`dopt`) instead of declaring and using individual variables
>> where the structure already provides fields. For example, the
>> `output_clean` variable seems redundant here:
>>
>> ```c
>> case 'c':
>> output_clean = true;
>> dopt.outputClean = 1;
>> break;
output_clean is not added by this patch. I will analyse this comment
and will respond in the next update.
>> ```
>>
>> In my attached delta file, I have replaced the unnecessary
>> `restrict_key` variable with `dopt.restrict_key`.
This is also not part of this patch. If you feel to add this in DOPT,
please suggest in separate thread.
>>
>> ### Cosmetic Issues
>>
>> 1. Please review the spacing around the pointer:
>> ```c
>> + ((ArchiveHandle * )fout) ->connection = conn;
>> + ((ArchiveHandle * ) fout) -> public.numWorkers = 1;
Fixed.
>> ```
>> 2. Please be consistent with the punctuation of single-line comments;
>> some end with a full stop (`.`) and others do not.
Based on nearby code comments, I made changes. I will try to fix these
inconsistencies..
>> 3. In the SGML documentation changes, some new statements start
>> with one space, and others start with two. Please adhere to a single
>> standard for indentation across the patch.
Okay. I will fix these.
>>
>> Regards,
>> Vaibhav
>> EnterpriseDB
>>
>> On Mon, Nov 3, 2025 at 5:24 PM Mahendra Singh Thalor <[email protected]> wrote:
>>>
>>> On Mon, 3 Nov 2025 at 12:06, Vaibhav Dalvi <[email protected]> wrote:
>>> >
>>> > Hi Mahendra,
>>> >
>>> > Thank you for your work on this feature.
>>> > I have just begun reviewing the latest patch and
>>> > encountered the following errors during the initial setup:
>>> >
>>> > ```
>>> > $ ./db/bin/pg_restore testdump_dir -C -d postgres -F d -p 5556
>>> > pg_restore: error: could not execute query: ERROR: syntax error at or near "\\"
>>> > LINE 1: \restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj...
>>> > ^
>>> > Command was: \restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
>>> >
>>> > pg_restore: error: could not execute query: ERROR: syntax error at or near "\\"
>>> > LINE 1: \unrestrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCj...
>>> > ^
>>> > Command was: \unrestrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
>>> >
>>> > pg_restore: error: could not execute query: ERROR: syntax error at or near "\\"
>>> > LINE 1: \connect template1
>>> > ^
>>> > Command was: \connect template1
>>> >
>>> > pg_restore: error: could not execute query: ERROR: syntax error at or near "\\"
>>> > LINE 1: \connect postgres
>>> > ^
>>> > Command was: \connect postgres
>>> > ```
>>> > To cross-check tried with plain dump(with pg_dumpall) and
>>> > restored(SQL file restore) without patch and didn't get above
>>> > connection errors.
>>> >
>>> > It appears there might be an issue with the dump file itself.
>>> > Please note that this is my first observation as I have just
>>> > started the review. I will continue with my assessment.
>>> >
>>> > Regards,
>>> > Vaibhav Dalvi
>>> > EnterpriseDB
>>>
>>> Thanks Vaibhav for the review.
>>> This change was added by me in v04. Only in the case of a file, we should restore these commands. Attached patch is fixing the same.
>>>
>>> If we dump and restore the same file with the same user, then we will get an error of ROLE CREATE as the same role is already created. I think, either we can ignore this error, or we can keep it as a restore can be done with different users.
>>>>
>>>> mst@localhost bin]$ ./pg_restore d1 -C -d postgres
>>>> pg_restore: error: could not execute query: ERROR: role "mst" already exists
>>>> Command was: CREATE ROLE mst;
>>>> ALTER ROLE mst WITH SUPERUSER INHERIT CREATEROLE CREATEDB LOGIN REPLICATION BYPASSRLS;
>>>>
>>>>
>>>> pg_restore: warning: errors ignored on restore: 1
>>>
>>>
>>>
>>> >
>>> > On Fri, Oct 31, 2025 at 2:51 PM Mahendra Singh Thalor <[email protected]> wrote:
>>> >>
>>> >> On Tue, 28 Oct 2025 at 11:32, Mahendra Singh Thalor <[email protected]> wrote:
>>> >> >
>>> >> > On Thu, 16 Oct 2025 at 16:24, Mahendra Singh Thalor <[email protected]> wrote:
>>> >> > >
>>> >> > > On Wed, 15 Oct 2025 at 23:05, Mahendra Singh Thalor <[email protected]> wrote:
>>> >> > > >
>>> >> > > > On Sun, 24 Aug 2025 at 22:12, Andrew Dunstan <[email protected]> wrote:
>>> >> > > > >
>>> >> > > > >
>>> >> > > > > On 2025-08-23 Sa 9:08 PM, Noah Misch wrote:
>>> >> > > > >
>>> >> > > > > On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
>>> >> > > > >
>>> >> > > > > OK, now that's reverted we should discuss how to proceed. I had two thoughts
>>> >> > > > > - we could use invent a JSON format for the globals, or we could just use
>>> >> > > > > the existing archive format. I think the archive format is pretty flexible,
>>> >> > > > > and should be able to accommodate this. The downside is it's not humanly
>>> >> > > > > readable. The upside is that we don't need to do anything special either to
>>> >> > > > > write it or parse it.
>>> >> > > > >
>>> >> > > > > I would first try to use the existing archiver API, because that makes it
>>> >> > > > > harder to miss bugs. Any tension between that API and pg_dumpall is likely to
>>> >> > > > > have corresponding tension on the pg_restore side. Resolving that tension
>>> >> > > > > will reveal much of the project's scope that remained hidden during the v18
>>> >> > > > > attempt. Perhaps more important than that, using the archiver API means
>>> >> > > > > future pg_dump and pg_restore options are more likely to cooperate properly
>>> >> > > > > with $SUBJECT. In other words, I want it to be hard to add pg_dump/pg_restore
>>> >> > > > > features that malfunction only for $SUBJECT archives. The strength of the
>>> >> > > > > archiver architecture shows in how rarely new features need format-specific
>>> >> > > > > logic and how rarely format-specific bugs get reported. We've had little or
>>> >> > > > > no trouble with e.g. bugs that appear in -Fd but not in -Fc.
>>> >> > > > >
>>> >> > > > >
>>> >> > > > > Yeah, that's what we're going to try.
>>> >> > > > >
>>> >> > > > >
>>> >> > > > > cheers
>>> >> > > > >
>>> >> > > > >
>>> >> > > > > andrew
>>> >> > > > >
>>> >> > > > > --
>>> >> > > > > Andrew Dunstan
>>> >> > > > > EDB: https://www.enterprisedb.com
>>> >> > > >
>>> >> > > > Thanks Andrew, Noah and all others for feedback.
>>> >> > > >
>>> >> > > > Based on the above suggestions and discussions, I removed sql commands
>>> >> > > > from the global.dat file. For global commands, now we are making
>>> >> > > > toc.dat/toc.dmp/toc.tar file based on format specified and based on
>>> >> > > > format specified, we are making archive entries for these global
>>> >> > > > commands. By this approach, we removed the hard-coded parsing part of
>>> >> > > > the global.dat file and we are able to skip DROP DATABASE with the
>>> >> > > > globals-only option.
>>> >> > > >
>>> >> > > > Here, I am attaching a patch for review, testing and feedback. This is
>>> >> > > > a WIP patch. I will do some more code cleanup and will add some more
>>> >> > > > comments also. Please review this and let me know design level
>>> >> > > > feedback. Thanks Tushar Ahuja for some internal testing and feedback.
>>> >> > > >
>>> >> > >
>>> >> > > Hi,
>>> >> > > Here, I am attaching an updated patch. In offline discussion, Andrew
>>> >> > > reported some test-case failures(Thanks Andrew). I fixed those.
>>> >> > > Please let me know feedback for the patch.
>>> >> > >
>>> >> >
>>> >> > Hi,
>>> >> > Here I am attaching a re-based patch as v02 was failing on head.
>>> >> > Thanks Tushar for the testing.
>>> >> > Please review this and let me know feedback.
>>> >> >
>>> >>
>>> >> Hi all,
>>> >> Here I am attaching an updated patch for review and testing. Based on
>>> >> some offline comments by Andrew, I did some code cleanup.
>>> >> Please consider this patch for feedback.
>>> >>
>>> >> --
>>> >> Thanks and Regards
>>> >> Mahendra Singh Thalor
>>> >> EnterpriseDB: http://www.enterprisedb.com
>>>
>>>
>>>
>>> --
>>> Thanks and Regards
>>> Mahendra Singh Thalor
>>> EnterpriseDB: http://www.enterprisedb.com
Here, I am attaching an updated patch for the review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v06_06112025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (84.5K, 2-v06_06112025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
From 352b7abe680ac767499ea3bfca07f86b9f0637fb Mon Sep 17 00:00:00 2001
From: ThalorMahendra <[email protected]>
Date: Thu, 6 Nov 2025 10:33:16 +0530
Subject: [PATCH] Non text modes for pg_dumpall, correspondingly change
pg_restore
pg_dumpall acquires a new -F/--format option, with the same meanings as
pg_dump. The default is p, meaning plain text. For any other value, a
directory is created containing two files, toc.dat/.dmp/.tar and map.dat. The
first contains commands restoring the global data based on -F, and the second
contains a map from oids to database names. It will also contain a
subdirectory called databases, inside which it will create archives in
the specified format, named using the database oids.
In these casess the -f argument is required.
If pg_restore encounters a directory containing map.dat,
it restores the global settings from toc.dat/.dmp/.tar if exist, and then
restores each database.
pg_restore acquires two new options: -g/--globals-only which suppresses
restoration of any databases, and --exclude-database which inhibits
restoration of particualr database(s) in the same way the same option
works in pg_dumpall.
v06
---
doc/src/sgml/ref/pg_dumpall.sgml | 89 +++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/connectdb.c | 1 -
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 10 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 31 +-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 621 ++++++++++++++++++++++-----
src/bin/pg_dump/pg_restore.c | 595 ++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 10 +
src/bin/pg_dump/t/007_pg_dumpall.pl | 396 +++++++++++++++++
14 files changed, 1680 insertions(+), 147 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
create mode 100755 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 9f639f61db0..4063e88d388 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,10 +139,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>toc.dat/toc.dmp/toc.tar</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index a468a38361a..7497b527ae6 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -591,6 +615,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
index d55d53dbeea..f44a8a45fca 100644
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -287,7 +287,6 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
exit_nicely(1);
}
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index f3c669f484e..3e21aaf5780 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..5974d6706fd 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index d9041dad720..f631d945472 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -312,7 +312,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, bool globals_only);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 59eaecb4ed7..f5df9ac5c2b 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, bool globals_only)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,17 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ /* Skip DROP DATABASE if globals_only. */
+ if (globals_only && te && te->tag && (strcmp(te->tag, "DROP_DATABASE") == 0))
+ continue;
+
+ /* Skip for RESTRICT, UNRESTRICT, CONNECT. */
+ if (!ropt->filename && te && te->tag &&
+ ((strcmp(te->tag, "RESTRICT") == 0) ||
+ (strcmp(te->tag, "UNRESTRICT") == 0) ||
+ (strcmp(te->tag, "CONNECT") == 0)))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1332,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1695,7 +1711,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1732,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..818b80a9369 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a00918bacb4..13e1764ec70 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index bb451c1bae1..5b9144aa002 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -30,6 +30,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -65,9 +66,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,6 +78,9 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
+static void createOneArchiveEntry(const char *query, const char *tag);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -123,6 +128,13 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static const CatalogId nilCatalogId = {0, 0};
+static ArchiveMode archiveMode = archModeWrite;
+static DataDirSyncMethod sync_method = DATA_DIR_SYNC_METHOD_FSYNC;
+static ArchiveFormat archDumpFormat = archNull;
int
main(int argc, char *argv[])
@@ -148,6 +160,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +210,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -208,6 +222,8 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
+ char global_path[MAXPGPATH];
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -246,7 +262,9 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ InitDumpOptions(&dopt);
+
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -257,6 +275,7 @@ main(int argc, char *argv[])
case 'c':
output_clean = true;
+ dopt.outputClean = 1;
break;
case 'd':
@@ -274,7 +293,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -314,6 +335,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -429,6 +451,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -489,6 +526,35 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ /* set file path for global sql commands. */
+ if (archDumpFormat == archCustom)
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", filename);
+ else if (archDumpFormat == archTar)
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", filename);
+ else if (archDumpFormat == archDirectory)
+ snprintf(global_path, MAXPGPATH, "%s", filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -538,19 +604,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -585,37 +638,114 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
if (verbose)
dumpTimestamp("Started on");
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ /* create a archive file for global commands. */
+ if (filename && archDumpFormat != archNull)
+ {
+ /* Open the output file */
+ fout = CreateArchive(global_path, archDumpFormat, compression_spec,
+ dosync, archiveMode, NULL, sync_method);
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
+ ((ArchiveHandle*)fout)->connection = conn;
+ ((ArchiveHandle*)fout)->public.numWorkers = 1;
+
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump\n--\n\n", "COMMENT");
- /* Replicate encoding and std_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
- if (strcmp(std_strings, "off") == 0)
- fprintf(OPF, "SET escape_string_warning = off;\n");
- fprintf(OPF, "\n");
+ /* create entry for restrict */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\\restrict %s\n\n", restrict_key);
+ createOneArchiveEntry(qry->data, "RESTRICT");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* default_transaction_read_only = off */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving default_transaction_read_only = off");
+ appendPQExpBuffer(qry, "SET default_transaction_read_only = off;\n");
+ createOneArchiveEntry(qry->data, "DEFAULT_TRANSACTION_READ_ONLY");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpEncoding: put the correct encoding into the archive */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ const char *encname = pg_encoding_to_char(encoding);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+
+ pg_log_info("saving encoding = %s", encname);
+ createOneArchiveEntry(qry->data, "ENCODING");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpStdStrings: put the correct escape string behavior into the archive */
+ {
+ const char *stdstrings = std_strings ? "on" : "off";
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving \"standard_conforming_strings = %s\"", stdstrings);
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = '%s';\n",
+ stdstrings);
+ createOneArchiveEntry(qry->data, "STDSTRINGS");
+ destroyPQExpBuffer(qry);
+ }
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and std_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
+ if (strcmp(std_strings, "off") == 0)
+ fprintf(OPF, "SET escape_string_warning = off;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -659,27 +789,51 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump will
+ * handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
+ else
+ {
+ /* create entry for unrestrict */
+ PQExpBuffer qry = createPQExpBuffer();
- if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ appendPQExpBuffer(qry, "\\unrestrict %s\n\n", restrict_key);
+ createOneArchiveEntry(qry->data, "UNRESTRICT");
+ destroyPQExpBuffer(qry);
+ }
- PQfinish(conn);
+ if (!globals_only && !roles_only && !tablespaces_only)
+ dumpDatabases(conn, archDumpFormat);
- if (verbose)
+ if (verbose && archDumpFormat == archNull)
dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (filename)
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+
+ if (archDumpFormat != archNull)
+ {
+ RestoreOptions *ropt;
+
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump complete\n--\n\n", "COMMENT");
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
+
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
+ }
+ else if (filename)
{
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -690,12 +844,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -770,6 +926,7 @@ static void
dropRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer delQry = createPQExpBuffer();
PGresult *res;
int i_rolname;
int i;
@@ -791,7 +948,12 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -799,15 +961,21 @@ dropRoles(PGconn *conn)
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(rolename));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropRoles");
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -889,7 +1057,12 @@ dumpRoles(PGconn *conn)
i_is_current_user = PQfnumber(res, "is_current_user");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -993,7 +1166,10 @@ dumpRoles(PGconn *conn)
"ROLE", rolename,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoles");
}
/*
@@ -1001,15 +1177,13 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
}
@@ -1088,7 +1262,12 @@ dumpRoleMembership(PGconn *conn)
i_set_option = PQfnumber(res, "set_option");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role memberships\n--\n\n", "COMMENT");
+ }
/*
* We can't dump these GRANT commands in arbitrary order, because a role
@@ -1167,6 +1346,7 @@ dumpRoleMembership(PGconn *conn)
char *grantor;
char *set_option = "true";
bool found;
+ PQExpBuffer creaQry = createPQExpBuffer();
/* If we already did this grant, don't do it again. */
if (done[i - start])
@@ -1223,8 +1403,8 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ appendPQExpBuffer(creaQry, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(creaQry, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1245,10 +1425,15 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(creaQry, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(creaQry, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(creaQry, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", creaQry->data);
+ else
+ createOneArchiveEntry(creaQry->data, "dumpRoleMembership");
}
}
@@ -1260,7 +1445,8 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1288,7 +1474,12 @@ dumpRoleGUCPrivs(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role privileges on configuration parameters\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1312,14 +1503,19 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoleGUCPrivs");
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1331,6 +1527,7 @@ dropTablespaces(PGconn *conn)
{
PGresult *res;
int i;
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Get all tablespaces except built-in ones (which we assume are named
@@ -1342,20 +1539,31 @@ dropTablespaces(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(spcname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropTablespaces");
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1382,7 +1590,12 @@ dumpTablespaces(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1451,14 +1664,19 @@ dumpTablespaces(PGconn *conn)
"TABLESPACE", spcname,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpTablespaces");
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1482,7 +1700,12 @@ dropDBs(PGconn *conn)
"ORDER BY datname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop databases (except postgres and template1)\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1497,15 +1720,23 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(dbname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "DROP_DATABASE");
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1532,7 +1763,18 @@ dumpUserConfig(PGconn *conn, const char *username)
char *sanitized;
sanitized = sanitize_line(username, true);
- fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
}
@@ -1542,7 +1784,11 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpUserConfig");
}
PQclear(res);
@@ -1608,10 +1854,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1625,19 +1874,48 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Databases\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Databases\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Databases\n--\n\n", "COMMENT");
+ }
+
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1654,7 +1932,18 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
/*
@@ -1669,24 +1958,46 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
+ fprintf(OPF, "\\connect %s\n\n", dbname);
else
{
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\\connect %s\n\n", dbname);
+ createOneArchiveEntry(qry->data, "CONNECT");
+ destroyPQExpBuffer(qry);
}
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1695,6 +2006,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1704,7 +2019,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1713,17 +2029,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1807,7 +2142,18 @@ dumpTimestamp(const char *msg)
time_t now = time(NULL);
if (strftime(buf, sizeof(buf), PGDUMP_STRFTIME_FMT, localtime(&now)) != 0)
- fprintf(OPF, "-- %s %s\n\n", msg, buf);
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "-- %s %s\n\n", msg, buf);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "-- %s %s\n\n", msg, buf);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+ }
}
/*
@@ -1868,3 +2214,66 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+/*
+ * createDumpId
+ *
+ * This will return next last used oid.
+ */
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
+
+/*
+ * createOneArchiveEntry
+ *
+ * This creates one archive entry based on format.
+ */
+static void
+createOneArchiveEntry(const char *query, const char *tag)
+{
+ Assert(fout != NULL);
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = tag,
+ .section = SECTION_PRE_DATA,
+ .createStmt = query));
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c9776306c5c..97a6bcb6d31 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,31 +41,61 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num,
+ bool globals_only);
+static int restore_global_objects(const char *inputFileSpec,
+ RestoreOptions *opts, int numWorkers, bool append_data,
+ int num, bool globals_only);
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +119,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +173,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +202,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +229,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global sql commands. */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -321,6 +356,10 @@ main(int argc, char **argv)
opts->restrict_key = pg_strdup(optarg);
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -347,6 +386,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -472,6 +518,105 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If map.dat file is present, then restore all the
+ * databases from map.dat , but skip restoring those matching
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "map.dat") ||
+ file_exists_in_directory(inputFileSpec, "toc.tar") ||
+ file_exists_in_directory(inputFileSpec, "toc.dmp")))
+ {
+ char global_path[MAXPGPATH];
+
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring an archive created by pg_dumpall");
+ else if (opts->tocFile)
+ pg_fatal("option -L/--use-list cannot be used when restoring an archive created by pg_dumpall");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option -C/--create must be specified when restoring an archive created by pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ n_errors = restore_global_objects(global_path, opts, numWorkers, false, 0, globals_only);
+
+ pg_log_info("database restoring skipped because option -g/--globals-only was specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else /* process if map.dat file does not exist. */
+ n_errors = restore_one_database(inputFileSpec, opts,
+ numWorkers, false, 0, globals_only);
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ *
+ * If globals_only is set, then skip DROP DATABASE commands from restore.
+ */
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ return restore_one_database(inputFileSpec, opts, numWorkers,
+ append_data, num, globals_only);
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -479,9 +624,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -501,25 +652,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, globals_only);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -537,6 +684,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -553,6 +701,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -588,8 +737,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -694,3 +843,415 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn && db_exclude_patterns.head != NULL)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no database connection while doing pg_restore");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+ PQExpBuffer db_lit = createPQExpBuffer();
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches exclude pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ destroyPQExpBuffer(db_lit);
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ destroyPQExpBuffer(query);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+
+
+ /*
+ * If there is no map.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen <= 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbname, db_oid, map_file_path);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+ PGconn *conn = NULL;
+ char global_path[MAXPGPATH];
+
+ /* Based on file, set path. */
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
+
+ /* If map.dat has no entries, return after processing global commands. */
+ if (dbname_oid_list.head == NULL)
+ return restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Open toc.dat file and execute/append all the global sql commands. */
+ n_errors_total = restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ count++;
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..56e89da1e5e
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,12 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +250,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100755
index 00000000000..3c7d2ad7c53
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,396 @@
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# Scape tablespace locations on Windows.
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT super TO grant7 WITH INHERIT TRUE GRANTED BY\E
+ (.*\n)*
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: When --list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: When non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+$node->stop('fast');
+
+done_testing();
--
2.39.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-11 05:59 Mahendra Singh Thalor <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 2 replies; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-11-11 05:59 UTC (permalink / raw)
To: Vaibhav Dalvi <[email protected]>; +Cc: [email protected]
On Thu, 6 Nov 2025 at 11:03, Mahendra Singh Thalor <[email protected]> wrote:
>
> Thanks Vaibhav, Tushar and Andrew for the review and testing.
>
> On Mon, 3 Nov 2025 at 17:30, Vaibhav Dalvi
> <[email protected]> wrote:
> >
> > Hi Mahendra,
> >
> > I have a few more review comments regarding the patch:
> >
> > 1. Is the following change in `src/bin/pg_dump/connectdb.c` intentional?
> >
> > ```
> > --- a/src/bin/pg_dump/connectdb.c
> > +++ b/src/bin/pg_dump/connectdb.c
>
> Yes, we need this. If there is any error, then we were trying to
> disconnect the database in 2 places so we were getting a crash. I will
> try to reproduce crashe without this patch and will respond.
>
> On Tue, 4 Nov 2025 at 18:23, tushar <[email protected]> wrote:
> > Thanks Mahendra, I am getting a segmentation fault against v05 patch.
> >
> > [edb@1a1c15437e7c bin]$ ./pg_dumpall -Ft --file a.3 -v
> > pg_dumpall: executing SELECT pg_catalog.set_config('search_path', '', false);
> > Segmentation fault
> >
> > Issue is coming with all output file formats -F[t/c/d] except plain
> >
> > regards,
>
> Thanks for the report. Fixed,
>
> On Tue, 4 Nov 2025 at 22:25, Andrew Dunstan <[email protected]> wrote:
> > Yeah, I don't think we need to dump the timestamp in non-text modes. This fix worked for me:
> >
> >
> > diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
> > index 601b9f9738e..f66cc26d9a2 100644
> > --- a/src/bin/pg_dump/pg_dumpall.c
> > +++ b/src/bin/pg_dump/pg_dumpall.c
> > @@ -638,7 +638,7 @@ main(int argc, char *argv[])
> > if (quote_all_identifiers)
> > executeCommand(conn, "SET quote_all_identifiers = true");
> >
> > - if (verbose)
> > + if (verbose && archDumpFormat == archNull)
> > dumpTimestamp("Started on");
>
> Thanks Andrew. Yes, we should not dump timestamp in non-text modes.
>
> On Wed, 5 Nov 2025 at 18:47, Vaibhav Dalvi
> <[email protected]> wrote:
> >
> > Hi Mahendra,
> >
> > Here are a few more comments following my review of the patch:
> >
> > ### 1\. Incorrect Comment for `-g` (globals-only) Option
> >
> > The comment for the `-g` case in the code states that it restores the
> > `global.dat` file. However, in the non-text dump output, I only see the
> > following files: `databases`, `map.dat`, and `toc.dat`.
>
> Fixed.
>
> >
> > ```c
> > + case 'g':
> > + /* restore only global.dat file from directory */
> > + globals_only = true;
> > + break;
>
> Fixed.
>
> > ```
> >
> > Please update this comment to accurately reflect the file being restored
> > (e.g., `toc.dat` or the global objects within the archive).
>
> Fixed.
>
> >
> > ### 2\. Incorrect Order of `case` Statements in `pg_restore.c`
> >
> > The new `case 7` statement in `pg_restore.c` appears to be
> > inserted before `case 6`, disrupting the numerical order.
> >
> > ```c
> > + case 7: /* database patterns to skip */
> > + simple_string_list_append(&db_exclude_patterns, optarg);
> > + break;
> >
> > case 6:
> > opts->restrict_key = pg_strdup(optarg);
> > ```
> >
> > Please re-order the `case` statements so they follow ascending
> > numerical order.
>
> Fixed.
>
> >
> > ### 3\. Missing Example in SGML Documentation
> >
> > The SGML documentation for `pg_dumpall` is missing an explicit
> > example demonstrating its use with non-text formats (e.g., directory format).
> > It would be beneficial to include a clear example for this new feature.
>
> I think we don't add such cases in doc. We already added test cases in
> code. If others also feel that we should add a test case in SGML, then
> I will update the doc with the test case.
>
> >
> > ### 4\. Cosmetic Issues
> >
> > Please address the following minor stylistic points:
> >
> > Please ensure the function signatures
> > adhere to standard coding style, particularly for line wrapping.
> > The following lines seem to have inconsistent indentation:
> >
> > ```c
> > static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
> > int numWorkers, bool append_data, int num, bool globals_only);
> > static int restore_all_databases(const char *inputFileSpec,
> > SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
> > ```
> >
> > Please fix instances where the 80-character line limit is
> > crossed, such as in the example below:
>
> Fixed.
>
> >
> > ```c
> > n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
> > ```
> >
> > I believe this concludes my formal review.
> >
> > Thanks,
> > Vaibhav Dalvi
> >
> > On Wed, Nov 5, 2025 at 12:29 PM Vaibhav Dalvi <[email protected]> wrote:
> >>
> >> Hi Mahendra,
> >>
> >> Thank you for the fix. Please find my further review comments below.
> >>
> >> ### Restrict-Key Option
> >>
> >> The `--restrict-key` option is currently being accepted by
> >> `pg_dumpall` even when non-plain formats are specified,
> >> which contradicts its intended use only with the plain format.
> >>
> >> For example:
> >>
> >> ```
> >> $ ./db/bin/pg_dump --format=d -f testdump_dir --restrict-key=RESTRICT_KEY
> >> pg_dump: error: option --restrict-key can only be used with --format=plain
> >> $ ./db/bin/pg_dumpall --format=d -f testdump_dir --restrict-key=RESTRICT_KEY
> >> pg_dumpall: error: invalid restrict key
> >> ```
> >>
> >> I have attached a delta patch that addresses the issue with the
> >> `--restrict-key` option. It would be beneficial to include a dedicated
> >> test case for this check.
>
> We should dump restrict-key with all modes as we need to restore with
> the "-f file" option in text mode.
> Ex: pg_dumpall --format=d -f testdump_dir
> and restore::: pg_restore testdump_dir -d dabasename -C -f testdumpfile
> (In testdumpfile, we will generate commands from archive dump)
>
> So I am not merging this delat patch.
>
> >>
> >> ### Use of Dump Options Structure (dopt)
> >>
> >> Please ensure consistency by utilizing the main dump options
> >> structure (`dopt`) instead of declaring and using individual variables
> >> where the structure already provides fields. For example, the
> >> `output_clean` variable seems redundant here:
> >>
> >> ```c
> >> case 'c':
> >> output_clean = true;
> >> dopt.outputClean = 1;
> >> break;
>
> output_clean is not added by this patch. I will analyse this comment
> and will respond in the next update.
>
> >> ```
> >>
> >> In my attached delta file, I have replaced the unnecessary
> >> `restrict_key` variable with `dopt.restrict_key`.
>
> This is also not part of this patch. If you feel to add this in DOPT,
> please suggest in separate thread.
>
> >>
> >> ### Cosmetic Issues
> >>
> >> 1. Please review the spacing around the pointer:
> >> ```c
> >> + ((ArchiveHandle * )fout) ->connection = conn;
> >> + ((ArchiveHandle * ) fout) -> public.numWorkers = 1;
>
> Fixed.
>
> >> ```
> >> 2. Please be consistent with the punctuation of single-line comments;
> >> some end with a full stop (`.`) and others do not.
>
> Based on nearby code comments, I made changes. I will try to fix these
> inconsistencies..
>
>
> >> 3. In the SGML documentation changes, some new statements start
> >> with one space, and others start with two. Please adhere to a single
> >> standard for indentation across the patch.
>
> Okay. I will fix these.
>
> >>
> >> Regards,
> >> Vaibhav
> >> EnterpriseDB
> >>
> >> On Mon, Nov 3, 2025 at 5:24 PM Mahendra Singh Thalor <[email protected]> wrote:
> >>>
> >>> On Mon, 3 Nov 2025 at 12:06, Vaibhav Dalvi <[email protected]> wrote:
> >>> >
> >>> > Hi Mahendra,
> >>> >
> >>> > Thank you for your work on this feature.
> >>> > I have just begun reviewing the latest patch and
> >>> > encountered the following errors during the initial setup:
> >>> >
> >>> > ```
> >>> > $ ./db/bin/pg_restore testdump_dir -C -d postgres -F d -p 5556
> >>> > pg_restore: error: could not execute query: ERROR: syntax error at or near "\\"
> >>> > LINE 1: \restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj...
> >>> > ^
> >>> > Command was: \restrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
> >>> >
> >>> > pg_restore: error: could not execute query: ERROR: syntax error at or near "\\"
> >>> > LINE 1: \unrestrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCj...
> >>> > ^
> >>> > Command was: \unrestrict aO9K1gzVZTlafidF5fWx8ADGzUnIiAcguFz5qskGaFDygTCjCj9vg3Xxys1b3hb
> >>> >
> >>> > pg_restore: error: could not execute query: ERROR: syntax error at or near "\\"
> >>> > LINE 1: \connect template1
> >>> > ^
> >>> > Command was: \connect template1
> >>> >
> >>> > pg_restore: error: could not execute query: ERROR: syntax error at or near "\\"
> >>> > LINE 1: \connect postgres
> >>> > ^
> >>> > Command was: \connect postgres
> >>> > ```
> >>> > To cross-check tried with plain dump(with pg_dumpall) and
> >>> > restored(SQL file restore) without patch and didn't get above
> >>> > connection errors.
> >>> >
> >>> > It appears there might be an issue with the dump file itself.
> >>> > Please note that this is my first observation as I have just
> >>> > started the review. I will continue with my assessment.
> >>> >
> >>> > Regards,
> >>> > Vaibhav Dalvi
> >>> > EnterpriseDB
> >>>
> >>> Thanks Vaibhav for the review.
> >>> This change was added by me in v04. Only in the case of a file, we should restore these commands. Attached patch is fixing the same.
> >>>
> >>> If we dump and restore the same file with the same user, then we will get an error of ROLE CREATE as the same role is already created. I think, either we can ignore this error, or we can keep it as a restore can be done with different users.
> >>>>
> >>>> mst@localhost bin]$ ./pg_restore d1 -C -d postgres
> >>>> pg_restore: error: could not execute query: ERROR: role "mst" already exists
> >>>> Command was: CREATE ROLE mst;
> >>>> ALTER ROLE mst WITH SUPERUSER INHERIT CREATEROLE CREATEDB LOGIN REPLICATION BYPASSRLS;
> >>>>
> >>>>
> >>>> pg_restore: warning: errors ignored on restore: 1
> >>>
> >>>
> >>>
> >>> >
> >>> > On Fri, Oct 31, 2025 at 2:51 PM Mahendra Singh Thalor <[email protected]> wrote:
> >>> >>
> >>> >> On Tue, 28 Oct 2025 at 11:32, Mahendra Singh Thalor <[email protected]> wrote:
> >>> >> >
> >>> >> > On Thu, 16 Oct 2025 at 16:24, Mahendra Singh Thalor <[email protected]> wrote:
> >>> >> > >
> >>> >> > > On Wed, 15 Oct 2025 at 23:05, Mahendra Singh Thalor <[email protected]> wrote:
> >>> >> > > >
> >>> >> > > > On Sun, 24 Aug 2025 at 22:12, Andrew Dunstan <[email protected]> wrote:
> >>> >> > > > >
> >>> >> > > > >
> >>> >> > > > > On 2025-08-23 Sa 9:08 PM, Noah Misch wrote:
> >>> >> > > > >
> >>> >> > > > > On Wed, Jul 30, 2025 at 02:51:59PM -0400, Andrew Dunstan wrote:
> >>> >> > > > >
> >>> >> > > > > OK, now that's reverted we should discuss how to proceed. I had two thoughts
> >>> >> > > > > - we could use invent a JSON format for the globals, or we could just use
> >>> >> > > > > the existing archive format. I think the archive format is pretty flexible,
> >>> >> > > > > and should be able to accommodate this. The downside is it's not humanly
> >>> >> > > > > readable. The upside is that we don't need to do anything special either to
> >>> >> > > > > write it or parse it.
> >>> >> > > > >
> >>> >> > > > > I would first try to use the existing archiver API, because that makes it
> >>> >> > > > > harder to miss bugs. Any tension between that API and pg_dumpall is likely to
> >>> >> > > > > have corresponding tension on the pg_restore side. Resolving that tension
> >>> >> > > > > will reveal much of the project's scope that remained hidden during the v18
> >>> >> > > > > attempt. Perhaps more important than that, using the archiver API means
> >>> >> > > > > future pg_dump and pg_restore options are more likely to cooperate properly
> >>> >> > > > > with $SUBJECT. In other words, I want it to be hard to add pg_dump/pg_restore
> >>> >> > > > > features that malfunction only for $SUBJECT archives. The strength of the
> >>> >> > > > > archiver architecture shows in how rarely new features need format-specific
> >>> >> > > > > logic and how rarely format-specific bugs get reported. We've had little or
> >>> >> > > > > no trouble with e.g. bugs that appear in -Fd but not in -Fc.
> >>> >> > > > >
> >>> >> > > > >
> >>> >> > > > > Yeah, that's what we're going to try.
> >>> >> > > > >
> >>> >> > > > >
> >>> >> > > > > cheers
> >>> >> > > > >
> >>> >> > > > >
> >>> >> > > > > andrew
> >>> >> > > > >
> >>> >> > > > > --
> >>> >> > > > > Andrew Dunstan
> >>> >> > > > > EDB: https://www.enterprisedb.com
> >>> >> > > >
> >>> >> > > > Thanks Andrew, Noah and all others for feedback.
> >>> >> > > >
> >>> >> > > > Based on the above suggestions and discussions, I removed sql commands
> >>> >> > > > from the global.dat file. For global commands, now we are making
> >>> >> > > > toc.dat/toc.dmp/toc.tar file based on format specified and based on
> >>> >> > > > format specified, we are making archive entries for these global
> >>> >> > > > commands. By this approach, we removed the hard-coded parsing part of
> >>> >> > > > the global.dat file and we are able to skip DROP DATABASE with the
> >>> >> > > > globals-only option.
> >>> >> > > >
> >>> >> > > > Here, I am attaching a patch for review, testing and feedback. This is
> >>> >> > > > a WIP patch. I will do some more code cleanup and will add some more
> >>> >> > > > comments also. Please review this and let me know design level
> >>> >> > > > feedback. Thanks Tushar Ahuja for some internal testing and feedback.
> >>> >> > > >
> >>> >> > >
> >>> >> > > Hi,
> >>> >> > > Here, I am attaching an updated patch. In offline discussion, Andrew
> >>> >> > > reported some test-case failures(Thanks Andrew). I fixed those.
> >>> >> > > Please let me know feedback for the patch.
> >>> >> > >
> >>> >> >
> >>> >> > Hi,
> >>> >> > Here I am attaching a re-based patch as v02 was failing on head.
> >>> >> > Thanks Tushar for the testing.
> >>> >> > Please review this and let me know feedback.
> >>> >> >
> >>> >>
> >>> >> Hi all,
> >>> >> Here I am attaching an updated patch for review and testing. Based on
> >>> >> some offline comments by Andrew, I did some code cleanup.
> >>> >> Please consider this patch for feedback.
> >>> >>
> >>> >> --
> >>> >> Thanks and Regards
> >>> >> Mahendra Singh Thalor
> >>> >> EnterpriseDB: http://www.enterprisedb.com
> >>>
> >>>
> >>>
> >>> --
> >>> Thanks and Regards
> >>> Mahendra Singh Thalor
> >>> EnterpriseDB: http://www.enterprisedb.com
>
> Here, I am attaching an updated patch for the review and testing.
>
> --
> Thanks and Regards
> Mahendra Singh Thalor
> EnterpriseDB: http://www.enterprisedb.com
Hi,
Here, I am attaching an updated patch for the review and testing.
FIX: as suggested by Vaibhav, added error for --restrict-key option
with non-text format.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v07_11112025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (84.4K, 2-v07_11112025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
From cef022cee856e71a4a4a078ff3610eec90e1d805 Mon Sep 17 00:00:00 2001
From: ThalorMahendra <[email protected]>
Date: Tue, 11 Nov 2025 11:25:34 +0530
Subject: [PATCH] Non text modes for pg_dumpall, correspondingly change
pg_restore
pg_dumpall acquires a new -F/--format option, with the same meanings as
pg_dump. The default is p, meaning plain text. For any other value, a
directory is created containing two files, toc.dat/.dmp/.tar and map.dat. The
first contains commands restoring the global data based on -F, and the second
contains a map from oids to database names. It will also contain a
subdirectory called databases, inside which it will create archives in
the specified format, named using the database oids.
In these casess the -f argument is required.
If pg_restore encounters a directory containing map.dat,
it restores the global settings from toc.dat/.dmp/.tar if exist, and then
restores each database.
pg_restore acquires two new options: -g/--globals-only which suppresses
restoration of any databases, and --exclude-database which inhibits
restoration of particualr database(s) in the same way the same option
works in pg_dumpall.
v07
---
doc/src/sgml/ref/pg_dumpall.sgml | 89 +++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/connectdb.c | 1 -
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 10 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 31 +-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 607 ++++++++++++++++++++++-----
src/bin/pg_dump/pg_restore.c | 595 +++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 15 +
src/bin/pg_dump/t/007_pg_dumpall.pl | 396 +++++++++++++++++
14 files changed, 1671 insertions(+), 147 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
create mode 100755 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 9f639f61db0..4063e88d388 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,10 +139,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>toc.dat/toc.dmp/toc.tar</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index a468a38361a..7497b527ae6 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -591,6 +615,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
index d55d53dbeea..f44a8a45fca 100644
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -287,7 +287,6 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
exit_nicely(1);
}
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index f3c669f484e..3e21aaf5780 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..5974d6706fd 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index d9041dad720..f631d945472 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -312,7 +312,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, bool globals_only);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 59eaecb4ed7..f5df9ac5c2b 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, bool globals_only)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,17 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ /* Skip DROP DATABASE if globals_only. */
+ if (globals_only && te && te->tag && (strcmp(te->tag, "DROP_DATABASE") == 0))
+ continue;
+
+ /* Skip for RESTRICT, UNRESTRICT, CONNECT. */
+ if (!ropt->filename && te && te->tag &&
+ ((strcmp(te->tag, "RESTRICT") == 0) ||
+ (strcmp(te->tag, "UNRESTRICT") == 0) ||
+ (strcmp(te->tag, "CONNECT") == 0)))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1332,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1695,7 +1711,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1732,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..818b80a9369 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a00918bacb4..13e1764ec70 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index bb451c1bae1..928ad7e5e0a 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -30,6 +30,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -65,9 +66,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,6 +78,9 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
+static void createOneArchiveEntry(const char *query, const char *tag);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -123,6 +128,13 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static const CatalogId nilCatalogId = {0, 0};
+static ArchiveMode archiveMode = archModeWrite;
+static DataDirSyncMethod sync_method = DATA_DIR_SYNC_METHOD_FSYNC;
+static ArchiveFormat archDumpFormat = archNull;
int
main(int argc, char *argv[])
@@ -148,6 +160,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +210,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -208,6 +222,8 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
+ char global_path[MAXPGPATH];
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -246,7 +262,9 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ InitDumpOptions(&dopt);
+
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -257,6 +275,7 @@ main(int argc, char *argv[])
case 'c':
output_clean = true;
+ dopt.outputClean = 1;
break;
case 'd':
@@ -274,7 +293,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -314,6 +335,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -429,6 +451,25 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
+ /* restrict-key is only supported with --format=plain */
+ if (archDumpFormat != archNull && restrict_key)
+ pg_fatal("option --restrict-key can only be used with --format=plain");
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -489,6 +530,35 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ /* set file path for global sql commands. */
+ if (archDumpFormat == archCustom)
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", filename);
+ else if (archDumpFormat == archTar)
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", filename);
+ else if (archDumpFormat == archDirectory)
+ snprintf(global_path, MAXPGPATH, "%s", filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -538,19 +608,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -585,37 +642,105 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
if (verbose)
dumpTimestamp("Started on");
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ /* create a archive file for global commands. */
+ if (filename && archDumpFormat != archNull)
+ {
+ /* Open the output file */
+ fout = CreateArchive(global_path, archDumpFormat, compression_spec,
+ dosync, archiveMode, NULL, sync_method);
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
+ ((ArchiveHandle*)fout)->connection = conn;
+ ((ArchiveHandle*)fout)->public.numWorkers = 1;
+
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
+
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump\n--\n\n", "COMMENT");
- /* Replicate encoding and std_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
- if (strcmp(std_strings, "off") == 0)
- fprintf(OPF, "SET escape_string_warning = off;\n");
- fprintf(OPF, "\n");
+ /* default_transaction_read_only = off */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving default_transaction_read_only = off");
+ appendPQExpBuffer(qry, "SET default_transaction_read_only = off;\n");
+ createOneArchiveEntry(qry->data, "DEFAULT_TRANSACTION_READ_ONLY");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpEncoding: put the correct encoding into the archive */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ const char *encname = pg_encoding_to_char(encoding);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+
+ pg_log_info("saving encoding = %s", encname);
+ createOneArchiveEntry(qry->data, "ENCODING");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpStdStrings: put the correct escape string behavior into the archive */
+ {
+ const char *stdstrings = std_strings ? "on" : "off";
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving \"standard_conforming_strings = %s\"", stdstrings);
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = '%s';\n",
+ stdstrings);
+ createOneArchiveEntry(qry->data, "STDSTRINGS");
+ destroyPQExpBuffer(qry);
+ }
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and std_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
+ if (strcmp(std_strings, "off") == 0)
+ fprintf(OPF, "SET escape_string_warning = off;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -659,27 +784,42 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump will
+ * handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
-
- PQfinish(conn);
+ dumpDatabases(conn, archDumpFormat);
- if (verbose)
+ if (verbose && archDumpFormat == archNull)
dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (filename)
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+
+ if (archDumpFormat != archNull)
+ {
+ RestoreOptions *ropt;
+
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump complete\n--\n\n", "COMMENT");
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
+
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
+ }
+ else if (filename)
{
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -690,12 +830,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -770,6 +912,7 @@ static void
dropRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer delQry = createPQExpBuffer();
PGresult *res;
int i_rolname;
int i;
@@ -791,7 +934,12 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -799,15 +947,21 @@ dropRoles(PGconn *conn)
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(rolename));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropRoles");
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -889,7 +1043,12 @@ dumpRoles(PGconn *conn)
i_is_current_user = PQfnumber(res, "is_current_user");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -993,7 +1152,10 @@ dumpRoles(PGconn *conn)
"ROLE", rolename,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoles");
}
/*
@@ -1001,15 +1163,13 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
}
@@ -1088,7 +1248,12 @@ dumpRoleMembership(PGconn *conn)
i_set_option = PQfnumber(res, "set_option");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role memberships\n--\n\n", "COMMENT");
+ }
/*
* We can't dump these GRANT commands in arbitrary order, because a role
@@ -1167,6 +1332,7 @@ dumpRoleMembership(PGconn *conn)
char *grantor;
char *set_option = "true";
bool found;
+ PQExpBuffer creaQry = createPQExpBuffer();
/* If we already did this grant, don't do it again. */
if (done[i - start])
@@ -1223,8 +1389,8 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ appendPQExpBuffer(creaQry, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(creaQry, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1245,10 +1411,15 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(creaQry, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(creaQry, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(creaQry, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", creaQry->data);
+ else
+ createOneArchiveEntry(creaQry->data, "dumpRoleMembership");
}
}
@@ -1260,7 +1431,8 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1288,7 +1460,12 @@ dumpRoleGUCPrivs(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role privileges on configuration parameters\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1312,14 +1489,19 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoleGUCPrivs");
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1331,6 +1513,7 @@ dropTablespaces(PGconn *conn)
{
PGresult *res;
int i;
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Get all tablespaces except built-in ones (which we assume are named
@@ -1342,20 +1525,31 @@ dropTablespaces(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(spcname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropTablespaces");
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1382,7 +1576,12 @@ dumpTablespaces(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1451,14 +1650,19 @@ dumpTablespaces(PGconn *conn)
"TABLESPACE", spcname,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpTablespaces");
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1482,7 +1686,12 @@ dropDBs(PGconn *conn)
"ORDER BY datname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop databases (except postgres and template1)\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1497,15 +1706,23 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(dbname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "DROP_DATABASE");
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1532,7 +1749,18 @@ dumpUserConfig(PGconn *conn, const char *username)
char *sanitized;
sanitized = sanitize_line(username, true);
- fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
}
@@ -1542,7 +1770,11 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpUserConfig");
}
PQclear(res);
@@ -1608,10 +1840,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1625,19 +1860,48 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Databases\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Databases\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Databases\n--\n\n", "COMMENT");
+ }
+
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1654,7 +1918,18 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
/*
@@ -1669,24 +1944,46 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
+ fprintf(OPF, "\\connect %s\n\n", dbname);
else
{
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\\connect %s\n\n", dbname);
+ createOneArchiveEntry(qry->data, "CONNECT");
+ destroyPQExpBuffer(qry);
}
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1695,6 +1992,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1704,7 +2005,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1713,17 +2015,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1807,7 +2128,18 @@ dumpTimestamp(const char *msg)
time_t now = time(NULL);
if (strftime(buf, sizeof(buf), PGDUMP_STRFTIME_FMT, localtime(&now)) != 0)
- fprintf(OPF, "-- %s %s\n\n", msg, buf);
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "-- %s %s\n\n", msg, buf);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "-- %s %s\n\n", msg, buf);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+ }
}
/*
@@ -1868,3 +2200,66 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+/*
+ * createDumpId
+ *
+ * This will return next last used oid.
+ */
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
+
+/*
+ * createOneArchiveEntry
+ *
+ * This creates one archive entry based on format.
+ */
+static void
+createOneArchiveEntry(const char *query, const char *tag)
+{
+ Assert(fout != NULL);
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = tag,
+ .section = SECTION_PRE_DATA,
+ .createStmt = query));
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c9776306c5c..97a6bcb6d31 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,31 +41,61 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num,
+ bool globals_only);
+static int restore_global_objects(const char *inputFileSpec,
+ RestoreOptions *opts, int numWorkers, bool append_data,
+ int num, bool globals_only);
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +119,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +173,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +202,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +229,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global sql commands. */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -321,6 +356,10 @@ main(int argc, char **argv)
opts->restrict_key = pg_strdup(optarg);
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -347,6 +386,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -472,6 +518,105 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If map.dat file is present, then restore all the
+ * databases from map.dat , but skip restoring those matching
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "map.dat") ||
+ file_exists_in_directory(inputFileSpec, "toc.tar") ||
+ file_exists_in_directory(inputFileSpec, "toc.dmp")))
+ {
+ char global_path[MAXPGPATH];
+
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring an archive created by pg_dumpall");
+ else if (opts->tocFile)
+ pg_fatal("option -L/--use-list cannot be used when restoring an archive created by pg_dumpall");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option -C/--create must be specified when restoring an archive created by pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ n_errors = restore_global_objects(global_path, opts, numWorkers, false, 0, globals_only);
+
+ pg_log_info("database restoring skipped because option -g/--globals-only was specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else /* process if map.dat file does not exist. */
+ n_errors = restore_one_database(inputFileSpec, opts,
+ numWorkers, false, 0, globals_only);
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ *
+ * If globals_only is set, then skip DROP DATABASE commands from restore.
+ */
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ return restore_one_database(inputFileSpec, opts, numWorkers,
+ append_data, num, globals_only);
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -479,9 +624,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -501,25 +652,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, globals_only);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -537,6 +684,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -553,6 +701,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -588,8 +737,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -694,3 +843,415 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn && db_exclude_patterns.head != NULL)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no database connection while doing pg_restore");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+ PQExpBuffer db_lit = createPQExpBuffer();
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches exclude pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ destroyPQExpBuffer(db_lit);
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ destroyPQExpBuffer(query);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+
+
+ /*
+ * If there is no map.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen <= 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbname, db_oid, map_file_path);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+ PGconn *conn = NULL;
+ char global_path[MAXPGPATH];
+
+ /* Based on file, set path. */
+ if (file_exists_in_directory(inputFileSpec, "toc.tar"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.tar", inputFileSpec);
+ else if (file_exists_in_directory(inputFileSpec, "toc.dmp"))
+ snprintf(global_path, MAXPGPATH, "%s/toc.dmp", inputFileSpec);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", inputFileSpec);
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
+
+ /* If map.dat has no entries, return after processing global commands. */
+ if (dbname_oid_list.head == NULL)
+ return restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Open toc.dat file and execute/append all the global sql commands. */
+ n_errors_total = restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ count++;
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..e8f800a48c1
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,12 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +250,13 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--restrict-key=uu', '-f dumpfile' ],
+ qr/\Qpg_dumpall: error: option --restrict-key can only be used with --format=plain\E/,
+ 'pg_dumpall: --restrict-key can only be used with plain dump format');
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100755
index 00000000000..3c7d2ad7c53
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,396 @@
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# Scape tablespace locations on Windows.
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT super TO grant7 WITH INHERIT TRUE GRANTED BY\E
+ (.*\n)*
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: When --list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: When non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+$node->stop('fast');
+
+done_testing();
--
2.39.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-11 15:11 Andrew Dunstan <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
1 sibling, 1 reply; 65+ messages in thread
From: Andrew Dunstan @ 2025-11-11 15:11 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; Vaibhav Dalvi <[email protected]>; +Cc: [email protected]
On 2025-11-11 Tu 12:59 AM, Mahendra Singh Thalor wrote:
>
> Hi,
> Here, I am attaching an updated patch for the review and testing.
>
> FIX: as suggested by Vaibhav, added error for --restrict-key option
> with non-text format.
>
Regarding the name and format of the globals toc file, I'm inclined to
think we should always use custom format, regardless of whether the
individual databases will be in custom, tar or directory formats, and
that it should be called something distinguishable, e.g. toc.glo.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-17 14:08 tushar <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
1 sibling, 0 replies; 65+ messages in thread
From: tushar @ 2025-11-17 14:08 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
On Tue, Nov 11, 2025 at 11:29 AM Mahendra Singh Thalor <[email protected]>
wrote:
> On Thu, 6 Nov 2025 at 11:03, Mahendra Singh Thalor <[email protected]>
> wrote:
> >
> > Thanks Vaibhav, Tushar and Andrew for the review and testing.
>
>
Thanks Mahendra, getting this error against v07 series patch
[edb@1a1c15437e7c bin]$ ./pg_dumpall -Ft -f tar.dumpc -v
pg_dumpall: executing SELECT pg_catalog.set_config('search_path', '',
false);
pg_dumpall: pg_dumpall.c:2256: createOneArchiveEntry: Assertion `fout !=
((void *)0)' failed.
Aborted
regards,
Tushar Ahuja
EDB https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-17 17:15 Mahendra Singh Thalor <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-11-17 17:15 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
Thanks Andrew for the review.
On Tue, 11 Nov 2025 at 20:41, Andrew Dunstan <[email protected]> wrote:
>
>
> On 2025-11-11 Tu 12:59 AM, Mahendra Singh Thalor wrote:
> >
> > Hi,
> > Here, I am attaching an updated patch for the review and testing.
> >
> > FIX: as suggested by Vaibhav, added error for --restrict-key option
> > with non-text format.
> >
>
>
> Regarding the name and format of the globals toc file, I'm inclined to
> think we should always use custom format, regardless of whether the
> individual databases will be in custom, tar or directory formats, and
> that it should be called something distinguishable, e.g. toc.glo.
>
I also agree with your point. Fixed.
On Mon, 17 Nov 2025 at 19:38, tushar <[email protected]> wrote:
>
>
>
> On Tue, Nov 11, 2025 at 11:29 AM Mahendra Singh Thalor <[email protected]> wrote:
>>
>> On Thu, 6 Nov 2025 at 11:03, Mahendra Singh Thalor <[email protected]> wrote:
>> >
>> > Thanks Vaibhav, Tushar and Andrew for the review and testing.
>>
>
> Thanks Mahendra, getting this error against v07 series patch
>
> [edb@1a1c15437e7c bin]$ ./pg_dumpall -Ft -f tar.dumpc -v
> pg_dumpall: executing SELECT pg_catalog.set_config('search_path', '', false);
> pg_dumpall: pg_dumpall.c:2256: createOneArchiveEntry: Assertion `fout != ((void *)0)' failed.
> Aborted
>
> regards,
Thanks Tushar for the report. Fixed.
Here, I am attaching an updated patch for the review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v08_17112025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (83.9K, 2-v08_17112025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
From d70167443370f1396f0f78485f746742fb92a821 Mon Sep 17 00:00:00 2001
From: ThalorMahendra <[email protected]>
Date: Mon, 17 Nov 2025 22:35:35 +0530
Subject: [PATCH] Non text modes for pg_dumpall, correspondingly change
pg_restore
pg_dumpall acquires a new -F/--format option, with the same meanings as
pg_dump. The default is p, meaning plain text. For any other value, a
directory is created containing two files, toc.glo and map.dat. The
first contains commands restoring the global data in custom format, and the second
contains a map from oids to database names in text format. It will also contain a
subdirectory called databases, inside which it will create archives in
the specified format, named using the database oids.
In these casess the -f argument is required.
If pg_restore encounters a directory containing map.dat and toc.glo,
it restores the global settings from toc.glo if exist, and then
restores each database.
pg_restore acquires two new options: -g/--globals-only which suppresses
restoration of any databases, and --exclude-database which inhibits
restoration of particualr database(s) in the same way the same option
works in pg_dumpall.
v08
---
doc/src/sgml/ref/pg_dumpall.sgml | 89 +++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/connectdb.c | 1 -
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 10 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 29 +-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 592 +++++++++++++++++++++-----
src/bin/pg_dump/pg_restore.c | 606 ++++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 27 ++
src/bin/pg_dump/t/007_pg_dumpall.pl | 396 +++++++++++++++++
14 files changed, 1677 insertions(+), 147 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
create mode 100755 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 9f639f61db0..4063e88d388 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,10 +139,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>toc.dat/toc.dmp/toc.tar</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index a468a38361a..7497b527ae6 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -591,6 +615,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
index d55d53dbeea..f44a8a45fca 100644
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -287,7 +287,6 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
exit_nicely(1);
}
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index f3c669f484e..3e21aaf5780 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..5974d6706fd 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index d9041dad720..f631d945472 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -312,7 +312,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, bool globals_only);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index c84b017f21b..5b8dd295070 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, bool globals_only)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,15 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ /* Skip DROP DATABASE if globals_only. */
+ if (globals_only && te && te->tag && (strcmp(te->tag, "DROP_DATABASE") == 0))
+ continue;
+
+ /* Skip for CONNECT meta command. */
+ if (!ropt->filename && te && te->tag &&
+ (strcmp(te->tag, "CONNECT") == 0))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1330,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1695,7 +1709,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1730,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..818b80a9369 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a00918bacb4..13e1764ec70 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index bb451c1bae1..0bf892b1fcc 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -30,6 +30,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -65,9 +66,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,6 +78,9 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
+static void createOneArchiveEntry(const char *query, const char *tag);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -123,6 +128,13 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static const CatalogId nilCatalogId = {0, 0};
+static ArchiveMode archiveMode = archModeWrite;
+static DataDirSyncMethod sync_method = DATA_DIR_SYNC_METHOD_FSYNC;
+static ArchiveFormat archDumpFormat = archNull;
int
main(int argc, char *argv[])
@@ -148,6 +160,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +210,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -208,6 +222,7 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -246,7 +261,9 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ InitDumpOptions(&dopt);
+
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -257,6 +274,7 @@ main(int argc, char *argv[])
case 'c':
output_clean = true;
+ dopt.outputClean = 1;
break;
case 'd':
@@ -274,7 +292,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -314,6 +334,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -429,6 +450,25 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
+ /* restrict-key is only supported with --format=plain */
+ if (archDumpFormat != archNull && restrict_key)
+ pg_fatal("option --restrict-key can only be used with --format=plain");
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -489,6 +529,27 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -538,19 +599,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -585,37 +633,110 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
- if (verbose)
+ if (verbose && archDumpFormat == archNull)
dumpTimestamp("Started on");
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ /* create a archive file for global commands. */
+ if (filename && archDumpFormat != archNull)
+ {
+ char global_path[MAXPGPATH];
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
+ /* Set file path for global sql commands. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", filename);
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+ /* Open the output file */
+ fout = CreateArchive(global_path, archCustom, compression_spec,
+ dosync, archiveMode, NULL, sync_method);
+
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
+ ((ArchiveHandle*)fout)->connection = conn;
+ ((ArchiveHandle*)fout)->public.numWorkers = 1;
+
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
- /* Replicate encoding and std_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
- if (strcmp(std_strings, "off") == 0)
- fprintf(OPF, "SET escape_string_warning = off;\n");
- fprintf(OPF, "\n");
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump\n--\n\n", "COMMENT");
+
+ /* default_transaction_read_only = off */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving default_transaction_read_only = off");
+ appendPQExpBuffer(qry, "SET default_transaction_read_only = off;\n");
+ createOneArchiveEntry(qry->data, "DEFAULT_TRANSACTION_READ_ONLY");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpEncoding: put the correct encoding into the archive */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ const char *encname = pg_encoding_to_char(encoding);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+
+ pg_log_info("saving encoding = %s", encname);
+ createOneArchiveEntry(qry->data, "ENCODING");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpStdStrings: put the correct escape string behavior into the archive */
+ {
+ const char *stdstrings = std_strings ? "on" : "off";
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving \"standard_conforming_strings = %s\"", stdstrings);
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = '%s';\n",
+ stdstrings);
+ createOneArchiveEntry(qry->data, "STDSTRINGS");
+ destroyPQExpBuffer(qry);
+ }
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and std_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
+ if (strcmp(std_strings, "off") == 0)
+ fprintf(OPF, "SET escape_string_warning = off;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -659,27 +780,42 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump will
+ * handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
- PQfinish(conn);
-
- if (verbose)
+ if (verbose && archDumpFormat == archNull)
dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (filename)
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+
+ if (archDumpFormat != archNull)
+ {
+ RestoreOptions *ropt;
+
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump complete\n--\n\n", "COMMENT");
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
+
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
+ }
+ else if (filename)
{
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -690,12 +826,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -770,6 +908,7 @@ static void
dropRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer delQry = createPQExpBuffer();
PGresult *res;
int i_rolname;
int i;
@@ -791,7 +930,12 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -799,15 +943,21 @@ dropRoles(PGconn *conn)
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(rolename));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropRoles");
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -889,7 +1039,12 @@ dumpRoles(PGconn *conn)
i_is_current_user = PQfnumber(res, "is_current_user");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -993,7 +1148,10 @@ dumpRoles(PGconn *conn)
"ROLE", rolename,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoles");
}
/*
@@ -1001,15 +1159,13 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
}
@@ -1088,7 +1244,12 @@ dumpRoleMembership(PGconn *conn)
i_set_option = PQfnumber(res, "set_option");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role memberships\n--\n\n", "COMMENT");
+ }
/*
* We can't dump these GRANT commands in arbitrary order, because a role
@@ -1167,6 +1328,7 @@ dumpRoleMembership(PGconn *conn)
char *grantor;
char *set_option = "true";
bool found;
+ PQExpBuffer creaQry = createPQExpBuffer();
/* If we already did this grant, don't do it again. */
if (done[i - start])
@@ -1223,8 +1385,8 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ appendPQExpBuffer(creaQry, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(creaQry, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1245,10 +1407,15 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(creaQry, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(creaQry, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(creaQry, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", creaQry->data);
+ else
+ createOneArchiveEntry(creaQry->data, "dumpRoleMembership");
}
}
@@ -1260,7 +1427,8 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1288,7 +1456,12 @@ dumpRoleGUCPrivs(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role privileges on configuration parameters\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1312,14 +1485,19 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoleGUCPrivs");
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1331,6 +1509,7 @@ dropTablespaces(PGconn *conn)
{
PGresult *res;
int i;
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Get all tablespaces except built-in ones (which we assume are named
@@ -1342,20 +1521,31 @@ dropTablespaces(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(spcname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropTablespaces");
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1382,7 +1572,12 @@ dumpTablespaces(PGconn *conn)
"ORDER BY 1");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1451,14 +1646,19 @@ dumpTablespaces(PGconn *conn)
"TABLESPACE", spcname,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpTablespaces");
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1482,7 +1682,12 @@ dropDBs(PGconn *conn)
"ORDER BY datname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop databases (except postgres and template1)\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1497,15 +1702,23 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(dbname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "DROP_DATABASE");
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1532,7 +1745,18 @@ dumpUserConfig(PGconn *conn, const char *username)
char *sanitized;
sanitized = sanitize_line(username, true);
- fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
}
@@ -1542,7 +1766,11 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpUserConfig");
}
PQclear(res);
@@ -1608,10 +1836,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1625,19 +1856,48 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Databases\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Databases\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Databases\n--\n\n", "COMMENT");
+ }
+
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1654,7 +1914,18 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
/*
@@ -1669,24 +1940,46 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
+ fprintf(OPF, "\\connect %s\n\n", dbname);
else
{
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\\connect %s\n\n", dbname);
+ createOneArchiveEntry(qry->data, "CONNECT");
+ destroyPQExpBuffer(qry);
}
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1695,6 +1988,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1704,7 +2001,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1713,17 +2011,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1868,3 +2185,66 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+/*
+ * createDumpId
+ *
+ * This will return next last used oid.
+ */
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
+
+/*
+ * createOneArchiveEntry
+ *
+ * This creates one archive entry based on format.
+ */
+static void
+createOneArchiveEntry(const char *query, const char *tag)
+{
+ Assert(fout != NULL);
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = tag,
+ .section = SECTION_PRE_DATA,
+ .createStmt = query));
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c9776306c5c..ea9f64637ea 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,31 +41,61 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
#include "pg_backup_utils.h"
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num,
+ bool globals_only);
+static int restore_global_objects(const char *inputFileSpec,
+ RestoreOptions *opts, int numWorkers, bool append_data,
+ int num, bool globals_only);
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +119,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +173,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +202,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +229,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global sql commands. */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -321,6 +356,10 @@ main(int argc, char **argv)
opts->restrict_key = pg_strdup(optarg);
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -347,6 +386,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -472,6 +518,121 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.glo file is present, then restore all the
+ * databases from map.dat , but skip restoring those matching
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "toc.glo")))
+ {
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring an archive created by pg_dumpall");
+ else if (opts->tocFile)
+ pg_fatal("option -L/--use-list cannot be used when restoring an archive created by pg_dumpall");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option -C/--create must be specified when restoring an archive created by pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ char global_path[MAXPGPATH];
+
+ /* Set path for toc.glo file. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+ n_errors = restore_global_objects(global_path, opts, numWorkers, false, 0, globals_only);
+
+ pg_log_info("database restoring skipped because option -g/--globals-only was specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else
+ {
+ if (db_exclude_patterns.head != NULL)
+ {
+ simple_string_list_destroy(&db_exclude_patterns);
+ pg_fatal("option --exclude-database can be used only when restoring an archive created by pg_dumpall");
+ }
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring an archive created by pg_dumpall");
+
+ /* Process if toc.glo file does not exist. */
+ n_errors = restore_one_database(inputFileSpec, opts,
+ numWorkers, false, 0, globals_only);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ *
+ * If globals_only is set, then skip DROP DATABASE commands from restore.
+ */
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ int nerror;
+ int format = opts->format;
+
+ /* Set format as custom so that toc.glo file can be read. */
+ opts->format = archCustom;
+
+ nerror = restore_one_database(inputFileSpec, opts, numWorkers,
+ append_data, num, globals_only);
+
+ /* Reset format value. */
+ opts->format = format;
+
+ return nerror;
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -479,9 +640,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -501,25 +668,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, globals_only);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -537,6 +700,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -553,6 +717,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -588,8 +753,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -694,3 +859,410 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn && db_exclude_patterns.head != NULL)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no database connection while doing pg_restore");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+ PQExpBuffer db_lit = createPQExpBuffer();
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches exclude pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ destroyPQExpBuffer(db_lit);
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ destroyPQExpBuffer(query);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+
+
+ /*
+ * If there is no map.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen <= 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbname, db_oid, map_file_path);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+ PGconn *conn = NULL;
+ char global_path[MAXPGPATH];
+
+ /* Set path for toc.glo file. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
+
+ /* If map.dat has no entries, return after processing global commands. */
+ if (dbname_oid_list.head == NULL)
+ return restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Open toc.dat file and execute/append all the global sql commands. */
+ n_errors_total = restore_global_objects(global_path, opts, numWorkers, false, 0, false);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ count++;
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..083f5c5bf9d
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -244,4 +244,31 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--restrict-key=uu', '-f dumpfile' ],
+ qr/\Qpg_dumpall: error: option --restrict-key can only be used with --format=plain\E/,
+ 'pg_dumpall: --restrict-key can only be used with plain dump format');
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option --exclude-database can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --exclude-database is used in pg_restore with dump of pg_dump'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--globals-only', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option -g\/--globals-only can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --globals-only is not used in pg_restore with dump of pg_dump'
+);
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100755
index 00000000000..3c7d2ad7c53
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,396 @@
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# Scape tablespace locations on Windows.
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT super TO grant7 WITH INHERIT TRUE GRANTED BY\E
+ (.*\n)*
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: When --list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: When non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+$node->stop('fast');
+
+done_testing();
--
2.39.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-18 10:34 Vaibhav Dalvi <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Vaibhav Dalvi @ 2025-11-18 10:34 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: [email protected]
Hi Mahendra,
Thanks Mahendra for working on this.
Looks like my previous comment below is not addressed:
1.
### Use of Dump Options Structure (dopt)
> Please ensure consistency by utilizing the main dump options
> structure (`dopt`) instead of declaring and using individual variables
> where the structure already provides fields. For example, the
> `output_clean` variable seems redundant here:
> ```c
> case 'c':
> output_clean = true;
> dopt.outputClean = 1;
> break;
> ```
I agree that the output_clean variable is not added by your patch
but the introduction of dopt by your patch makes it redundant because
dopt has dopt.outputClean. Please look at below code from pg_dump.c
for the reference:
case 'c': /* clean (i.e., drop) schema prior to create */
dopt.outputClean = 1;
break;
case 25:
dopt.restrict_key = pg_strdup(optarg);
break;
2.
### 3\. Missing Example in SGML Documentation
> The SGML documentation for `pg_dumpall` is missing an explicit
> example demonstrating its use with non-text formats (e.g., directory
> format).
> It would be beneficial to include a clear example for this new feature.
I think pg_dumpall should have separate examples similar to pg_dump
rather than referencing the pg_dump example because pg_dumpall
doesn't have to mention the database name without -l or --database
in the command.
3.
> > 1. Is the following change in `src/bin/pg_dump/connectdb.c` intentional?
>
> >
> > ```
> > --- a/src/bin/pg_dump/connectdb.c
> > +++ b/src/bin/pg_dump/connectdb.c
> Yes, we need this. If there is any error, then we were trying to
> disconnect the database in 2 places so we were getting a crash. I will
> try to reproduce crashe without this patch and will respond.
>
> Have you added a test case in the regression suite which fails if we remove
this particular change and works well with the change? or if possible could
you please demonstrate here at least.
4. The variable name *append_data* doesn't look meaningful to me.
Instead we can use *append_database/**append_databases*?
because if this variable is set then we dump the databases along with
global objects. In case of pg_dump, append_data or data_only does make
sense to differentiate between schema and data but in case of pg_dumpall
if this variable is set then we're dumping schema as well as data i.e.
in-short
the databases.
------------------------------------ pg_dumpall.c
----------------------------------------
5. The variable name formatName doesn't follow the naming convention of
variables available around it. I think use of format_name/formatname would
be better.
char *use_role = NULL;
> const char *dumpencoding = NULL;
> + const char *formatName = "p";
> trivalue prompt_password = TRI_DEFAULT;
> bool data_only = false;
> bool globals_only = false;
------------------------------------ pg_restore.c
----------------------------------------
6. Fourth parameter (i.e. append_data) to function restore_global_objects()
is redundant.
All the time value provided by all callers to this parameter is false.
I would suggest removing this parameter and in the definition of this
function
call function restore_one_database() with false as 4th argument. Find diff
below:
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -64,8 +64,7 @@ static int restore_one_database(const char
*inputFileSpec, RestoreOptions *opts,
int
numWorkers, bool append_data, int num,
bool
globals_only);
static int restore_global_objects(const char *inputFileSpec,
- RestoreOptions *opts, int numWorkers, bool append_data,
- int num, bool globals_only);
+ RestoreOptions *opts, int numWorkers, int num, bool
globals_only);
static int restore_all_databases(const char *inputFileSpec,
SimpleStringList db_exclude_patterns, RestoreOptions *opts,
int numWorkers);
static int get_dbnames_list_to_restore(PGconn *conn,
@@ -554,7 +553,7 @@ main(int argc, char **argv)
/* Set path for toc.glo file. */
snprintf(global_path, MAXPGPATH, "%s/toc.glo",
inputFileSpec);
- n_errors = restore_global_objects(global_path,
opts, numWorkers, false, 0, globals_only);
+ n_errors = restore_global_objects(global_path,
opts, numWorkers, 0, globals_only);
pg_log_info("database restoring skipped because
option -g/--globals-only was specified");
}
@@ -602,7 +601,7 @@ main(int argc, char **argv)
* If globals_only is set, then skip DROP DATABASE commands from restore.
*/
static int restore_global_objects(const char *inputFileSpec,
RestoreOptions *opts,
- int numWorkers, bool append_data, int num, bool
globals_only)
+ int numWorkers, int num, bool globals_only)
{
int nerror;
int format = opts->format;
@@ -610,8 +609,8 @@ static int restore_global_objects(const char
*inputFileSpec, RestoreOptions *opt
/* Set format as custom so that toc.glo file can be read. */
opts->format = archCustom;
- nerror = restore_one_database(inputFileSpec, opts, numWorkers,
- append_data, num, globals_only);
+ nerror = restore_one_database(inputFileSpec, opts, numWorkers,
false, num,
+
globals_only);
/* Reset format value. */
opts->format = format;
@@ -1097,7 +1096,7 @@ restore_all_databases(const char *inputFileSpec,
/* If map.dat has no entries, return after processing global
commands. */
if (dbname_oid_list.head == NULL)
- return restore_global_objects(global_path, opts,
numWorkers, false, 0, false);
+ return restore_global_objects(global_path, opts,
numWorkers, 0, false);
pg_log_info(ngettext("found %d database name in \"%s\"",
"found %d database names
in \"%s\"",
@@ -1151,7 +1150,7 @@ restore_all_databases(const char *inputFileSpec,
PQfinish(conn);
/* Open toc.dat file and execute/append all the global sql
commands. */
- n_errors_total = restore_global_objects(global_path, opts,
numWorkers, false, 0, false);
+ n_errors_total = restore_global_objects(global_path, opts,
numWorkers, 0, false);
Regression is successful with these changes.
7. Fix indentation:
> static int restore_global_objects(const char *inputFileSpec,
> RestoreOptions *opts, int numWorkers, bool append_data,
> int num, bool globals_only);
> static int restore_all_databases(const char *inputFileSpec,
> SimpleStringList db_exclude_patterns, RestoreOptions *opts, int
> numWorkers);
8. Remove extra line:
> +
> static void usage(const char *progname);
9. Remove extra space after map.dat and before comma:
> + * databases from map.dat , but skip restoring those matching
10. Fix 80 char limits:
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1,
false);
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec,
&dbname_oid_list);
+ return restore_global_objects(global_path, opts, numWorkers, false, 0,
false);
+ n_errors_total = restore_global_objects(global_path, opts, numWorkers,
false, 0, false);
+ pg_log_warning("errors ignored on database \"%s\" restore: %d",
dbidname->str, n_errors);
Regards,
Vaibhav
On Mon, Nov 17, 2025 at 10:45 PM Mahendra Singh Thalor <[email protected]>
wrote:
> Thanks Andrew for the review.
> On Tue, 11 Nov 2025 at 20:41, Andrew Dunstan <[email protected]> wrote:
> >
> >
> > On 2025-11-11 Tu 12:59 AM, Mahendra Singh Thalor wrote:
> > >
> > > Hi,
> > > Here, I am attaching an updated patch for the review and testing.
> > >
> > > FIX: as suggested by Vaibhav, added error for --restrict-key option
> > > with non-text format.
> > >
> >
> >
> > Regarding the name and format of the globals toc file, I'm inclined to
> > think we should always use custom format, regardless of whether the
> > individual databases will be in custom, tar or directory formats, and
> > that it should be called something distinguishable, e.g. toc.glo.
> >
>
> I also agree with your point. Fixed.
>
> On Mon, 17 Nov 2025 at 19:38, tushar <[email protected]>
> wrote:
> >
> >
> >
> > On Tue, Nov 11, 2025 at 11:29 AM Mahendra Singh Thalor <
> [email protected]> wrote:
> >>
> >> On Thu, 6 Nov 2025 at 11:03, Mahendra Singh Thalor <[email protected]>
> wrote:
> >> >
> >> > Thanks Vaibhav, Tushar and Andrew for the review and testing.
> >>
> >
> > Thanks Mahendra, getting this error against v07 series patch
> >
> > [edb@1a1c15437e7c bin]$ ./pg_dumpall -Ft -f tar.dumpc -v
> > pg_dumpall: executing SELECT pg_catalog.set_config('search_path', '',
> false);
> > pg_dumpall: pg_dumpall.c:2256: createOneArchiveEntry: Assertion `fout !=
> ((void *)0)' failed.
> > Aborted
> >
> > regards,
>
> Thanks Tushar for the report. Fixed.
>
> Here, I am attaching an updated patch for the review and testing.
>
> --
> Thanks and Regards
> Mahendra Singh Thalor
> EnterpriseDB: http://www.enterprisedb.com
>
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-27 08:15 Mahendra Singh Thalor <[email protected]>
parent: Vaibhav Dalvi <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-11-27 08:15 UTC (permalink / raw)
To: Vaibhav Dalvi <[email protected]>; +Cc: [email protected]
Thanks Vaibhav for the review.
On Tue, 18 Nov 2025 at 16:05, Vaibhav Dalvi
<[email protected]> wrote:
>
> Hi Mahendra,
>
> Thanks Mahendra for working on this.
>
> Looks like my previous comment below is not addressed:
> 1.
>
>> ### Use of Dump Options Structure (dopt)
>> Please ensure consistency by utilizing the main dump options
>> structure (`dopt`) instead of declaring and using individual variables
>> where the structure already provides fields. For example, the
>> `output_clean` variable seems redundant here:
>> ```c
>> case 'c':
>> output_clean = true;
>> dopt.outputClean = 1;
>> break;
>> ```
>
Fixed. output_clean was a global variable because it was used in 2
functions. Now I am passing dopt. output_clean as function argument
for another function.
>
> I agree that the output_clean variable is not added by your patch
> but the introduction of dopt by your patch makes it redundant because
> dopt has dopt.outputClean. Please look at below code from pg_dump.c
> for the reference:
>
> case 'c': /* clean (i.e., drop) schema prior to create */
> dopt.outputClean = 1;
> break;
> case 25:
> dopt.restrict_key = pg_strdup(optarg);
> break;
>
> 2.
>
>> ### 3\. Missing Example in SGML Documentation
>> The SGML documentation for `pg_dumpall` is missing an explicit
>> example demonstrating its use with non-text formats (e.g., directory format).
>> It would be beneficial to include a clear example for this new feature.
>
>
> I think pg_dumpall should have separate examples similar to pg_dump
> rather than referencing the pg_dump example because pg_dumpall
> doesn't have to mention the database name without -l or --database
> in the command.
>
Fixed. Added some examples.
> 3.
>>
>> > 1. Is the following change in `src/bin/pg_dump/connectdb.c` intentional?
>>
>> >
>> > ```
>> > --- a/src/bin/pg_dump/connectdb.c
>> > +++ b/src/bin/pg_dump/connectdb.c
>> Yes, we need this. If there is any error, then we were trying to
>> disconnect the database in 2 places so we were getting a crash. I will
>> try to reproduce crashe without this patch and will respond.
>
> Have you added a test case in the regression suite which fails if we remove
> this particular change and works well with the change? or if possible could
> you please demonstrate here at least.
Fixed. With AH(archive), we should not free pointers by this exec call
as we free this by exit_nicely hook. (we register AH by
on_exit_close_archive).
>
> 4. The variable name append_data doesn't look meaningful to me.
> Instead we can use append_database/append_databases?
> because if this variable is set then we dump the databases along with
> global objects. In case of pg_dump, append_data or data_only does make
> sense to differentiate between schema and data but in case of pg_dumpall
> if this variable is set then we're dumping schema as well as data i.e. in-short
> the databases.
>
As of now, I am keeping this append_data as this was from an already
committed patch.
> ------------------------------------ pg_dumpall.c ----------------------------------------
>
> 5. The variable name formatName doesn't follow the naming convention of
> variables available around it. I think use of format_name/formatname would
> be better.
>
>> char *use_role = NULL;
>> const char *dumpencoding = NULL;
>> + const char *formatName = "p";
>> trivalue prompt_password = TRI_DEFAULT;
>> bool data_only = false;
>> bool globals_only = false;
>
Fixed.
>
> ------------------------------------ pg_restore.c ----------------------------------------
>
> 6. Fourth parameter (i.e. append_data) to function restore_global_objects() is redundant.
> All the time value provided by all callers to this parameter is false.
>
> I would suggest removing this parameter and in the definition of this function
> call function restore_one_database() with false as 4th argument. Find diff below:
>
Fixed.
> --- a/src/bin/pg_dump/pg_restore.c
> +++ b/src/bin/pg_dump/pg_restore.c
> @@ -64,8 +64,7 @@ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
> int numWorkers, bool append_data, int num,
> bool globals_only);
> static int restore_global_objects(const char *inputFileSpec,
> - RestoreOptions *opts, int numWorkers, bool append_data,
> - int num, bool globals_only);
> + RestoreOptions *opts, int numWorkers, int num, bool globals_only);
> static int restore_all_databases(const char *inputFileSpec,
> SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
> static int get_dbnames_list_to_restore(PGconn *conn,
> @@ -554,7 +553,7 @@ main(int argc, char **argv)
>
> /* Set path for toc.glo file. */
> snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
> - n_errors = restore_global_objects(global_path, opts, numWorkers, false, 0, globals_only);
> + n_errors = restore_global_objects(global_path, opts, numWorkers, 0, globals_only);
>
> pg_log_info("database restoring skipped because option -g/--globals-only was specified");
> }
> @@ -602,7 +601,7 @@ main(int argc, char **argv)
> * If globals_only is set, then skip DROP DATABASE commands from restore.
> */
> static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
> - int numWorkers, bool append_data, int num, bool globals_only)
> + int numWorkers, int num, bool globals_only)
> {
> int nerror;
> int format = opts->format;
> @@ -610,8 +609,8 @@ static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opt
> /* Set format as custom so that toc.glo file can be read. */
> opts->format = archCustom;
>
> - nerror = restore_one_database(inputFileSpec, opts, numWorkers,
> - append_data, num, globals_only);
> + nerror = restore_one_database(inputFileSpec, opts, numWorkers, false, num,
> + globals_only);
>
> /* Reset format value. */
> opts->format = format;
> @@ -1097,7 +1096,7 @@ restore_all_databases(const char *inputFileSpec,
>
> /* If map.dat has no entries, return after processing global commands. */
> if (dbname_oid_list.head == NULL)
> - return restore_global_objects(global_path, opts, numWorkers, false, 0, false);
> + return restore_global_objects(global_path, opts, numWorkers, 0, false);
>
> pg_log_info(ngettext("found %d database name in \"%s\"",
> "found %d database names in \"%s\"",
> @@ -1151,7 +1150,7 @@ restore_all_databases(const char *inputFileSpec,
> PQfinish(conn);
>
> /* Open toc.dat file and execute/append all the global sql commands. */
> - n_errors_total = restore_global_objects(global_path, opts, numWorkers, false, 0, false);
> + n_errors_total = restore_global_objects(global_path, opts, numWorkers, 0, false);
>
> Regression is successful with these changes.
>
> 7. Fix indentation:
>>
>> static int restore_global_objects(const char *inputFileSpec,
>> RestoreOptions *opts, int numWorkers, bool append_data,
>> int num, bool globals_only);
>> static int restore_all_databases(const char *inputFileSpec,
>> SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
Fixed some.
>
>
> 8. Remove extra line:
>>
>> +
>> static void usage(const char *progname);
>
Fixed.
>
> 9. Remove extra space after map.dat and before comma:
>>
>> + * databases from map.dat , but skip restoring those matching
>
Fixed.
>
> 10. Fix 80 char limits:
>
> + n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
>
> + num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
>
> + return restore_global_objects(global_path, opts, numWorkers, false, 0, false);
>
> + n_errors_total = restore_global_objects(global_path, opts, numWorkers, false, 0, false);
>
> + pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
>
Fixed some.
I will do some more cleanup in the coming versions.
Here, I am attaching an updated patch for the review and testing.
>
> Regards,
> Vaibhav
>
> On Mon, Nov 17, 2025 at 10:45 PM Mahendra Singh Thalor <[email protected]> wrote:
>>
>> Thanks Andrew for the review.
>> On Tue, 11 Nov 2025 at 20:41, Andrew Dunstan <[email protected]> wrote:
>> >
>> >
>> > On 2025-11-11 Tu 12:59 AM, Mahendra Singh Thalor wrote:
>> > >
>> > > Hi,
>> > > Here, I am attaching an updated patch for the review and testing.
>> > >
>> > > FIX: as suggested by Vaibhav, added error for --restrict-key option
>> > > with non-text format.
>> > >
>> >
>> >
>> > Regarding the name and format of the globals toc file, I'm inclined to
>> > think we should always use custom format, regardless of whether the
>> > individual databases will be in custom, tar or directory formats, and
>> > that it should be called something distinguishable, e.g. toc.glo.
>> >
>>
>> I also agree with your point. Fixed.
>>
>> On Mon, 17 Nov 2025 at 19:38, tushar <[email protected]> wrote:
>> >
>> >
>> >
>> > On Tue, Nov 11, 2025 at 11:29 AM Mahendra Singh Thalor <[email protected]> wrote:
>> >>
>> >> On Thu, 6 Nov 2025 at 11:03, Mahendra Singh Thalor <[email protected]> wrote:
>> >> >
>> >> > Thanks Vaibhav, Tushar and Andrew for the review and testing.
>> >>
>> >
>> > Thanks Mahendra, getting this error against v07 series patch
>> >
>> > [edb@1a1c15437e7c bin]$ ./pg_dumpall -Ft -f tar.dumpc -v
>> > pg_dumpall: executing SELECT pg_catalog.set_config('search_path', '', false);
>> > pg_dumpall: pg_dumpall.c:2256: createOneArchiveEntry: Assertion `fout != ((void *)0)' failed.
>> > Aborted
>> >
>> > regards,
>>
>> Thanks Tushar for the report. Fixed.
>>
>> Here, I am attaching an updated patch for the review and testing.
>>
>> --
>> Thanks and Regards
>> Mahendra Singh Thalor
>> EnterpriseDB: http://www.enterprisedb.com
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v09_27112025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (89.1K, 2-v09_27112025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
From 71b9a213e7bb1b68e4d05b373516e0eca6337f38 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Thu, 27 Nov 2025 13:25:40 +0530
Subject: [PATCH] Non text modes for pg_dumpall, correspondingly change
pg_restore
pg_dumpall acquires a new -F/--format option, with the same meanings as
pg_dump. The default is p, meaning plain text. For any other value, a
directory is created containing two files, toc.glo and map.dat. The
first contains commands restoring the global data in custom format, and the second
contains a map from oids to database names in text format. It will also contain a
subdirectory called databases, inside which it will create archives in
the specified format, named using the database oids.
In these casess the -f argument is required.
If pg_restore encounters a directory containing map.dat and toc.glo,
it restores the global settings from toc.glo if exist, and then
restores each database.
pg_restore acquires two new options: -g/--globals-only which suppresses
restoration of any databases, and --exclude-database which inhibits
restoration of particualr database(s) in the same way the same option
works in pg_dumpall.
v09
---
doc/src/sgml/ref/pg_dumpall.sgml | 104 ++++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/connectdb.c | 7 +-
src/bin/pg_dump/connectdb.h | 2 +-
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 10 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 29 +-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 617 +++++++++++++++++++++------
src/bin/pg_dump/pg_restore.c | 605 +++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 27 ++
src/bin/pg_dump/t/007_pg_dumpall.pl | 396 +++++++++++++++++
15 files changed, 1705 insertions(+), 166 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
create mode 100755 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 8834b7ec141..75de1fee330 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,10 +139,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>toc.dat/toc.dmp/toc.tar</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
@@ -937,9 +1020,13 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
<title>Examples</title>
<para>
To dump all databases:
-
+ If format is given, then dump will be based on format, default plain.
<screen>
<prompt>$</prompt> <userinput>pg_dumpall > db.out</userinput>
+</screen>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dumpall --format=d/a/c/p -f db.out</userinput>
</screen>
</para>
@@ -956,6 +1043,15 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
the script will attempt to drop other databases immediately, and that
will fail for the database you are connected to.
</para>
+
+ <para>
+ If dump was taken in non-text format, then use pg_restore to restore all databases.
+<screen>
+<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
+</screen>
+ This will restore all the databases. If user don't want to restore some databases, then use
+ --exclude-pattern to skip those.
+</para>
</refsect1>
<refsect1>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index a468a38361a..7497b527ae6 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -591,6 +615,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
index d55d53dbeea..d3e9e27003e 100644
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -225,7 +225,7 @@ ConnectDatabase(const char *dbname, const char *connection_string,
exit_nicely(1);
}
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL, false));
return conn;
}
@@ -275,7 +275,7 @@ constructConnStr(const char **keywords, const char **values)
* Run a query, return the results, exit program on failure.
*/
PGresult *
-executeQuery(PGconn *conn, const char *query)
+executeQuery(PGconn *conn, const char *query, bool is_archive)
{
PGresult *res;
@@ -287,7 +287,8 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
+ if (!is_archive)
+ PQfinish(conn);
exit_nicely(1);
}
diff --git a/src/bin/pg_dump/connectdb.h b/src/bin/pg_dump/connectdb.h
index 6c1e1954769..0b741b68cb1 100644
--- a/src/bin/pg_dump/connectdb.h
+++ b/src/bin/pg_dump/connectdb.h
@@ -22,5 +22,5 @@ extern PGconn *ConnectDatabase(const char *dbname, const char *connection_string
trivalue prompt_password, bool fail_on_error,
const char *progname, const char **connstr, int *server_version,
char *password, char *override_dbname);
-extern PGresult *executeQuery(PGconn *conn, const char *query);
+extern PGresult *executeQuery(PGconn *conn, const char *query, bool is_archive);
#endif /* CONNECTDB_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index f3c669f484e..3e21aaf5780 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..5974d6706fd 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index d9041dad720..f631d945472 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -312,7 +312,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, bool globals_only);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index c84b017f21b..5b8dd295070 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, bool globals_only)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,15 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ /* Skip DROP DATABASE if globals_only. */
+ if (globals_only && te && te->tag && (strcmp(te->tag, "DROP_DATABASE") == 0))
+ continue;
+
+ /* Skip for CONNECT meta command. */
+ if (!ropt->filename && te && te->tag &&
+ (strcmp(te->tag, "CONNECT") == 0))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1330,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1695,7 +1709,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1730,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..818b80a9369 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a00918bacb4..13e1764ec70 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index bb451c1bae1..725365f6519 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -30,6 +30,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -65,9 +66,9 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, bool output_clean);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts, char *dbfile);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,11 +77,13 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
+static void createOneArchiveEntry(const char *query, const char *tag);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
static const char *connstr = "";
-static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
static bool dosync = true;
@@ -123,6 +126,10 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static ArchiveFormat archDumpFormat = archNull;
int
main(int argc, char *argv[])
@@ -148,6 +155,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +205,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *format_name = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -208,6 +217,7 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -246,7 +256,9 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ InitDumpOptions(&dopt);
+
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -256,7 +268,7 @@ main(int argc, char *argv[])
break;
case 'c':
- output_clean = true;
+ dopt.outputClean = true;
break;
case 'd':
@@ -274,7 +286,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ format_name = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -314,6 +328,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -419,7 +434,7 @@ main(int argc, char *argv[])
exit_nicely(1);
}
- if (if_exists && !output_clean)
+ if (if_exists && !dopt.outputClean)
pg_fatal("option --if-exists requires option -c/--clean");
if (roles_only && tablespaces_only)
@@ -429,6 +444,25 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(format_name);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
+ /* restrict-key is only supported with --format=plain */
+ if (archDumpFormat != archNull && restrict_key)
+ pg_fatal("option --restrict-key can only be used with --format=plain");
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -489,6 +523,27 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -538,19 +593,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -585,37 +627,110 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
- if (verbose)
+ if (verbose && archDumpFormat == archNull)
dumpTimestamp("Started on");
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ /* create a archive file for global commands. */
+ if (filename && archDumpFormat != archNull)
+ {
+ char global_path[MAXPGPATH];
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
+ /* Set file path for global sql commands. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", filename);
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+ /* Open the output file */
+ fout = CreateArchive(global_path, archCustom, compression_spec,
+ dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC);
+
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
+ ((ArchiveHandle*)fout)->connection = conn;
+ ((ArchiveHandle*)fout)->public.numWorkers = 1;
+
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
- /* Replicate encoding and std_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
- if (strcmp(std_strings, "off") == 0)
- fprintf(OPF, "SET escape_string_warning = off;\n");
- fprintf(OPF, "\n");
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump\n--\n\n", "COMMENT");
+
+ /* default_transaction_read_only = off */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving default_transaction_read_only = off");
+ appendPQExpBuffer(qry, "SET default_transaction_read_only = off;\n");
+ createOneArchiveEntry(qry->data, "DEFAULT_TRANSACTION_READ_ONLY");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpEncoding: put the correct encoding into the archive */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ const char *encname = pg_encoding_to_char(encoding);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+
+ pg_log_info("saving encoding = %s", encname);
+ createOneArchiveEntry(qry->data, "ENCODING");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpStdStrings: put the correct escape string behavior into the archive */
+ {
+ const char *stdstrings = std_strings ? "on" : "off";
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving \"standard_conforming_strings = %s\"", stdstrings);
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = '%s';\n",
+ stdstrings);
+ createOneArchiveEntry(qry->data, "STDSTRINGS");
+ destroyPQExpBuffer(qry);
+ }
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and std_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
+ if (strcmp(std_strings, "off") == 0)
+ fprintf(OPF, "SET escape_string_warning = off;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -625,7 +740,7 @@ main(int argc, char *argv[])
* and tablespaces never depend on each other. Roles could have
* grants to each other, but DROP ROLE will clean those up silently.
*/
- if (output_clean)
+ if (dopt.outputClean)
{
if (!globals_only && !roles_only && !tablespaces_only)
dropDBs(conn);
@@ -659,27 +774,42 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump will
+ * handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, dopt.outputClean);
- PQfinish(conn);
-
- if (verbose)
+ if (verbose && archDumpFormat == archNull)
dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (filename)
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+
+ if (archDumpFormat != archNull)
+ {
+ RestoreOptions *ropt;
+
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump complete\n--\n\n", "COMMENT");
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
+
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
+ }
+ else if (filename)
{
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -690,12 +820,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -770,6 +902,7 @@ static void
dropRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer delQry = createPQExpBuffer();
PGresult *res;
int i_rolname;
int i;
@@ -786,12 +919,17 @@ dropRoles(PGconn *conn)
"FROM %s "
"ORDER BY 1", role_catalog);
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
i_rolname = PQfnumber(res, "rolname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -799,15 +937,21 @@ dropRoles(PGconn *conn)
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(rolename));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropRoles");
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -871,7 +1015,7 @@ dumpRoles(PGconn *conn)
"FROM %s "
"ORDER BY 2", role_catalog);
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
i_oid = PQfnumber(res, "oid");
i_rolname = PQfnumber(res, "rolname");
@@ -889,7 +1033,12 @@ dumpRoles(PGconn *conn)
i_is_current_user = PQfnumber(res, "is_current_user");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -993,7 +1142,10 @@ dumpRoles(PGconn *conn)
"ROLE", rolename,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoles");
}
/*
@@ -1001,15 +1153,13 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
}
@@ -1076,7 +1226,7 @@ dumpRoleMembership(PGconn *conn)
"LEFT JOIN %s ug on ug.oid = a.grantor "
"WHERE NOT (ur.rolname ~ '^pg_' AND um.rolname ~ '^pg_')"
"ORDER BY 1,2,3", role_catalog, role_catalog, role_catalog);
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
i_role = PQfnumber(res, "role");
i_member = PQfnumber(res, "member");
i_grantor = PQfnumber(res, "grantor");
@@ -1088,7 +1238,12 @@ dumpRoleMembership(PGconn *conn)
i_set_option = PQfnumber(res, "set_option");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role memberships\n--\n\n", "COMMENT");
+ }
/*
* We can't dump these GRANT commands in arbitrary order, because a role
@@ -1167,6 +1322,7 @@ dumpRoleMembership(PGconn *conn)
char *grantor;
char *set_option = "true";
bool found;
+ PQExpBuffer creaQry = createPQExpBuffer();
/* If we already did this grant, don't do it again. */
if (done[i - start])
@@ -1223,8 +1379,8 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ appendPQExpBuffer(creaQry, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(creaQry, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1245,10 +1401,15 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(creaQry, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(creaQry, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(creaQry, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", creaQry->data);
+ else
+ createOneArchiveEntry(creaQry->data, "dumpRoleMembership");
}
}
@@ -1260,7 +1421,8 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1285,10 +1447,15 @@ dumpRoleGUCPrivs(PGconn *conn)
"paracl, "
"pg_catalog.acldefault('p', " CppAsString2(BOOTSTRAP_SUPERUSERID) ") AS acldefault "
"FROM pg_catalog.pg_parameter_acl "
- "ORDER BY 1");
+ "ORDER BY 1", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role privileges on configuration parameters\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1312,14 +1479,19 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoleGUCPrivs");
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1331,6 +1503,7 @@ dropTablespaces(PGconn *conn)
{
PGresult *res;
int i;
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Get all tablespaces except built-in ones (which we assume are named
@@ -1339,23 +1512,34 @@ dropTablespaces(PGconn *conn)
res = executeQuery(conn, "SELECT spcname "
"FROM pg_catalog.pg_tablespace "
"WHERE spcname !~ '^pg_' "
- "ORDER BY 1");
+ "ORDER BY 1", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(spcname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropTablespaces");
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1379,10 +1563,15 @@ dumpTablespaces(PGconn *conn)
"pg_catalog.shobj_description(oid, 'pg_tablespace') "
"FROM pg_catalog.pg_tablespace "
"WHERE spcname !~ '^pg_' "
- "ORDER BY 1");
+ "ORDER BY 1", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1451,14 +1640,19 @@ dumpTablespaces(PGconn *conn)
"TABLESPACE", spcname,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpTablespaces");
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1479,10 +1673,15 @@ dropDBs(PGconn *conn)
"SELECT datname "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
- "ORDER BY datname");
+ "ORDER BY datname", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop databases (except postgres and template1)\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1497,15 +1696,23 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(dbname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "DROP_DATABASE");
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1525,14 +1732,25 @@ dumpUserConfig(PGconn *conn, const char *username)
appendStringLiteralConn(buf, username, conn);
appendPQExpBufferChar(buf, ')');
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
if (PQntuples(res) > 0)
{
char *sanitized;
sanitized = sanitize_line(username, true);
- fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
}
@@ -1542,7 +1760,11 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpUserConfig");
}
PQclear(res);
@@ -1591,7 +1813,7 @@ expand_dbname_patterns(PGconn *conn,
exit_nicely(1);
}
- res = executeQuery(conn, query->data);
+ res = executeQuery(conn, query->data, fout ? true : false);
for (int i = 0; i < PQntuples(res); i++)
{
simple_string_list_append(names, PQgetvalue(res, i, 0));
@@ -1608,10 +1830,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, bool output_clean)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1625,19 +1850,49 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
- "ORDER BY (datname <> 'template1'), datname");
+ "ORDER BY (datname <> 'template1'), datname",
+ fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Databases\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Databases\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Databases\n--\n\n", "COMMENT");
+ }
+
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1654,7 +1909,18 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
/*
@@ -1669,24 +1935,46 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
+ fprintf(OPF, "\\connect %s\n\n", dbname);
else
{
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\\connect %s\n\n", dbname);
+ createOneArchiveEntry(qry->data, "CONNECT");
+ destroyPQExpBuffer(qry);
}
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1695,6 +1983,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1704,7 +1996,7 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1713,17 +2005,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1766,7 +2077,7 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
PGresult *res;
buildShSecLabelQuery(catalog_name, objectId, sql);
- res = executeQuery(conn, sql->data);
+ res = executeQuery(conn, sql->data, fout ? true : false);
emitShSecLabels(conn, res, buffer, objtype, objname);
PQclear(res);
@@ -1868,3 +2179,67 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+/*
+ * createDumpId
+ *
+ * This will return next last used oid.
+ */
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
+
+/*
+ * createOneArchiveEntry
+ *
+ * This creates one archive entry based on format.
+ */
+static void
+createOneArchiveEntry(const char *query, const char *tag)
+{
+ CatalogId nilCatalogId = {0, 0};
+ Assert(fout != NULL);
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = tag,
+ .section = SECTION_PRE_DATA,
+ .createStmt = query));
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c9776306c5c..18ea8869a97 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,12 +41,16 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -54,18 +58,43 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num,
+ bool globals_only);
+static int restore_global_objects(const char *inputFileSpec,
+ RestoreOptions *opts, int numWorkers,
+ int num, bool globals_only);
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +118,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +172,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +201,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +228,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global sql commands. */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -321,6 +355,10 @@ main(int argc, char **argv)
opts->restrict_key = pg_strdup(optarg);
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -347,6 +385,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -472,6 +517,121 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.glo file is present, then restore all the
+ * databases from map.dat, but skip restoring those matching
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "toc.glo")))
+ {
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring an archive created by pg_dumpall");
+ else if (opts->tocFile)
+ pg_fatal("option -L/--use-list cannot be used when restoring an archive created by pg_dumpall");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option -C/--create must be specified when restoring an archive created by pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ char global_path[MAXPGPATH];
+
+ /* Set path for toc.glo file. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+ n_errors = restore_global_objects(global_path, opts, numWorkers, 0, globals_only);
+
+ pg_log_info("database restoring skipped because option -g/--globals-only was specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else
+ {
+ if (db_exclude_patterns.head != NULL)
+ {
+ simple_string_list_destroy(&db_exclude_patterns);
+ pg_fatal("option --exclude-database can be used only when restoring an archive created by pg_dumpall");
+ }
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring an archive created by pg_dumpall");
+
+ /* Process if toc.glo file does not exist. */
+ n_errors = restore_one_database(inputFileSpec, opts,
+ numWorkers, false, 0, globals_only);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ *
+ * If globals_only is set, then skip DROP DATABASE commands from restore.
+ */
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, int num, bool globals_only)
+{
+ int nerror;
+ int format = opts->format;
+
+ /* Set format as custom so that toc.glo file can be read. */
+ opts->format = archCustom;
+
+ nerror = restore_one_database(inputFileSpec, opts, numWorkers,
+ false, num, globals_only);
+
+ /* Reset format value. */
+ opts->format = format;
+
+ return nerror;
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -479,9 +639,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -501,25 +667,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, globals_only);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -537,6 +699,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -553,6 +716,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -588,8 +752,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -694,3 +858,410 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn && db_exclude_patterns.head != NULL)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no database connection while doing pg_restore");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+ PQExpBuffer db_lit = createPQExpBuffer();
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data, false);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches exclude pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ destroyPQExpBuffer(db_lit);
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ destroyPQExpBuffer(query);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+
+
+ /*
+ * If there is no map.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen <= 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbname, db_oid, map_file_path);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+ PGconn *conn = NULL;
+ char global_path[MAXPGPATH];
+
+ /* Set path for toc.glo file. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
+
+ /* If map.dat has no entries, return after processing global commands. */
+ if (dbname_oid_list.head == NULL)
+ return restore_global_objects(global_path, opts, numWorkers, 0, false);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Open toc.dat file and execute/append all the global sql commands. */
+ n_errors_total = restore_global_objects(global_path, opts, numWorkers, 0, false);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ count++;
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..083f5c5bf9d
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -244,4 +244,31 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--restrict-key=uu', '-f dumpfile' ],
+ qr/\Qpg_dumpall: error: option --restrict-key can only be used with --format=plain\E/,
+ 'pg_dumpall: --restrict-key can only be used with plain dump format');
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option --exclude-database can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --exclude-database is used in pg_restore with dump of pg_dump'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--globals-only', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option -g\/--globals-only can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --globals-only is not used in pg_restore with dump of pg_dump'
+);
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100755
index 00000000000..3c7d2ad7c53
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,396 @@
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# Scape tablespace locations on Windows.
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT super TO grant7 WITH INHERIT TRUE GRANTED BY\E
+ (.*\n)*
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: When --list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: When non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+$node->stop('fast');
+
+done_testing();
--
2.47.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-11-27 09:19 Mahendra Singh Thalor <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-11-27 09:19 UTC (permalink / raw)
To: Vaibhav Dalvi <[email protected]>; +Cc: [email protected]
On Thu, 27 Nov 2025 at 13:45, Mahendra Singh Thalor <[email protected]> wrote:
>
> Thanks Vaibhav for the review.
>
> On Tue, 18 Nov 2025 at 16:05, Vaibhav Dalvi
> <[email protected]> wrote:
> >
> > Hi Mahendra,
> >
> > Thanks Mahendra for working on this.
> >
> > Looks like my previous comment below is not addressed:
> > 1.
> >
> >> ### Use of Dump Options Structure (dopt)
> >> Please ensure consistency by utilizing the main dump options
> >> structure (`dopt`) instead of declaring and using individual variables
> >> where the structure already provides fields. For example, the
> >> `output_clean` variable seems redundant here:
> >> ```c
> >> case 'c':
> >> output_clean = true;
> >> dopt.outputClean = 1;
> >> break;
> >> ```
> >
>
> Fixed. output_clean was a global variable because it was used in 2
> functions. Now I am passing dopt. output_clean as function argument
> for another function.
>
> >
> > I agree that the output_clean variable is not added by your patch
> > but the introduction of dopt by your patch makes it redundant because
> > dopt has dopt.outputClean. Please look at below code from pg_dump.c
> > for the reference:
> >
> > case 'c': /* clean (i.e., drop) schema prior to create */
> > dopt.outputClean = 1;
> > break;
> > case 25:
> > dopt.restrict_key = pg_strdup(optarg);
> > break;
> >
> > 2.
> >
> >> ### 3\. Missing Example in SGML Documentation
> >> The SGML documentation for `pg_dumpall` is missing an explicit
> >> example demonstrating its use with non-text formats (e.g., directory format).
> >> It would be beneficial to include a clear example for this new feature.
> >
> >
> > I think pg_dumpall should have separate examples similar to pg_dump
> > rather than referencing the pg_dump example because pg_dumpall
> > doesn't have to mention the database name without -l or --database
> > in the command.
> >
>
> Fixed. Added some examples.
>
> > 3.
> >>
> >> > 1. Is the following change in `src/bin/pg_dump/connectdb.c` intentional?
> >>
> >> >
> >> > ```
> >> > --- a/src/bin/pg_dump/connectdb.c
> >> > +++ b/src/bin/pg_dump/connectdb.c
> >> Yes, we need this. If there is any error, then we were trying to
> >> disconnect the database in 2 places so we were getting a crash. I will
> >> try to reproduce crashe without this patch and will respond.
> >
> > Have you added a test case in the regression suite which fails if we remove
> > this particular change and works well with the change? or if possible could
> > you please demonstrate here at least.
>
> Fixed. With AH(archive), we should not free pointers by this exec call
> as we free this by exit_nicely hook. (we register AH by
> on_exit_close_archive).
>
> >
> > 4. The variable name append_data doesn't look meaningful to me.
> > Instead we can use append_database/append_databases?
> > because if this variable is set then we dump the databases along with
> > global objects. In case of pg_dump, append_data or data_only does make
> > sense to differentiate between schema and data but in case of pg_dumpall
> > if this variable is set then we're dumping schema as well as data i.e. in-short
> > the databases.
> >
>
> As of now, I am keeping this append_data as this was from an already
> committed patch.
>
> > ------------------------------------ pg_dumpall.c ----------------------------------------
> >
> > 5. The variable name formatName doesn't follow the naming convention of
> > variables available around it. I think use of format_name/formatname would
> > be better.
> >
> >> char *use_role = NULL;
> >> const char *dumpencoding = NULL;
> >> + const char *formatName = "p";
> >> trivalue prompt_password = TRI_DEFAULT;
> >> bool data_only = false;
> >> bool globals_only = false;
> >
>
> Fixed.
>
> >
> > ------------------------------------ pg_restore.c ----------------------------------------
> >
> > 6. Fourth parameter (i.e. append_data) to function restore_global_objects() is redundant.
> > All the time value provided by all callers to this parameter is false.
> >
> > I would suggest removing this parameter and in the definition of this function
> > call function restore_one_database() with false as 4th argument. Find diff below:
> >
>
> Fixed.
>
> > --- a/src/bin/pg_dump/pg_restore.c
> > +++ b/src/bin/pg_dump/pg_restore.c
> > @@ -64,8 +64,7 @@ static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
> > int numWorkers, bool append_data, int num,
> > bool globals_only);
> > static int restore_global_objects(const char *inputFileSpec,
> > - RestoreOptions *opts, int numWorkers, bool append_data,
> > - int num, bool globals_only);
> > + RestoreOptions *opts, int numWorkers, int num, bool globals_only);
> > static int restore_all_databases(const char *inputFileSpec,
> > SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
> > static int get_dbnames_list_to_restore(PGconn *conn,
> > @@ -554,7 +553,7 @@ main(int argc, char **argv)
> >
> > /* Set path for toc.glo file. */
> > snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
> > - n_errors = restore_global_objects(global_path, opts, numWorkers, false, 0, globals_only);
> > + n_errors = restore_global_objects(global_path, opts, numWorkers, 0, globals_only);
> >
> > pg_log_info("database restoring skipped because option -g/--globals-only was specified");
> > }
> > @@ -602,7 +601,7 @@ main(int argc, char **argv)
> > * If globals_only is set, then skip DROP DATABASE commands from restore.
> > */
> > static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
> > - int numWorkers, bool append_data, int num, bool globals_only)
> > + int numWorkers, int num, bool globals_only)
> > {
> > int nerror;
> > int format = opts->format;
> > @@ -610,8 +609,8 @@ static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opt
> > /* Set format as custom so that toc.glo file can be read. */
> > opts->format = archCustom;
> >
> > - nerror = restore_one_database(inputFileSpec, opts, numWorkers,
> > - append_data, num, globals_only);
> > + nerror = restore_one_database(inputFileSpec, opts, numWorkers, false, num,
> > + globals_only);
> >
> > /* Reset format value. */
> > opts->format = format;
> > @@ -1097,7 +1096,7 @@ restore_all_databases(const char *inputFileSpec,
> >
> > /* If map.dat has no entries, return after processing global commands. */
> > if (dbname_oid_list.head == NULL)
> > - return restore_global_objects(global_path, opts, numWorkers, false, 0, false);
> > + return restore_global_objects(global_path, opts, numWorkers, 0, false);
> >
> > pg_log_info(ngettext("found %d database name in \"%s\"",
> > "found %d database names in \"%s\"",
> > @@ -1151,7 +1150,7 @@ restore_all_databases(const char *inputFileSpec,
> > PQfinish(conn);
> >
> > /* Open toc.dat file and execute/append all the global sql commands. */
> > - n_errors_total = restore_global_objects(global_path, opts, numWorkers, false, 0, false);
> > + n_errors_total = restore_global_objects(global_path, opts, numWorkers, 0, false);
> >
> > Regression is successful with these changes.
> >
> > 7. Fix indentation:
> >>
> >> static int restore_global_objects(const char *inputFileSpec,
> >> RestoreOptions *opts, int numWorkers, bool append_data,
> >> int num, bool globals_only);
> >> static int restore_all_databases(const char *inputFileSpec,
> >> SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
>
> Fixed some.
>
> >
> >
> > 8. Remove extra line:
> >>
> >> +
> >> static void usage(const char *progname);
> >
>
> Fixed.
>
> >
> > 9. Remove extra space after map.dat and before comma:
> >>
> >> + * databases from map.dat , but skip restoring those matching
> >
>
> Fixed.
>
> >
> > 10. Fix 80 char limits:
> >
> > + n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
> >
> > + num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
> >
> > + return restore_global_objects(global_path, opts, numWorkers, false, 0, false);
> >
> > + n_errors_total = restore_global_objects(global_path, opts, numWorkers, false, 0, false);
> >
> > + pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
> >
>
> Fixed some.
> I will do some more cleanup in the coming versions.
>
> Here, I am attaching an updated patch for the review and testing.
>
>
> >
> > Regards,
> > Vaibhav
> >
> > On Mon, Nov 17, 2025 at 10:45 PM Mahendra Singh Thalor <[email protected]> wrote:
> >>
> >> Thanks Andrew for the review.
> >> On Tue, 11 Nov 2025 at 20:41, Andrew Dunstan <[email protected]> wrote:
> >> >
> >> >
> >> > On 2025-11-11 Tu 12:59 AM, Mahendra Singh Thalor wrote:
> >> > >
> >> > > Hi,
> >> > > Here, I am attaching an updated patch for the review and testing.
> >> > >
> >> > > FIX: as suggested by Vaibhav, added error for --restrict-key option
> >> > > with non-text format.
> >> > >
> >> >
> >> >
> >> > Regarding the name and format of the globals toc file, I'm inclined to
> >> > think we should always use custom format, regardless of whether the
> >> > individual databases will be in custom, tar or directory formats, and
> >> > that it should be called something distinguishable, e.g. toc.glo.
> >> >
> >>
> >> I also agree with your point. Fixed.
> >>
> >> On Mon, 17 Nov 2025 at 19:38, tushar <[email protected]> wrote:
> >> >
> >> >
> >> >
> >> > On Tue, Nov 11, 2025 at 11:29 AM Mahendra Singh Thalor <[email protected]> wrote:
> >> >>
> >> >> On Thu, 6 Nov 2025 at 11:03, Mahendra Singh Thalor <[email protected]> wrote:
> >> >> >
> >> >> > Thanks Vaibhav, Tushar and Andrew for the review and testing.
> >> >>
> >> >
> >> > Thanks Mahendra, getting this error against v07 series patch
> >> >
> >> > [edb@1a1c15437e7c bin]$ ./pg_dumpall -Ft -f tar.dumpc -v
> >> > pg_dumpall: executing SELECT pg_catalog.set_config('search_path', '', false);
> >> > pg_dumpall: pg_dumpall.c:2256: createOneArchiveEntry: Assertion `fout != ((void *)0)' failed.
> >> > Aborted
> >> >
> >> > regards,
> >>
> >> Thanks Tushar for the report. Fixed.
> >>
> >> Here, I am attaching an updated patch for the review and testing.
> >>
> >> --
> >> Thanks and Regards
> >> Mahendra Singh Thalor
> >> EnterpriseDB: http://www.enterprisedb.com
>
>
> --
> Thanks and Regards
> Mahendra Singh Thalor
> EnterpriseDB: http://www.enterprisedb.com
Hi,
CI was reporting an error for an unused variable.
[08:37:07.338] user 0m14.312s
[08:37:07.338] sys 0m9.155s
[08:37:07.338] make -s -j${BUILD_JOBS} clean
[08:37:07.850] time make -s -j${BUILD_JOBS} world-bin
[08:37:17.443] pg_restore.c:1080:8: error: variable 'count' set but
not used [-Werror,-Wunused-but-set-variable]
[08:37:17.443] 1080 | int count = 0;
[08:37:17.443] | ^
[08:37:17.443] 1 error generated.
[08:37:17.443] make[3]: *** [<builtin>: pg_restore.o] Error 1
[08:37:17.443] make[3]: *** Waiting for unfinished jobs....
[08:37:17.708] make[2]: *** [Makefile:45: all-pg_dump-recurse] Error 2
[08:37:17.709] make[1]: *** [Makefile:42: all-bin-recurse] Error 2
[08:37:17.709] mak
Fixed. Here, I am attaching an updated patch for the review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v10_27112025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (89.1K, 2-v10_27112025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
From ab50bb52bf2a97c248decfd5e5b3ca64e9e27d9d Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Thu, 27 Nov 2025 14:43:16 +0530
Subject: [PATCH] Non text modes for pg_dumpall, correspondingly change
pg_restore
pg_dumpall acquires a new -F/--format option, with the same meanings as
pg_dump. The default is p, meaning plain text. For any other value, a
directory is created containing two files, toc.glo and map.dat. The
first contains commands restoring the global data in custom format, and the second
contains a map from oids to database names in text format. It will also contain a
subdirectory called databases, inside which it will create archives in
the specified format, named using the database oids.
In these casess the -f argument is required.
If pg_restore encounters a directory containing map.dat and toc.glo,
it restores the global settings from toc.glo if exist, and then
restores each database.
pg_restore acquires two new options: -g/--globals-only which suppresses
restoration of any databases, and --exclude-database which inhibits
restoration of particualr database(s) in the same way the same option
works in pg_dumpall.
v10
---
doc/src/sgml/ref/pg_dumpall.sgml | 104 ++++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/connectdb.c | 7 +-
src/bin/pg_dump/connectdb.h | 2 +-
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 10 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 29 +-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 617 +++++++++++++++++++++------
src/bin/pg_dump/pg_restore.c | 602 +++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 27 ++
src/bin/pg_dump/t/007_pg_dumpall.pl | 396 +++++++++++++++++
15 files changed, 1702 insertions(+), 166 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
create mode 100755 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 8834b7ec141..75de1fee330 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,10 +139,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>toc.dat/toc.dmp/toc.tar</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
@@ -937,9 +1020,13 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
<title>Examples</title>
<para>
To dump all databases:
-
+ If format is given, then dump will be based on format, default plain.
<screen>
<prompt>$</prompt> <userinput>pg_dumpall > db.out</userinput>
+</screen>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dumpall --format=d/a/c/p -f db.out</userinput>
</screen>
</para>
@@ -956,6 +1043,15 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
the script will attempt to drop other databases immediately, and that
will fail for the database you are connected to.
</para>
+
+ <para>
+ If dump was taken in non-text format, then use pg_restore to restore all databases.
+<screen>
+<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
+</screen>
+ This will restore all the databases. If user don't want to restore some databases, then use
+ --exclude-pattern to skip those.
+</para>
</refsect1>
<refsect1>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index a468a38361a..7497b527ae6 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -591,6 +615,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
index d55d53dbeea..d3e9e27003e 100644
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -225,7 +225,7 @@ ConnectDatabase(const char *dbname, const char *connection_string,
exit_nicely(1);
}
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL, false));
return conn;
}
@@ -275,7 +275,7 @@ constructConnStr(const char **keywords, const char **values)
* Run a query, return the results, exit program on failure.
*/
PGresult *
-executeQuery(PGconn *conn, const char *query)
+executeQuery(PGconn *conn, const char *query, bool is_archive)
{
PGresult *res;
@@ -287,7 +287,8 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
+ if (!is_archive)
+ PQfinish(conn);
exit_nicely(1);
}
diff --git a/src/bin/pg_dump/connectdb.h b/src/bin/pg_dump/connectdb.h
index 6c1e1954769..0b741b68cb1 100644
--- a/src/bin/pg_dump/connectdb.h
+++ b/src/bin/pg_dump/connectdb.h
@@ -22,5 +22,5 @@ extern PGconn *ConnectDatabase(const char *dbname, const char *connection_string
trivalue prompt_password, bool fail_on_error,
const char *progname, const char **connstr, int *server_version,
char *password, char *override_dbname);
-extern PGresult *executeQuery(PGconn *conn, const char *query);
+extern PGresult *executeQuery(PGconn *conn, const char *query, bool is_archive);
#endif /* CONNECTDB_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index f3c669f484e..3e21aaf5780 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..5974d6706fd 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index d9041dad720..f631d945472 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -312,7 +312,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, bool globals_only);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index c84b017f21b..5b8dd295070 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, bool globals_only)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,15 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ /* Skip DROP DATABASE if globals_only. */
+ if (globals_only && te && te->tag && (strcmp(te->tag, "DROP_DATABASE") == 0))
+ continue;
+
+ /* Skip for CONNECT meta command. */
+ if (!ropt->filename && te && te->tag &&
+ (strcmp(te->tag, "CONNECT") == 0))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1330,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1695,7 +1709,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1730,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..818b80a9369 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a00918bacb4..13e1764ec70 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index bb451c1bae1..725365f6519 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -30,6 +30,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -65,9 +66,9 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, bool output_clean);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts, char *dbfile);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,11 +77,13 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
+static void createOneArchiveEntry(const char *query, const char *tag);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
static const char *connstr = "";
-static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
static bool dosync = true;
@@ -123,6 +126,10 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static ArchiveFormat archDumpFormat = archNull;
int
main(int argc, char *argv[])
@@ -148,6 +155,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +205,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *format_name = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -208,6 +217,7 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -246,7 +256,9 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ InitDumpOptions(&dopt);
+
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -256,7 +268,7 @@ main(int argc, char *argv[])
break;
case 'c':
- output_clean = true;
+ dopt.outputClean = true;
break;
case 'd':
@@ -274,7 +286,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ format_name = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -314,6 +328,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -419,7 +434,7 @@ main(int argc, char *argv[])
exit_nicely(1);
}
- if (if_exists && !output_clean)
+ if (if_exists && !dopt.outputClean)
pg_fatal("option --if-exists requires option -c/--clean");
if (roles_only && tablespaces_only)
@@ -429,6 +444,25 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(format_name);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
+ /* restrict-key is only supported with --format=plain */
+ if (archDumpFormat != archNull && restrict_key)
+ pg_fatal("option --restrict-key can only be used with --format=plain");
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -489,6 +523,27 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -538,19 +593,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -585,37 +627,110 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
- if (verbose)
+ if (verbose && archDumpFormat == archNull)
dumpTimestamp("Started on");
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ /* create a archive file for global commands. */
+ if (filename && archDumpFormat != archNull)
+ {
+ char global_path[MAXPGPATH];
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
+ /* Set file path for global sql commands. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", filename);
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+ /* Open the output file */
+ fout = CreateArchive(global_path, archCustom, compression_spec,
+ dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC);
+
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
+ ((ArchiveHandle*)fout)->connection = conn;
+ ((ArchiveHandle*)fout)->public.numWorkers = 1;
+
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
- /* Replicate encoding and std_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
- if (strcmp(std_strings, "off") == 0)
- fprintf(OPF, "SET escape_string_warning = off;\n");
- fprintf(OPF, "\n");
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump\n--\n\n", "COMMENT");
+
+ /* default_transaction_read_only = off */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving default_transaction_read_only = off");
+ appendPQExpBuffer(qry, "SET default_transaction_read_only = off;\n");
+ createOneArchiveEntry(qry->data, "DEFAULT_TRANSACTION_READ_ONLY");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpEncoding: put the correct encoding into the archive */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ const char *encname = pg_encoding_to_char(encoding);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+
+ pg_log_info("saving encoding = %s", encname);
+ createOneArchiveEntry(qry->data, "ENCODING");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpStdStrings: put the correct escape string behavior into the archive */
+ {
+ const char *stdstrings = std_strings ? "on" : "off";
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving \"standard_conforming_strings = %s\"", stdstrings);
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = '%s';\n",
+ stdstrings);
+ createOneArchiveEntry(qry->data, "STDSTRINGS");
+ destroyPQExpBuffer(qry);
+ }
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and std_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
+ if (strcmp(std_strings, "off") == 0)
+ fprintf(OPF, "SET escape_string_warning = off;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -625,7 +740,7 @@ main(int argc, char *argv[])
* and tablespaces never depend on each other. Roles could have
* grants to each other, but DROP ROLE will clean those up silently.
*/
- if (output_clean)
+ if (dopt.outputClean)
{
if (!globals_only && !roles_only && !tablespaces_only)
dropDBs(conn);
@@ -659,27 +774,42 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump will
+ * handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, dopt.outputClean);
- PQfinish(conn);
-
- if (verbose)
+ if (verbose && archDumpFormat == archNull)
dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (filename)
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+
+ if (archDumpFormat != archNull)
+ {
+ RestoreOptions *ropt;
+
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump complete\n--\n\n", "COMMENT");
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
+
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
+ }
+ else if (filename)
{
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -690,12 +820,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -770,6 +902,7 @@ static void
dropRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer delQry = createPQExpBuffer();
PGresult *res;
int i_rolname;
int i;
@@ -786,12 +919,17 @@ dropRoles(PGconn *conn)
"FROM %s "
"ORDER BY 1", role_catalog);
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
i_rolname = PQfnumber(res, "rolname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -799,15 +937,21 @@ dropRoles(PGconn *conn)
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(rolename));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropRoles");
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -871,7 +1015,7 @@ dumpRoles(PGconn *conn)
"FROM %s "
"ORDER BY 2", role_catalog);
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
i_oid = PQfnumber(res, "oid");
i_rolname = PQfnumber(res, "rolname");
@@ -889,7 +1033,12 @@ dumpRoles(PGconn *conn)
i_is_current_user = PQfnumber(res, "is_current_user");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -993,7 +1142,10 @@ dumpRoles(PGconn *conn)
"ROLE", rolename,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoles");
}
/*
@@ -1001,15 +1153,13 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
}
@@ -1076,7 +1226,7 @@ dumpRoleMembership(PGconn *conn)
"LEFT JOIN %s ug on ug.oid = a.grantor "
"WHERE NOT (ur.rolname ~ '^pg_' AND um.rolname ~ '^pg_')"
"ORDER BY 1,2,3", role_catalog, role_catalog, role_catalog);
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
i_role = PQfnumber(res, "role");
i_member = PQfnumber(res, "member");
i_grantor = PQfnumber(res, "grantor");
@@ -1088,7 +1238,12 @@ dumpRoleMembership(PGconn *conn)
i_set_option = PQfnumber(res, "set_option");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role memberships\n--\n\n", "COMMENT");
+ }
/*
* We can't dump these GRANT commands in arbitrary order, because a role
@@ -1167,6 +1322,7 @@ dumpRoleMembership(PGconn *conn)
char *grantor;
char *set_option = "true";
bool found;
+ PQExpBuffer creaQry = createPQExpBuffer();
/* If we already did this grant, don't do it again. */
if (done[i - start])
@@ -1223,8 +1379,8 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ appendPQExpBuffer(creaQry, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(creaQry, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1245,10 +1401,15 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(creaQry, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(creaQry, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(creaQry, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", creaQry->data);
+ else
+ createOneArchiveEntry(creaQry->data, "dumpRoleMembership");
}
}
@@ -1260,7 +1421,8 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1285,10 +1447,15 @@ dumpRoleGUCPrivs(PGconn *conn)
"paracl, "
"pg_catalog.acldefault('p', " CppAsString2(BOOTSTRAP_SUPERUSERID) ") AS acldefault "
"FROM pg_catalog.pg_parameter_acl "
- "ORDER BY 1");
+ "ORDER BY 1", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role privileges on configuration parameters\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1312,14 +1479,19 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoleGUCPrivs");
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1331,6 +1503,7 @@ dropTablespaces(PGconn *conn)
{
PGresult *res;
int i;
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Get all tablespaces except built-in ones (which we assume are named
@@ -1339,23 +1512,34 @@ dropTablespaces(PGconn *conn)
res = executeQuery(conn, "SELECT spcname "
"FROM pg_catalog.pg_tablespace "
"WHERE spcname !~ '^pg_' "
- "ORDER BY 1");
+ "ORDER BY 1", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(spcname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropTablespaces");
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1379,10 +1563,15 @@ dumpTablespaces(PGconn *conn)
"pg_catalog.shobj_description(oid, 'pg_tablespace') "
"FROM pg_catalog.pg_tablespace "
"WHERE spcname !~ '^pg_' "
- "ORDER BY 1");
+ "ORDER BY 1", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1451,14 +1640,19 @@ dumpTablespaces(PGconn *conn)
"TABLESPACE", spcname,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpTablespaces");
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1479,10 +1673,15 @@ dropDBs(PGconn *conn)
"SELECT datname "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
- "ORDER BY datname");
+ "ORDER BY datname", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop databases (except postgres and template1)\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1497,15 +1696,23 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(dbname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "DROP_DATABASE");
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1525,14 +1732,25 @@ dumpUserConfig(PGconn *conn, const char *username)
appendStringLiteralConn(buf, username, conn);
appendPQExpBufferChar(buf, ')');
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
if (PQntuples(res) > 0)
{
char *sanitized;
sanitized = sanitize_line(username, true);
- fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
}
@@ -1542,7 +1760,11 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpUserConfig");
}
PQclear(res);
@@ -1591,7 +1813,7 @@ expand_dbname_patterns(PGconn *conn,
exit_nicely(1);
}
- res = executeQuery(conn, query->data);
+ res = executeQuery(conn, query->data, fout ? true : false);
for (int i = 0; i < PQntuples(res); i++)
{
simple_string_list_append(names, PQgetvalue(res, i, 0));
@@ -1608,10 +1830,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, bool output_clean)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1625,19 +1850,49 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
- "ORDER BY (datname <> 'template1'), datname");
+ "ORDER BY (datname <> 'template1'), datname",
+ fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Databases\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Databases\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Databases\n--\n\n", "COMMENT");
+ }
+
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1654,7 +1909,18 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
/*
@@ -1669,24 +1935,46 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
+ fprintf(OPF, "\\connect %s\n\n", dbname);
else
{
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\\connect %s\n\n", dbname);
+ createOneArchiveEntry(qry->data, "CONNECT");
+ destroyPQExpBuffer(qry);
}
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1695,6 +1983,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1704,7 +1996,7 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1713,17 +2005,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1766,7 +2077,7 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
PGresult *res;
buildShSecLabelQuery(catalog_name, objectId, sql);
- res = executeQuery(conn, sql->data);
+ res = executeQuery(conn, sql->data, fout ? true : false);
emitShSecLabels(conn, res, buffer, objtype, objname);
PQclear(res);
@@ -1868,3 +2179,67 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+/*
+ * createDumpId
+ *
+ * This will return next last used oid.
+ */
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
+
+/*
+ * createOneArchiveEntry
+ *
+ * This creates one archive entry based on format.
+ */
+static void
+createOneArchiveEntry(const char *query, const char *tag)
+{
+ CatalogId nilCatalogId = {0, 0};
+ Assert(fout != NULL);
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = tag,
+ .section = SECTION_PRE_DATA,
+ .createStmt = query));
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c9776306c5c..9ef84e5a9ec 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,12 +41,16 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -54,18 +58,43 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num,
+ bool globals_only);
+static int restore_global_objects(const char *inputFileSpec,
+ RestoreOptions *opts, int numWorkers,
+ int num, bool globals_only);
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +118,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +172,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +201,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +228,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global sql commands. */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -321,6 +355,10 @@ main(int argc, char **argv)
opts->restrict_key = pg_strdup(optarg);
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -347,6 +385,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -472,6 +517,121 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.glo file is present, then restore all the
+ * databases from map.dat, but skip restoring those matching
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "toc.glo")))
+ {
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring an archive created by pg_dumpall");
+ else if (opts->tocFile)
+ pg_fatal("option -L/--use-list cannot be used when restoring an archive created by pg_dumpall");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option -C/--create must be specified when restoring an archive created by pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ char global_path[MAXPGPATH];
+
+ /* Set path for toc.glo file. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+ n_errors = restore_global_objects(global_path, opts, numWorkers, 0, globals_only);
+
+ pg_log_info("database restoring skipped because option -g/--globals-only was specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else
+ {
+ if (db_exclude_patterns.head != NULL)
+ {
+ simple_string_list_destroy(&db_exclude_patterns);
+ pg_fatal("option --exclude-database can be used only when restoring an archive created by pg_dumpall");
+ }
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring an archive created by pg_dumpall");
+
+ /* Process if toc.glo file does not exist. */
+ n_errors = restore_one_database(inputFileSpec, opts,
+ numWorkers, false, 0, globals_only);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ *
+ * If globals_only is set, then skip DROP DATABASE commands from restore.
+ */
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, int num, bool globals_only)
+{
+ int nerror;
+ int format = opts->format;
+
+ /* Set format as custom so that toc.glo file can be read. */
+ opts->format = archCustom;
+
+ nerror = restore_one_database(inputFileSpec, opts, numWorkers,
+ false, num, globals_only);
+
+ /* Reset format value. */
+ opts->format = format;
+
+ return nerror;
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -479,9 +639,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -501,25 +667,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, globals_only);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -537,6 +699,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -553,6 +716,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -588,8 +752,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -694,3 +858,407 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn && db_exclude_patterns.head != NULL)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no database connection while doing pg_restore");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+ PQExpBuffer db_lit = createPQExpBuffer();
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data, false);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches exclude pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ destroyPQExpBuffer(db_lit);
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ destroyPQExpBuffer(query);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+
+
+ /*
+ * If there is no map.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen <= 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbname, db_oid, map_file_path);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+ PGconn *conn = NULL;
+ char global_path[MAXPGPATH];
+
+ /* Set path for toc.glo file. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
+
+ /* If map.dat has no entries, return after processing global commands. */
+ if (dbname_oid_list.head == NULL)
+ return restore_global_objects(global_path, opts, numWorkers, 0, false);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Open toc.dat file and execute/append all the global sql commands. */
+ n_errors_total = restore_global_objects(global_path, opts, numWorkers, 0, false);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..083f5c5bf9d
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -244,4 +244,31 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--restrict-key=uu', '-f dumpfile' ],
+ qr/\Qpg_dumpall: error: option --restrict-key can only be used with --format=plain\E/,
+ 'pg_dumpall: --restrict-key can only be used with plain dump format');
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option --exclude-database can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --exclude-database is used in pg_restore with dump of pg_dump'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--globals-only', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option -g\/--globals-only can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --globals-only is not used in pg_restore with dump of pg_dump'
+);
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100755
index 00000000000..3c7d2ad7c53
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,396 @@
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# Scape tablespace locations on Windows.
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT super TO grant7 WITH INHERIT TRUE GRANTED BY\E
+ (.*\n)*
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: When --list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: When non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+$node->stop('fast');
+
+done_testing();
--
2.47.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-12-01 13:06 tushar <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: tushar @ 2025-12-01 13:06 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
On Thu, Nov 27, 2025 at 2:49 PM Mahendra Singh Thalor <[email protected]>
wrote:
>
> Fixed. Here, I am attaching an updated patch for the review and testing.
>
Thanks Mahendra, please refer this scenario where restoring the
dump(database contain tablespace) throwing an error
*Steps to reproduce *
initdb (./initdb -D data) , start the server ( ./pg_ctl -D data start) ,
connect to psql terminal ( ./psql postgres)
create a directory ( \! mkdir /tmp/abc) , create a tablespace ( create
tablespace a location '/tmp/abc'); )
create a table ( create table t(n int) tablespace a; ) , insert data (
insert into t values ('a'); )
perform pg_dumpall with option -c ( ./pg_dumpall -Fc -f my.d)
try to perform pg_restore with option --no-tablespaces ( ./pg_restore
--no-tablespaces -Fc my.d -d postgres -C)
Getting this error :
"
pg_restore: error: could not execute query: ERROR: role "edb" already
exists
Command was: CREATE ROLE edb;
ALTER ROLE edb WITH SUPERUSER INHERIT CREATEROLE CREATEDB LOGIN REPLICATION
BYPASSRLS;
pg_restore: error: could not execute query: ERROR: directory
"/tmp/abc/PG_19_202511281" already in use as a tablespace
Command was: CREATE TABLESPACE a OWNER edb LOCATION '/tmp/abc';
pg_restore: warning: errors ignored on restore: 2
"
regards,
Tushar
https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-12-01 17:17 tushar <[email protected]>
parent: tushar <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: tushar @ 2025-12-01 17:17 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
On Mon, Dec 1, 2025 at 6:36 PM tushar <[email protected]> wrote:
>
>
> On Thu, Nov 27, 2025 at 2:49 PM Mahendra Singh Thalor <[email protected]>
> wrote:
>
>>
>> Fixed. Here, I am attaching an updated patch for the review and testing.
>>
>
> Thanks Mahendra, please refer this scenario where restoring the
> dump(database contain tablespace) throwing an error
>
> *Steps to reproduce *
> initdb (./initdb -D data) , start the server ( ./pg_ctl -D data start) ,
> connect to psql terminal ( ./psql postgres)
> create a directory ( \! mkdir /tmp/abc) , create a tablespace ( create
> tablespace a location '/tmp/abc'); )
> create a table ( create table t(n int) tablespace a; ) , insert data (
> insert into t values ('a'); )
> perform pg_dumpall with option -c ( ./pg_dumpall -Fc -f my.d)
> try to perform pg_restore with option --no-tablespaces ( ./pg_restore
> --no-tablespaces -Fc my.d -d postgres -C)
> Getting this error :
> "
> pg_restore: error: could not execute query: ERROR: role "edb" already
> exists
> Command was: CREATE ROLE edb;
> ALTER ROLE edb WITH SUPERUSER INHERIT CREATEROLE CREATEDB LOGIN
> REPLICATION BYPASSRLS;
>
> pg_restore: error: could not execute query: ERROR: directory
> "/tmp/abc/PG_19_202511281" already in use as a tablespace
> Command was: CREATE TABLESPACE a OWNER edb LOCATION '/tmp/abc';
>
> pg_restore: warning: errors ignored on restore: 2
> "
>
>
I have observed that when combining the --globals-only option with certain
other switches during a pg_restore - operation fails silently.
The attempted restore does not execute, but no error message or warning is
displayed unless the --verbose option is also used.
--this will just run without any message but objects also not going to
create
./pg_restore -Fc ok31. -C -d postgres -t mytable --globals-only
./pg_restore -Fc ok31. -C -d postgres -no-tablespace --globals-only
./pg_restore -Fc ok31. -C -d postgres -no-data --globals-only
with --verbose
[edb@1a1c15437e7c bin]$ ./pg_restore -Fc ok31. -C -d postgres -t myable
--globals-only -v
pg_restore: connecting to database for restore
pg_restore: executing SELECT pg_catalog.set_config('search_path', '',
false);
pg_restore: implied no-schema restore
pg_restore: database restoring skipped because option -g/--globals-only was
specified
we should probably add some message there.
regards,
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-12-02 13:15 tushar <[email protected]>
parent: tushar <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: tushar @ 2025-12-02 13:15 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
On Mon, Dec 1, 2025 at 10:47 PM tushar <[email protected]>
wrote:
>
>>
> I have observed that when combining the --globals-only option with certain
> other switches during a pg_restore - operation fails silently.
> The attempted restore does not execute, but no error message or warning is
> displayed unless the --verbose option is also used.
>
> --this will just run without any message but objects also not going to
> create
> ./pg_restore -Fc ok31. -C -d postgres -t mytable --globals-only
> ./pg_restore -Fc ok31. -C -d postgres -no-tablespace --globals-only
> ./pg_restore -Fc ok31. -C -d postgres -no-data --globals-only
>
> with --verbose
> [edb@1a1c15437e7c bin]$ ./pg_restore -Fc ok31. -C -d postgres -t myable
> --globals-only -v
> pg_restore: connecting to database for restore
> pg_restore: executing SELECT pg_catalog.set_config('search_path', '',
> false);
> pg_restore: implied no-schema restore
> pg_restore: database restoring skipped because option -g/--globals-only
> was specified
>
> we should probably add some message there.
>
>
Please refer this scenario where "--no-comments" switch is ignoring when
used with -Ft/c option of pg_dumpall
*Test Case to reproduce:*
--Connect to psql terminal , create a table and comment :
postgres=# create table t(n int);
CREATE TABLE
postgres=# insert into t values (1);
INSERT 0 1
postgres=# comment on table t is 'testing...';
COMMENT
postgres=# SELECT obj_description('public.t'::regclass, 'pg_class') AS
table_comment ;
table_comment
---------------
testing...
(1 row)
--perform pg_dumpall with
(a) -Fp (./pg_dumpall -Fp --no-comments -f dump.plain)
(b) -Ft (./pg_dumpall -Ft --no-comments -f dump.tar)
Case 1: restore (a) , just run the file (dump.plain) on psql terminal ,
fire this query :
postgres=# SELECT
obj_description('public.t'::regclass, 'pg_class') AS table_comment;
table_comment
---------------
(1 row)
Seems expected .
Case 2: restore (b) via command ( ./pg_restore -Ft dump.tar -d postgres -p
5806 -C )
fire this query :
postgres=# SELECT obj_description('public.t'::regclass, 'pg_class') AS
table_comment ;
table_comment
---------------
testing...
(1 row)
Seems not expected i.e pg_dumpall with option -Ft still taking table
comments and ignoring --no-comments switch.
regards,
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-12-08 06:44 Mahendra Singh Thalor <[email protected]>
parent: tushar <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-12-08 06:44 UTC (permalink / raw)
To: tushar <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
Thanks Tushar for the testing and reports.
On Tue, 2 Dec 2025 at 18:45, tushar <[email protected]> wrote:
>
>
>
> On Mon, Dec 1, 2025 at 10:47 PM tushar <[email protected]> wrote:
>>>
>>>
>>
>> I have observed that when combining the --globals-only option with certain other switches during a pg_restore - operation fails silently.
>> The attempted restore does not execute, but no error message or warning is displayed unless the --verbose option is also used.
>>
>> --this will just run without any message but objects also not going to create
>> ./pg_restore -Fc ok31. -C -d postgres -t mytable --globals-only
>> ./pg_restore -Fc ok31. -C -d postgres -no-tablespace --globals-only
>> ./pg_restore -Fc ok31. -C -d postgres -no-data --globals-only
>>
>> with --verbose
>> [edb@1a1c15437e7c bin]$ ./pg_restore -Fc ok31. -C -d postgres -t myable --globals-only -v
>> pg_restore: connecting to database for restore
>> pg_restore: executing SELECT pg_catalog.set_config('search_path', '', false);
>> pg_restore: implied no-schema restore
>> pg_restore: database restoring skipped because option -g/--globals-only was specified
>>
>> we should probably add some message there.
>>
>
> Please refer this scenario where "--no-comments" switch is ignoring when used with -Ft/c option of pg_dumpall
>
> Test Case to reproduce:
> --Connect to psql terminal , create a table and comment :
> postgres=# create table t(n int);
> CREATE TABLE
> postgres=# insert into t values (1);
> INSERT 0 1
> postgres=# comment on table t is 'testing...';
> COMMENT
> postgres=# SELECT obj_description('public.t'::regclass, 'pg_class') AS table_comment ;
> table_comment
> ---------------
> testing...
> (1 row)
>
> --perform pg_dumpall with
> (a) -Fp (./pg_dumpall -Fp --no-comments -f dump.plain)
> (b) -Ft (./pg_dumpall -Ft --no-comments -f dump.tar)
>
> Case 1: restore (a) , just run the file (dump.plain) on psql terminal , fire this query :
> postgres=# SELECT
> obj_description('public.t'::regclass, 'pg_class') AS table_comment;
> table_comment
> ---------------
>
> (1 row)
> Seems expected .
>
> Case 2: restore (b) via command ( ./pg_restore -Ft dump.tar -d postgres -p 5806 -C )
> fire this query :
> postgres=# SELECT obj_description('public.t'::regclass, 'pg_class') AS table_comment ;
> table_comment
> ---------------
> testing...
> (1 row)
>
> Seems not expected i.e pg_dumpall with option -Ft still taking table comments and ignoring --no-comments switch.
>
> regards,
I tried to fix these issues in the attached patch.
Here, I am attaching an updated patch for the review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v11_08122025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (89.8K, 2-v11_08122025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
From a989b60741926a089a0c2fc372cb2ef007310a96 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Mon, 8 Dec 2025 12:06:28 +0530
Subject: [PATCH] Non text modes for pg_dumpall, correspondingly change
pg_restore
pg_dumpall acquires a new -F/--format option, with the same meanings as
pg_dump. The default is p, meaning plain text. For any other value, a
directory is created containing two files, toc.glo and map.dat. The
first contains commands restoring the global data in custom format, and the second
contains a map from oids to database names in text format. It will also contain a
subdirectory called databases, inside which it will create archives in
the specified format, named using the database oids.
In these casess the -f argument is required.
If pg_restore encounters a directory containing map.dat and toc.glo,
it restores the global settings from toc.glo if exist, and then
restores each database.
pg_restore acquires two new options: -g/--globals-only which suppresses
restoration of any databases, and --exclude-database which inhibits
restoration of particualr database(s) in the same way the same option
works in pg_dumpall.
v11
---
doc/src/sgml/ref/pg_dumpall.sgml | 104 ++++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/connectdb.c | 7 +-
src/bin/pg_dump/connectdb.h | 2 +-
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 10 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 34 +-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 617 +++++++++++++++++++++------
src/bin/pg_dump/pg_restore.c | 609 +++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 27 ++
src/bin/pg_dump/t/007_pg_dumpall.pl | 396 +++++++++++++++++
15 files changed, 1713 insertions(+), 167 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
create mode 100755 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 8834b7ec141..75de1fee330 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,10 +139,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>toc.dat/toc.dmp/toc.tar</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
@@ -937,9 +1020,13 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
<title>Examples</title>
<para>
To dump all databases:
-
+ If format is given, then dump will be based on format, default plain.
<screen>
<prompt>$</prompt> <userinput>pg_dumpall > db.out</userinput>
+</screen>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dumpall --format=d/a/c/p -f db.out</userinput>
</screen>
</para>
@@ -956,6 +1043,15 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
the script will attempt to drop other databases immediately, and that
will fail for the database you are connected to.
</para>
+
+ <para>
+ If dump was taken in non-text format, then use pg_restore to restore all databases.
+<screen>
+<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
+</screen>
+ This will restore all the databases. If user don't want to restore some databases, then use
+ --exclude-pattern to skip those.
+</para>
</refsect1>
<refsect1>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index a468a38361a..7497b527ae6 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -591,6 +615,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
index d55d53dbeea..d3e9e27003e 100644
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -225,7 +225,7 @@ ConnectDatabase(const char *dbname, const char *connection_string,
exit_nicely(1);
}
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL, false));
return conn;
}
@@ -275,7 +275,7 @@ constructConnStr(const char **keywords, const char **values)
* Run a query, return the results, exit program on failure.
*/
PGresult *
-executeQuery(PGconn *conn, const char *query)
+executeQuery(PGconn *conn, const char *query, bool is_archive)
{
PGresult *res;
@@ -287,7 +287,8 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
+ if (!is_archive)
+ PQfinish(conn);
exit_nicely(1);
}
diff --git a/src/bin/pg_dump/connectdb.h b/src/bin/pg_dump/connectdb.h
index 6c1e1954769..0b741b68cb1 100644
--- a/src/bin/pg_dump/connectdb.h
+++ b/src/bin/pg_dump/connectdb.h
@@ -22,5 +22,5 @@ extern PGconn *ConnectDatabase(const char *dbname, const char *connection_string
trivalue prompt_password, bool fail_on_error,
const char *progname, const char **connstr, int *server_version,
char *password, char *override_dbname);
-extern PGresult *executeQuery(PGconn *conn, const char *query);
+extern PGresult *executeQuery(PGconn *conn, const char *query, bool is_archive);
#endif /* CONNECTDB_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index f3c669f484e..3e21aaf5780 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..5974d6706fd 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index d9041dad720..f631d945472 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -312,7 +312,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, bool globals_only);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index c84b017f21b..d35232cd038 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, bool globals_only)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,20 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ /* Skip DROP DATABASE if globals_only. */
+ if (globals_only && te && te->tag && (strcmp(te->tag, "DROP_DATABASE") == 0))
+ continue;
+
+ /* Skip for CONNECT meta command. */
+ if (!ropt->filename && te && te->tag &&
+ (strcmp(te->tag, "CONNECT") == 0))
+ continue;
+
+ /* Skip if no-tablespace is given. */
+ if (ropt->noTablespace && te && te->tag && ((strcmp(te->tag, "dumpTablespaces") == 0) ||
+ (strcmp(te->tag, "dropTablespaces") == 0)))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1335,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1695,7 +1714,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1735,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..818b80a9369 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 2445085dbbd..e1a1711254d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index bb451c1bae1..01e3683c84b 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -30,6 +30,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -65,9 +66,9 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, bool output_clean);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts, char *dbfile);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,11 +77,13 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
+static void createOneArchiveEntry(const char *query, const char *tag);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
static const char *connstr = "";
-static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
static bool dosync = true;
@@ -123,6 +126,10 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static ArchiveFormat archDumpFormat = archNull;
int
main(int argc, char *argv[])
@@ -148,6 +155,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +205,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *format_name = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -208,6 +217,7 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -246,7 +256,9 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ InitDumpOptions(&dopt);
+
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -256,7 +268,7 @@ main(int argc, char *argv[])
break;
case 'c':
- output_clean = true;
+ dopt.outputClean = true;
break;
case 'd':
@@ -274,7 +286,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ format_name = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -314,6 +328,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -419,7 +434,7 @@ main(int argc, char *argv[])
exit_nicely(1);
}
- if (if_exists && !output_clean)
+ if (if_exists && !dopt.outputClean)
pg_fatal("option --if-exists requires option -c/--clean");
if (roles_only && tablespaces_only)
@@ -429,6 +444,25 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(format_name);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
+ /* restrict-key is only supported with --format=plain */
+ if (archDumpFormat != archNull && restrict_key)
+ pg_fatal("option --restrict-key can only be used with --format=plain");
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -489,6 +523,27 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -538,19 +593,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -585,37 +627,110 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
- if (verbose)
+ if (verbose && archDumpFormat == archNull)
dumpTimestamp("Started on");
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ /* create a archive file for global commands. */
+ if (filename && archDumpFormat != archNull)
+ {
+ char global_path[MAXPGPATH];
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
+ /* Set file path for global sql commands. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", filename);
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+ /* Open the output file */
+ fout = CreateArchive(global_path, archCustom, compression_spec,
+ dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC);
+
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
+ ((ArchiveHandle*)fout)->connection = conn;
+ ((ArchiveHandle*)fout)->public.numWorkers = 1;
+
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
- /* Replicate encoding and std_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
- if (strcmp(std_strings, "off") == 0)
- fprintf(OPF, "SET escape_string_warning = off;\n");
- fprintf(OPF, "\n");
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump\n--\n\n", "COMMENT");
+
+ /* default_transaction_read_only = off */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving default_transaction_read_only = off");
+ appendPQExpBuffer(qry, "SET default_transaction_read_only = off;\n");
+ createOneArchiveEntry(qry->data, "DEFAULT_TRANSACTION_READ_ONLY");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpEncoding: put the correct encoding into the archive */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ const char *encname = pg_encoding_to_char(encoding);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+
+ pg_log_info("saving encoding = %s", encname);
+ createOneArchiveEntry(qry->data, "ENCODING");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpStdStrings: put the correct escape string behavior into the archive */
+ {
+ const char *stdstrings = std_strings ? "on" : "off";
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving \"standard_conforming_strings = %s\"", stdstrings);
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = '%s';\n",
+ stdstrings);
+ createOneArchiveEntry(qry->data, "STDSTRINGS");
+ destroyPQExpBuffer(qry);
+ }
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and std_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
+ if (strcmp(std_strings, "off") == 0)
+ fprintf(OPF, "SET escape_string_warning = off;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -625,7 +740,7 @@ main(int argc, char *argv[])
* and tablespaces never depend on each other. Roles could have
* grants to each other, but DROP ROLE will clean those up silently.
*/
- if (output_clean)
+ if (dopt.outputClean)
{
if (!globals_only && !roles_only && !tablespaces_only)
dropDBs(conn);
@@ -659,27 +774,42 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump will
+ * handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, dopt.outputClean);
- PQfinish(conn);
-
- if (verbose)
+ if (verbose && archDumpFormat == archNull)
dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (filename)
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+
+ if (archDumpFormat != archNull)
+ {
+ RestoreOptions *ropt;
+
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump complete\n--\n\n", "COMMENT");
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
+
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
+ }
+ else if (filename)
{
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -690,12 +820,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -770,6 +902,7 @@ static void
dropRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer delQry = createPQExpBuffer();
PGresult *res;
int i_rolname;
int i;
@@ -786,12 +919,17 @@ dropRoles(PGconn *conn)
"FROM %s "
"ORDER BY 1", role_catalog);
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
i_rolname = PQfnumber(res, "rolname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -799,15 +937,21 @@ dropRoles(PGconn *conn)
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(rolename));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropRoles");
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -871,7 +1015,7 @@ dumpRoles(PGconn *conn)
"FROM %s "
"ORDER BY 2", role_catalog);
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
i_oid = PQfnumber(res, "oid");
i_rolname = PQfnumber(res, "rolname");
@@ -889,7 +1033,12 @@ dumpRoles(PGconn *conn)
i_is_current_user = PQfnumber(res, "is_current_user");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -993,7 +1142,10 @@ dumpRoles(PGconn *conn)
"ROLE", rolename,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoles");
}
/*
@@ -1001,15 +1153,13 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
}
@@ -1076,7 +1226,7 @@ dumpRoleMembership(PGconn *conn)
"LEFT JOIN %s ug on ug.oid = a.grantor "
"WHERE NOT (ur.rolname ~ '^pg_' AND um.rolname ~ '^pg_')"
"ORDER BY 1,2,3", role_catalog, role_catalog, role_catalog);
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
i_role = PQfnumber(res, "role");
i_member = PQfnumber(res, "member");
i_grantor = PQfnumber(res, "grantor");
@@ -1088,7 +1238,12 @@ dumpRoleMembership(PGconn *conn)
i_set_option = PQfnumber(res, "set_option");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role memberships\n--\n\n", "COMMENT");
+ }
/*
* We can't dump these GRANT commands in arbitrary order, because a role
@@ -1167,6 +1322,7 @@ dumpRoleMembership(PGconn *conn)
char *grantor;
char *set_option = "true";
bool found;
+ PQExpBuffer creaQry = createPQExpBuffer();
/* If we already did this grant, don't do it again. */
if (done[i - start])
@@ -1223,8 +1379,8 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ appendPQExpBuffer(creaQry, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(creaQry, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1245,10 +1401,15 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(creaQry, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(creaQry, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(creaQry, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", creaQry->data);
+ else
+ createOneArchiveEntry(creaQry->data, "dumpRoleMembership");
}
}
@@ -1260,7 +1421,8 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1285,10 +1447,15 @@ dumpRoleGUCPrivs(PGconn *conn)
"paracl, "
"pg_catalog.acldefault('p', " CppAsString2(BOOTSTRAP_SUPERUSERID) ") AS acldefault "
"FROM pg_catalog.pg_parameter_acl "
- "ORDER BY 1");
+ "ORDER BY 1", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role privileges on configuration parameters\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1312,14 +1479,19 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoleGUCPrivs");
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1331,6 +1503,7 @@ dropTablespaces(PGconn *conn)
{
PGresult *res;
int i;
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Get all tablespaces except built-in ones (which we assume are named
@@ -1339,23 +1512,34 @@ dropTablespaces(PGconn *conn)
res = executeQuery(conn, "SELECT spcname "
"FROM pg_catalog.pg_tablespace "
"WHERE spcname !~ '^pg_' "
- "ORDER BY 1");
+ "ORDER BY 1", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(spcname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropTablespaces");
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1379,10 +1563,15 @@ dumpTablespaces(PGconn *conn)
"pg_catalog.shobj_description(oid, 'pg_tablespace') "
"FROM pg_catalog.pg_tablespace "
"WHERE spcname !~ '^pg_' "
- "ORDER BY 1");
+ "ORDER BY 1", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1451,14 +1640,19 @@ dumpTablespaces(PGconn *conn)
"TABLESPACE", spcname,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpTablespaces");
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1479,10 +1673,15 @@ dropDBs(PGconn *conn)
"SELECT datname "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
- "ORDER BY datname");
+ "ORDER BY datname", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop databases (except postgres and template1)\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1497,15 +1696,23 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(dbname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "DROP_DATABASE");
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1525,14 +1732,25 @@ dumpUserConfig(PGconn *conn, const char *username)
appendStringLiteralConn(buf, username, conn);
appendPQExpBufferChar(buf, ')');
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
if (PQntuples(res) > 0)
{
char *sanitized;
sanitized = sanitize_line(username, true);
- fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
}
@@ -1542,7 +1760,11 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpUserConfig");
}
PQclear(res);
@@ -1591,7 +1813,7 @@ expand_dbname_patterns(PGconn *conn,
exit_nicely(1);
}
- res = executeQuery(conn, query->data);
+ res = executeQuery(conn, query->data, fout ? true : false);
for (int i = 0; i < PQntuples(res); i++)
{
simple_string_list_append(names, PQgetvalue(res, i, 0));
@@ -1608,10 +1830,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, bool output_clean)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1625,19 +1850,49 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
- "ORDER BY (datname <> 'template1'), datname");
+ "ORDER BY (datname <> 'template1'), datname",
+ fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Databases\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Databases\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Databases\n--\n\n", "COMMENT");
+ }
+
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1654,7 +1909,18 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
/*
@@ -1669,24 +1935,46 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
+ fprintf(OPF, "\\connect %s\n\n", dbname);
else
{
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\\connect %s\n\n", dbname);
+ createOneArchiveEntry(qry->data, "CONNECT");
+ destroyPQExpBuffer(qry);
}
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1695,6 +1983,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1704,7 +1996,7 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1713,17 +2005,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s -f %s %s", pg_dump_bin,
+ pgdumpopts->data, dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1766,7 +2077,7 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
PGresult *res;
buildShSecLabelQuery(catalog_name, objectId, sql);
- res = executeQuery(conn, sql->data);
+ res = executeQuery(conn, sql->data, fout ? true : false);
emitShSecLabels(conn, res, buffer, objtype, objname);
PQclear(res);
@@ -1868,3 +2179,67 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+/*
+ * createDumpId
+ *
+ * This will return next last used oid.
+ */
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
+
+/*
+ * createOneArchiveEntry
+ *
+ * This creates one archive entry based on format.
+ */
+static void
+createOneArchiveEntry(const char *query, const char *tag)
+{
+ CatalogId nilCatalogId = {0, 0};
+ Assert(fout != NULL);
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = tag,
+ .section = SECTION_PRE_DATA,
+ .createStmt = query));
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index c9776306c5c..610b2ebf96f 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,12 +41,16 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -54,18 +58,44 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num,
+ bool globals_only);
+static int restore_global_objects(const char *inputFileSpec,
+ RestoreOptions *opts, int numWorkers,
+ int num, bool globals_only);
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+static bool data_only = false;
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
- bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +119,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +173,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +202,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +229,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global sql commands. */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -321,6 +356,10 @@ main(int argc, char **argv)
opts->restrict_key = pg_strdup(optarg);
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -347,6 +386,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -409,6 +455,9 @@ main(int argc, char **argv)
if (opts->single_txn && opts->txn_size > 0)
pg_fatal("options -1/--single-transaction and --transaction-size cannot be used together");
+ if (data_only && globals_only)
+ pg_fatal("options -a/--data-only and -g/--globals-only cannot be used together");
+
/*
* -C is not compatible with -1, because we can't create a database inside
* a transaction block.
@@ -472,6 +521,122 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.glo file is present, then restore all the
+ * databases from map.dat, but skip restoring those matching
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "toc.glo")))
+ {
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring an archive created by pg_dumpall");
+ else if (opts->tocFile)
+ pg_fatal("option -L/--use-list cannot be used when restoring an archive created by pg_dumpall");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option -C/--create must be specified when restoring an archive created by pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ char global_path[MAXPGPATH];
+
+ /* Set path for toc.glo file. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+ n_errors = restore_global_objects(global_path, opts, numWorkers, 0, globals_only);
+
+ pg_log_info("database restoring skipped because option -g/--globals-only was specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else
+ {
+ if (db_exclude_patterns.head != NULL)
+ {
+ simple_string_list_destroy(&db_exclude_patterns);
+ pg_fatal("option --exclude-database can be used only when restoring an archive created by pg_dumpall");
+ }
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring an archive created by pg_dumpall");
+
+ /* Process if toc.glo file does not exist. */
+ n_errors = restore_one_database(inputFileSpec, opts,
+ numWorkers, false, 0, globals_only);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ *
+ * If globals_only is set, then skip DROP DATABASE commands from restore.
+ */
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, int num, bool globals_only)
+{
+ int nerror = 0;
+ int format = opts->format;
+
+ /* Set format as custom so that toc.glo file can be read. */
+ opts->format = archCustom;
+
+ if (!data_only)
+ nerror = restore_one_database(inputFileSpec, opts, numWorkers,
+ false, num, globals_only);
+
+ /* Reset format value. */
+ opts->format = format;
+
+ return nerror;
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -479,9 +644,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -501,25 +672,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, globals_only);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -537,6 +704,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -553,6 +721,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -588,8 +757,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -694,3 +863,407 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn && db_exclude_patterns.head != NULL)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no database connection while doing pg_restore");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+ PQExpBuffer db_lit = createPQExpBuffer();
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data, false);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches exclude pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ destroyPQExpBuffer(db_lit);
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ destroyPQExpBuffer(query);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+
+
+ /*
+ * If there is no map.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen <= 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbname, db_oid, map_file_path);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+ PGconn *conn = NULL;
+ char global_path[MAXPGPATH];
+
+ /* Set path for toc.glo file. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
+
+ /* If map.dat has no entries, return after processing global commands. */
+ if (dbname_oid_list.head == NULL)
+ return restore_global_objects(global_path, opts, numWorkers, 0, false);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Open toc.dat file and execute/append all the global sql commands. */
+ n_errors_total = restore_global_objects(global_path, opts, numWorkers, 0, false);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..083f5c5bf9d
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -244,4 +244,31 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--restrict-key=uu', '-f dumpfile' ],
+ qr/\Qpg_dumpall: error: option --restrict-key can only be used with --format=plain\E/,
+ 'pg_dumpall: --restrict-key can only be used with plain dump format');
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option --exclude-database can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --exclude-database is used in pg_restore with dump of pg_dump'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--globals-only', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option -g\/--globals-only can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --globals-only is not used in pg_restore with dump of pg_dump'
+);
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100755
index 00000000000..3c7d2ad7c53
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,396 @@
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# Scape tablespace locations on Windows.
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT super TO grant7 WITH INHERIT TRUE GRANTED BY\E
+ (.*\n)*
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: When --list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: When non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+$node->stop('fast');
+
+done_testing();
--
2.47.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-12-08 17:09 tushar <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: tushar @ 2025-12-08 17:09 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
On Mon, Dec 8, 2025 at 12:14 PM Mahendra Singh Thalor <[email protected]>
wrote:
>
> I tried to fix these issues in the attached patch.
>
> Here, I am attaching an updated patch for the review and testing.
>
Thanks Mahendra, I am not able to apply the patche against the latest
sources, seems like you need to rebase it
[edb@1a1c15437e7c pg]$ git apply
/tmp/v11_08122025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch
error: patch failed: src/bin/pg_dump/pg_dumpall.c:419
error: src/bin/pg_dump/pg_dumpall.c: patch does not apply
error: patch failed: src/bin/pg_dump/pg_restore.c:409
error: src/bin/pg_dump/pg_restore.c: patch does not apply
[edb@1a1c15437e7c pg]$
regards,
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-12-08 18:48 Mahendra Singh Thalor <[email protected]>
parent: tushar <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-12-08 18:48 UTC (permalink / raw)
To: tushar <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
On Mon, 8 Dec 2025 at 22:39, tushar <[email protected]> wrote:
>
>
>
> On Mon, Dec 8, 2025 at 12:14 PM Mahendra Singh Thalor <[email protected]> wrote:
>>
>>
>> I tried to fix these issues in the attached patch.
>>
>> Here, I am attaching an updated patch for the review and testing.
>
>
> Thanks Mahendra, I am not able to apply the patche against the latest sources, seems like you need to rebase it
>
> [edb@1a1c15437e7c pg]$ git apply /tmp/v11_08122025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch
> error: patch failed: src/bin/pg_dump/pg_dumpall.c:419
> error: src/bin/pg_dump/pg_dumpall.c: patch does not apply
> error: patch failed: src/bin/pg_dump/pg_restore.c:409
> error: src/bin/pg_dump/pg_restore.c: patch does not apply
> [edb@1a1c15437e7c pg]$
>
> regards,
Thanks Tushar for the report.
In the last commit, there were some changes for error messages so this
was not applying cleanly.
> I have observed that when combining the --globals-only option with certain other switches during a pg_restore - operation fails silently.
> The attempted restore does not execute, but no error message or warning is displayed unless the --verbose option is also used.
>
> --this will just run without any message but objects also not going to create
> ./pg_restore -Fc ok31. -C -d postgres -t mytable --globals-only
> ./pg_restore -Fc ok31. -C -d postgres -no-tablespace --globals-only
> ./pg_restore -Fc ok31. -C -d postgres -no-data --globals-only
>
> with --verbose
> [edb@1a1c15437e7c bin]$ ./pg_restore -Fc ok31. -C -d postgres -t myable --globals-only -v
> pg_restore: connecting to database for restore
> pg_restore: executing SELECT pg_catalog.set_config('search_path', '', false);
> pg_restore: implied no-schema restore
> pg_restore: database restoring skipped because option -g/--globals-only was specified
>
> we should probably add some message there.
All these are good to me. In a successful case, we don't receive any
error message.(expected)
Here, I am attaching an updated patch for the review and testing. This
can be applied on commit d0d0ba6cf66c4043501f6f7.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v12_09122025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (89.9K, 2-v12_09122025-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
From 253baa9cca7ed9719e248d892c9b9665cc832c43 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Tue, 9 Dec 2025 00:08:18 +0530
Subject: [PATCH] Non text modes for pg_dumpall, correspondingly change
pg_restore
pg_dumpall acquires a new -F/--format option, with the same meanings as
pg_dump. The default is p, meaning plain text. For any other value, a
directory is created containing two files, toc.glo and map.dat. The
first contains commands restoring the global data in custom format, and the second
contains a map from oids to database names in text format. It will also contain a
subdirectory called databases, inside which it will create archives in
the specified format, named using the database oids.
In these casess the -f argument is required.
If pg_restore encounters a directory containing map.dat and toc.glo,
it restores the global settings from toc.glo if exist, and then
restores each database.
pg_restore acquires two new options: -g/--globals-only which suppresses
restoration of any databases, and --exclude-database which inhibits
restoration of particualr database(s) in the same way the same option
works in pg_dumpall.
v12
---
doc/src/sgml/ref/pg_dumpall.sgml | 104 ++++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/connectdb.c | 7 +-
src/bin/pg_dump/connectdb.h | 2 +-
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 10 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 34 +-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 618 +++++++++++++++++++++------
src/bin/pg_dump/pg_restore.c | 617 +++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 27 ++
src/bin/pg_dump/t/007_pg_dumpall.pl | 396 +++++++++++++++++
15 files changed, 1722 insertions(+), 167 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
create mode 100755 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 8834b7ec141..75de1fee330 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,10 +139,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>toc.dat/toc.dmp/toc.tar</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
@@ -937,9 +1020,13 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
<title>Examples</title>
<para>
To dump all databases:
-
+ If format is given, then dump will be based on format, default plain.
<screen>
<prompt>$</prompt> <userinput>pg_dumpall > db.out</userinput>
+</screen>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dumpall --format=d/a/c/p -f db.out</userinput>
</screen>
</para>
@@ -956,6 +1043,15 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
the script will attempt to drop other databases immediately, and that
will fail for the database you are connected to.
</para>
+
+ <para>
+ If dump was taken in non-text format, then use pg_restore to restore all databases.
+<screen>
+<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
+</screen>
+ This will restore all the databases. If user don't want to restore some databases, then use
+ --exclude-pattern to skip those.
+</para>
</refsect1>
<refsect1>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index a468a38361a..7497b527ae6 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -591,6 +615,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
index d55d53dbeea..d3e9e27003e 100644
--- a/src/bin/pg_dump/connectdb.c
+++ b/src/bin/pg_dump/connectdb.c
@@ -225,7 +225,7 @@ ConnectDatabase(const char *dbname, const char *connection_string,
exit_nicely(1);
}
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL, false));
return conn;
}
@@ -275,7 +275,7 @@ constructConnStr(const char **keywords, const char **values)
* Run a query, return the results, exit program on failure.
*/
PGresult *
-executeQuery(PGconn *conn, const char *query)
+executeQuery(PGconn *conn, const char *query, bool is_archive)
{
PGresult *res;
@@ -287,7 +287,8 @@ executeQuery(PGconn *conn, const char *query)
{
pg_log_error("query failed: %s", PQerrorMessage(conn));
pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
+ if (!is_archive)
+ PQfinish(conn);
exit_nicely(1);
}
diff --git a/src/bin/pg_dump/connectdb.h b/src/bin/pg_dump/connectdb.h
index 6c1e1954769..0b741b68cb1 100644
--- a/src/bin/pg_dump/connectdb.h
+++ b/src/bin/pg_dump/connectdb.h
@@ -22,5 +22,5 @@ extern PGconn *ConnectDatabase(const char *dbname, const char *connection_string
trivalue prompt_password, bool fail_on_error,
const char *progname, const char **connstr, int *server_version,
char *password, char *override_dbname);
-extern PGresult *executeQuery(PGconn *conn, const char *query);
+extern PGresult *executeQuery(PGconn *conn, const char *query, bool is_archive);
#endif /* CONNECTDB_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index f3c669f484e..3e21aaf5780 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..5974d6706fd 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index d9041dad720..f631d945472 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -312,7 +312,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data, bool globals_only);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index c84b017f21b..d35232cd038 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data, bool globals_only)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,20 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ /* Skip DROP DATABASE if globals_only. */
+ if (globals_only && te && te->tag && (strcmp(te->tag, "DROP_DATABASE") == 0))
+ continue;
+
+ /* Skip for CONNECT meta command. */
+ if (!ropt->filename && te && te->tag &&
+ (strcmp(te->tag, "CONNECT") == 0))
+ continue;
+
+ /* Skip if no-tablespace is given. */
+ if (ropt->noTablespace && te && te->tag && ((strcmp(te->tag, "dumpTablespaces") == 0) ||
+ (strcmp(te->tag, "dropTablespaces") == 0)))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1335,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1695,7 +1714,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1735,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..818b80a9369 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 24ad201af2f..f44d5e9d037 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1306,7 +1306,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 8fa04930399..8d4aac157ac 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -30,6 +30,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -65,9 +66,9 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, bool output_clean);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts, char *dbfile);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,11 +77,13 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
+static void createOneArchiveEntry(const char *query, const char *tag);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
static const char *connstr = "";
-static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
static bool dosync = true;
@@ -123,6 +126,10 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static ArchiveFormat archDumpFormat = archNull;
int
main(int argc, char *argv[])
@@ -148,6 +155,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +205,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *format_name = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -208,6 +217,7 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -245,8 +255,9 @@ main(int argc, char *argv[])
}
pgdumpopts = createPQExpBuffer();
+ InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -256,7 +267,7 @@ main(int argc, char *argv[])
break;
case 'c':
- output_clean = true;
+ dopt.outputClean = true;
break;
case 'd':
@@ -274,7 +285,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ format_name = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -314,6 +327,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -423,7 +437,7 @@ main(int argc, char *argv[])
exit_nicely(1);
}
- if (if_exists && !output_clean)
+ if (if_exists && !dopt.outputClean)
pg_fatal("option %s requires option %s",
"--if-exists", "-c/--clean");
@@ -435,6 +449,27 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(format_name);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option %s=d|c|t requires option %s",
+ "-F/--format", "-f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
+ /* restrict-key is only supported with --format=plain */
+ if (archDumpFormat != archNull && restrict_key)
+ pg_fatal("option %s can only be used with %s=plain",
+ "--restrict-key", "--format");
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -495,6 +530,27 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -544,19 +600,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -591,37 +634,110 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
- if (verbose)
+ if (verbose && archDumpFormat == archNull)
dumpTimestamp("Started on");
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ /* create a archive file for global commands. */
+ if (filename && archDumpFormat != archNull)
+ {
+ char global_path[MAXPGPATH];
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
+ /* Set file path for global sql commands. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", filename);
+
+ /* Open the output file */
+ fout = CreateArchive(global_path, archCustom, compression_spec,
+ dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC);
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
+ ((ArchiveHandle*)fout)->connection = conn;
+ ((ArchiveHandle*)fout)->public.numWorkers = 1;
- /* Replicate encoding and std_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
- if (strcmp(std_strings, "off") == 0)
- fprintf(OPF, "SET escape_string_warning = off;\n");
- fprintf(OPF, "\n");
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
+
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump\n--\n\n", "COMMENT");
+
+ /* default_transaction_read_only = off */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving default_transaction_read_only = off");
+ appendPQExpBuffer(qry, "SET default_transaction_read_only = off;\n");
+ createOneArchiveEntry(qry->data, "DEFAULT_TRANSACTION_READ_ONLY");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpEncoding: put the correct encoding into the archive */
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ const char *encname = pg_encoding_to_char(encoding);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+
+ pg_log_info("saving encoding = %s", encname);
+ createOneArchiveEntry(qry->data, "ENCODING");
+ destroyPQExpBuffer(qry);
+ }
+
+ /* dumpStdStrings: put the correct escape string behavior into the archive */
+ {
+ const char *stdstrings = std_strings ? "on" : "off";
+ PQExpBuffer qry = createPQExpBuffer();
+
+ pg_log_info("saving \"standard_conforming_strings = %s\"", stdstrings);
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = '%s';\n",
+ stdstrings);
+ createOneArchiveEntry(qry->data, "STDSTRINGS");
+ destroyPQExpBuffer(qry);
+ }
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and std_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = %s;\n", std_strings);
+ if (strcmp(std_strings, "off") == 0)
+ fprintf(OPF, "SET escape_string_warning = off;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -631,7 +747,7 @@ main(int argc, char *argv[])
* and tablespaces never depend on each other. Roles could have
* grants to each other, but DROP ROLE will clean those up silently.
*/
- if (output_clean)
+ if (dopt.outputClean)
{
if (!globals_only && !roles_only && !tablespaces_only)
dropDBs(conn);
@@ -665,27 +781,42 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump will
+ * handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
-
- PQfinish(conn);
+ dumpDatabases(conn, dopt.outputClean);
- if (verbose)
+ if (verbose && archDumpFormat == archNull)
dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (filename)
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+
+ if (archDumpFormat != archNull)
+ {
+ RestoreOptions *ropt;
+
+ createOneArchiveEntry("--\n-- PostgreSQL database cluster dump complete\n--\n\n", "COMMENT");
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
+
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
+ }
+ else if (filename)
{
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -696,12 +827,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -776,6 +909,7 @@ static void
dropRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer delQry = createPQExpBuffer();
PGresult *res;
int i_rolname;
int i;
@@ -792,12 +926,17 @@ dropRoles(PGconn *conn)
"FROM %s "
"ORDER BY 1", role_catalog);
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
i_rolname = PQfnumber(res, "rolname");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -805,15 +944,21 @@ dropRoles(PGconn *conn)
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(rolename));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropRoles");
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -877,7 +1022,7 @@ dumpRoles(PGconn *conn)
"FROM %s "
"ORDER BY 2", role_catalog);
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
i_oid = PQfnumber(res, "oid");
i_rolname = PQfnumber(res, "rolname");
@@ -895,7 +1040,12 @@ dumpRoles(PGconn *conn)
i_is_current_user = PQfnumber(res, "is_current_user");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Roles\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Roles\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Roles\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -999,7 +1149,10 @@ dumpRoles(PGconn *conn)
"ROLE", rolename,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoles");
}
/*
@@ -1007,15 +1160,13 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
}
@@ -1082,7 +1233,7 @@ dumpRoleMembership(PGconn *conn)
"LEFT JOIN %s ug on ug.oid = a.grantor "
"WHERE NOT (ur.rolname ~ '^pg_' AND um.rolname ~ '^pg_')"
"ORDER BY 1,2,3", role_catalog, role_catalog, role_catalog);
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
i_role = PQfnumber(res, "role");
i_member = PQfnumber(res, "member");
i_grantor = PQfnumber(res, "grantor");
@@ -1094,7 +1245,12 @@ dumpRoleMembership(PGconn *conn)
i_set_option = PQfnumber(res, "set_option");
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role memberships\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role memberships\n--\n\n", "COMMENT");
+ }
/*
* We can't dump these GRANT commands in arbitrary order, because a role
@@ -1173,6 +1329,7 @@ dumpRoleMembership(PGconn *conn)
char *grantor;
char *set_option = "true";
bool found;
+ PQExpBuffer creaQry = createPQExpBuffer();
/* If we already did this grant, don't do it again. */
if (done[i - start])
@@ -1229,8 +1386,8 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ appendPQExpBuffer(creaQry, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(creaQry, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1251,10 +1408,15 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(creaQry, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(creaQry, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(creaQry, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", creaQry->data);
+ else
+ createOneArchiveEntry(creaQry->data, "dumpRoleMembership");
}
}
@@ -1266,7 +1428,8 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1291,10 +1454,15 @@ dumpRoleGUCPrivs(PGconn *conn)
"paracl, "
"pg_catalog.acldefault('p', " CppAsString2(BOOTSTRAP_SUPERUSERID) ") AS acldefault "
"FROM pg_catalog.pg_parameter_acl "
- "ORDER BY 1");
+ "ORDER BY 1", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Role privileges on configuration parameters\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1318,14 +1486,19 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpRoleGUCPrivs");
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1337,6 +1510,7 @@ dropTablespaces(PGconn *conn)
{
PGresult *res;
int i;
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Get all tablespaces except built-in ones (which we assume are named
@@ -1345,23 +1519,34 @@ dropTablespaces(PGconn *conn)
res = executeQuery(conn, "SELECT spcname "
"FROM pg_catalog.pg_tablespace "
"WHERE spcname !~ '^pg_' "
- "ORDER BY 1");
+ "ORDER BY 1", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(spcname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "dropTablespaces");
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1385,10 +1570,15 @@ dumpTablespaces(PGconn *conn)
"pg_catalog.shobj_description(oid, 'pg_tablespace') "
"FROM pg_catalog.pg_tablespace "
"WHERE spcname !~ '^pg_' "
- "ORDER BY 1");
+ "ORDER BY 1", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Tablespaces\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1457,14 +1647,19 @@ dumpTablespaces(PGconn *conn)
"TABLESPACE", spcname,
buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpTablespaces");
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1485,10 +1680,15 @@ dropDBs(PGconn *conn)
"SELECT datname "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
- "ORDER BY datname");
+ "ORDER BY datname", fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Drop databases (except postgres and template1)\n--\n\n", "COMMENT");
+ }
for (i = 0; i < PQntuples(res); i++)
{
@@ -1503,15 +1703,23 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
+ PQExpBuffer delQry = createPQExpBuffer();
+
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(dbname));
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", delQry->data);
+ else
+ createOneArchiveEntry(delQry->data, "DROP_DATABASE");
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1531,14 +1739,25 @@ dumpUserConfig(PGconn *conn, const char *username)
appendStringLiteralConn(buf, username, conn);
appendPQExpBufferChar(buf, ')');
- res = executeQuery(conn, buf->data);
+ res = executeQuery(conn, buf->data, fout ? true : false);
if (PQntuples(res) > 0)
{
char *sanitized;
sanitized = sanitize_line(username, true);
- fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\n--\n-- User Config \"%s\"\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
}
@@ -1548,7 +1767,11 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ createOneArchiveEntry(buf->data, "dumpUserConfig");
}
PQclear(res);
@@ -1597,7 +1820,7 @@ expand_dbname_patterns(PGconn *conn,
exit_nicely(1);
}
- res = executeQuery(conn, query->data);
+ res = executeQuery(conn, query->data, fout ? true : false);
for (int i = 0; i < PQntuples(res); i++)
{
simple_string_list_append(names, PQgetvalue(res, i, 0));
@@ -1614,10 +1837,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, bool output_clean)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1631,19 +1857,49 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
- "ORDER BY (datname <> 'template1'), datname");
+ "ORDER BY (datname <> 'template1'), datname",
+ fout ? true : false);
if (PQntuples(res) > 0)
- fprintf(OPF, "--\n-- Databases\n--\n\n");
+ {
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Databases\n--\n\n");
+ else
+ createOneArchiveEntry("--\n-- Databases\n--\n\n", "COMMENT");
+ }
+
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1660,7 +1916,18 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ else
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+ createOneArchiveEntry(qry->data, "COMMENT");
+ destroyPQExpBuffer(qry);
+ }
+
free(sanitized);
/*
@@ -1675,24 +1942,46 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
+ fprintf(OPF, "\\connect %s\n\n", dbname);
else
{
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ PQExpBuffer qry = createPQExpBuffer();
+
+ appendPQExpBuffer(qry, "\\connect %s\n\n", dbname);
+ createOneArchiveEntry(qry->data, "CONNECT");
+ destroyPQExpBuffer(qry);
}
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1701,6 +1990,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1710,7 +2003,7 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1719,17 +2012,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s -f %s %s", pg_dump_bin,
+ pgdumpopts->data, dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1772,7 +2084,7 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
PGresult *res;
buildShSecLabelQuery(catalog_name, objectId, sql);
- res = executeQuery(conn, sql->data);
+ res = executeQuery(conn, sql->data, fout ? true : false);
emitShSecLabels(conn, res, buffer, objtype, objname);
PQclear(res);
@@ -1874,3 +2186,67 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+/*
+ * createDumpId
+ *
+ * This will return next last used oid.
+ */
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
+
+/*
+ * createOneArchiveEntry
+ *
+ * This creates one archive entry based on format.
+ */
+static void
+createOneArchiveEntry(const char *query, const char *tag)
+{
+ CatalogId nilCatalogId = {0, 0};
+ Assert(fout != NULL);
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = tag,
+ .section = SECTION_PRE_DATA,
+ .createStmt = query));
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 84b8d410c9e..f59813965bc 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,12 +41,16 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -54,18 +58,44 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num,
+ bool globals_only);
+static int restore_global_objects(const char *inputFileSpec,
+ RestoreOptions *opts, int numWorkers,
+ int num, bool globals_only);
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+static bool data_only = false;
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
- bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +119,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +173,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +202,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +229,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global sql commands. */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -321,6 +356,10 @@ main(int argc, char **argv)
opts->restrict_key = pg_strdup(optarg);
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -347,6 +386,14 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option %s cannot be used together with %s",
+ "--exclude-database", "-g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -420,6 +467,10 @@ main(int argc, char **argv)
pg_fatal("options %s and %s cannot be used together",
"-1/--single-transaction", "--transaction-size");
+ if (data_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-a/--data-only", "-g/--globals-only");
+
/*
* -C is not compatible with -1, because we can't create a database inside
* a transaction block.
@@ -485,6 +536,128 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.glo file is present, then restore all the
+ * databases from map.dat, but skip restoring those matching
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "toc.glo")))
+ {
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-l/--list");
+ else if (opts->tocFile)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-L/--use-list");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option %s must be specified when restoring an archive created by pg_dumpall",
+ "-C/--create");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ char global_path[MAXPGPATH];
+
+ /* Set path for toc.glo file. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+ n_errors = restore_global_objects(global_path, opts, numWorkers, 0, globals_only);
+
+ pg_log_info("database restoring skipped because option %s was specified",
+ "-g/--globals-only");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else
+ {
+ if (db_exclude_patterns.head != NULL)
+ {
+ simple_string_list_destroy(&db_exclude_patterns);
+ pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
+ "--exclude-database");
+ }
+
+ if (globals_only)
+ pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
+ "-g/--globals-only");
+
+ /* Process if toc.glo file does not exist. */
+ n_errors = restore_one_database(inputFileSpec, opts,
+ numWorkers, false, 0, globals_only);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ *
+ * If globals_only is set, then skip DROP DATABASE commands from restore.
+ */
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, int num, bool globals_only)
+{
+ int nerror = 0;
+ int format = opts->format;
+
+ /* Set format as custom so that toc.glo file can be read. */
+ opts->format = archCustom;
+
+ if (!data_only)
+ nerror = restore_one_database(inputFileSpec, opts, numWorkers,
+ false, num, globals_only);
+
+ /* Reset format value. */
+ opts->format = format;
+
+ return nerror;
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num, bool globals_only)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -492,9 +665,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -514,25 +693,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data, globals_only);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -550,6 +725,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -566,6 +742,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -601,8 +778,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -707,3 +884,407 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn && db_exclude_patterns.head != NULL)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no database connection while doing pg_restore");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+ PQExpBuffer db_lit = createPQExpBuffer();
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data, false);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches exclude pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ destroyPQExpBuffer(db_lit);
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ destroyPQExpBuffer(query);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+
+
+ /*
+ * If there is no map.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen <= 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbname, db_oid, map_file_path);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+ PGconn *conn = NULL;
+ char global_path[MAXPGPATH];
+
+ /* Set path for toc.glo file. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
+
+ /* If map.dat has no entries, return after processing global commands. */
+ if (dbname_oid_list.head == NULL)
+ return restore_global_objects(global_path, opts, numWorkers, 0, false);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Open toc.dat file and execute/append all the global sql commands. */
+ n_errors_total = restore_global_objects(global_path, opts, numWorkers, 0, false);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, 1, false);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..083f5c5bf9d
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -244,4 +244,31 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--restrict-key=uu', '-f dumpfile' ],
+ qr/\Qpg_dumpall: error: option --restrict-key can only be used with --format=plain\E/,
+ 'pg_dumpall: --restrict-key can only be used with plain dump format');
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option --exclude-database can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --exclude-database is used in pg_restore with dump of pg_dump'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--globals-only', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option -g\/--globals-only can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --globals-only is not used in pg_restore with dump of pg_dump'
+);
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100755
index 00000000000..3c7d2ad7c53
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,396 @@
+# Copyright (c) 2021-2025, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# Scape tablespace locations on Windows.
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT super TO grant7 WITH INHERIT TRUE GRANTED BY\E
+ (.*\n)*
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: When --list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: When non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+$node->stop('fast');
+
+done_testing();
--
2.47.3
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-12-10 13:38 tushar <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: tushar @ 2025-12-10 13:38 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
On Tue, Dec 9, 2025 at 12:18 AM Mahendra Singh Thalor <[email protected]>
wrote:
> On Mon, 8 Dec 2025 at 22:39, tushar <[email protected]> wrote:
>
> Here, I am attaching an updated patch for the review and testing. This
> can be applied on commit d0d0ba6cf66c4043501f6f7.
>
>
Thanks, Mahendra, please refer to this scenario where if
"--transaction-size" switch is used with pg_dumpall/pg_restore, then the
table creation fails (or the table is not created)
Steps to reproduce:
1. Connect to the psql terminal, create a table/insert rows { create table
t(n int); insert into t values (generate_series(1,15)); }
2. Perform pg_dump operation { ./pg_dumpall -Ft -f tar.dump }
3. new cluster:
try to restore with --transaction-size switch { ./pg_restore -Ft tar.dump
-C -d postgres --transaction-size=10 } = Table failed to create
I have checked via pg_dump/pg_restore using --transaction-size, and it is
working fine, i.e, table is created successfully
./pg_dump -Ft -f tar.d postgres
./pg_restore --transaction-size=10 -Ft -d new_database tar.d
regards,
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-12-11 16:09 Mahendra Singh Thalor <[email protected]>
parent: tushar <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-12-11 16:09 UTC (permalink / raw)
To: tushar <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
On Wed, 10 Dec 2025 at 19:08, tushar <[email protected]> wrote:
>
>
>
> On Tue, Dec 9, 2025 at 12:18 AM Mahendra Singh Thalor <[email protected]> wrote:
>>
>> On Mon, 8 Dec 2025 at 22:39, tushar <[email protected]> wrote:
>>
>> Here, I am attaching an updated patch for the review and testing. This
>> can be applied on commit d0d0ba6cf66c4043501f6f7.
>>
>
> Thanks, Mahendra, please refer to this scenario where if "--transaction-size" switch is used with pg_dumpall/pg_restore, then the table creation fails (or the table is not created)
>
> Steps to reproduce:
> 1. Connect to the psql terminal, create a table/insert rows { create table t(n int); insert into t values (generate_series(1,15)); }
> 2. Perform pg_dump operation { ./pg_dumpall -Ft -f tar.dump }
> 3. new cluster:
> try to restore with --transaction-size switch { ./pg_restore -Ft tar.dump -C -d postgres --transaction-size=10 } = Table failed to create
>
> I have checked via pg_dump/pg_restore using --transaction-size, and it is working fine, i.e, table is created successfully
>
> ./pg_dump -Ft -f tar.d postgres
> ./pg_restore --transaction-size=10 -Ft -d new_database tar.d
>
> regards,
>
>
Thanks Tushar for the report.
If transaction-size is given as non-zero, then pg_restore behaves like
"-e/--exit-on-error". means if there is any error in restore, then
exit without restoring the full cluster.
Here, in our case, as the cluster already has a role with the current
user in restore, we are reporting error "pg_restore: error: could not
execute query: ERROR: role "role" already exists" and after this
error, restore is exiting.
If you restore using a different role, then you will not get any error
and the full cluster will be restored. I will add some handling to
ignore the "CREATE ROLE current_user" command in pg_restore.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-12-12 13:40 tushar <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: tushar @ 2025-12-12 13:40 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
On Thu, Dec 11, 2025 at 9:39 PM Mahendra Singh Thalor <[email protected]>
wrote:
>
>
> Here, in our case, as the cluster already has a role with the current
> user in restore, we are reporting error "pg_restore: error: could not
> execute query: ERROR: role "role" already exists" and after this
> error, restore is exiting.
>
> If you restore using a different role, then you will not get any error
> and the full cluster will be restored. I will add some handling to
> ignore the "CREATE ROLE current_user" command in pg_restore.
>
> Thanks Mahendra, Could you please also add some error message for this
below
pg_restore command:
postgres=# create table t(n int);
CREATE TABLE
postgres=# insert into t values (1),(10),(100);
INSERT 0 3
Perform pg_dump: ./pg_dump -Ft -f a.a1 postgres
Perform pg_restore: /pg_restore -Ft a.a1 -f -C -v
pg_restore: creating TABLE "public.t"
pg_restore: processing data for table "public.t"
[edb@1a1c15437e7c bin]$ ./psql postgres
psql (19devel)
Type "help" for help.
postgres=# \dt
Did not find any tables.
postgres=#
regards,
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2025-12-12 16:17 Mahendra Singh Thalor <[email protected]>
parent: tushar <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Mahendra Singh Thalor @ 2025-12-12 16:17 UTC (permalink / raw)
To: tushar <[email protected]>; +Cc: Vaibhav Dalvi <[email protected]>; [email protected]
On Fri, 12 Dec 2025 at 19:10, tushar <[email protected]> wrote:
>
>
>
> On Thu, Dec 11, 2025 at 9:39 PM Mahendra Singh Thalor <[email protected]> wrote:
>>
>>
>>
>> Here, in our case, as the cluster already has a role with the current
>> user in restore, we are reporting error "pg_restore: error: could not
>> execute query: ERROR: role "role" already exists" and after this
>> error, restore is exiting.
>>
>> If you restore using a different role, then you will not get any error
>> and the full cluster will be restored. I will add some handling to
>> ignore the "CREATE ROLE current_user" command in pg_restore.
>>
> Thanks Mahendra, Could you please also add some error message for this below
> pg_restore command:
> postgres=# create table t(n int);
> CREATE TABLE
> postgres=# insert into t values (1),(10),(100);
> INSERT 0 3
> Perform pg_dump: ./pg_dump -Ft -f a.a1 postgres
> Perform pg_restore: /pg_restore -Ft a.a1 -f -C -v
> pg_restore: creating TABLE "public.t"
> pg_restore: processing data for table "public.t"
> [edb@1a1c15437e7c bin]$ ./psql postgres
> psql (19devel)
> Type "help" for help.
> postgres=# \dt
> Did not find any tables.
> postgres=#
>
> regards,
>
Hi Tushar,
This is the handling of command line arguments.
In code, after "-f", we expect file name, but here you are using "-C"
which will be considered as file name. This is the case for all the
command line arguments.
If pg_restore has the "-f" option, then the "-d database" name can't
be given and data will be copied into "-f filename" (it will not be
restored in the cluster).
Please let me know if you still have some doubts.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
end of thread, other threads:[~2025-12-12 16:17 UTC | newest]
Thread overview: 65+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-04-04 20:11 Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-04-10 17:13 ` Mahendra Singh Thalor <[email protected]>
2025-04-10 18:38 ` Ranier Vilela <[email protected]>
2025-04-10 18:58 ` Andrew Dunstan <[email protected]>
2025-04-10 21:45 ` Ranier Vilela <[email protected]>
2025-04-15 18:30 ` Mahendra Singh Thalor <[email protected]>
2025-07-08 21:28 ` Noah Misch <[email protected]>
2025-07-09 18:51 ` Mahendra Singh Thalor <[email protected]>
2025-07-16 00:19 ` Noah Misch <[email protected]>
2025-07-17 10:16 ` Mahendra Singh Thalor <[email protected]>
2025-07-17 10:18 ` Mahendra Singh Thalor <[email protected]>
2025-07-21 20:41 ` Andrew Dunstan <[email protected]>
2025-07-22 00:53 ` Noah Misch <[email protected]>
2025-07-22 01:43 ` Andrew Dunstan <[email protected]>
2025-07-24 20:33 ` Andrew Dunstan <[email protected]>
2025-07-25 16:21 ` Noah Misch <[email protected]>
2025-07-25 19:31 ` Andrew Dunstan <[email protected]>
2025-07-25 20:59 ` Tom Lane <[email protected]>
2025-07-27 23:56 ` Noah Misch <[email protected]>
2025-07-28 12:04 ` Andrew Dunstan <[email protected]>
2025-07-29 20:09 ` Andrew Dunstan <[email protected]>
2025-07-29 20:34 ` Noah Misch <[email protected]>
2025-07-30 01:07 ` Andrew Dunstan <[email protected]>
2025-07-30 18:51 ` Andrew Dunstan <[email protected]>
2025-07-31 09:44 ` Christoph Berg <[email protected]>
2025-07-31 13:40 ` Andrew Dunstan <[email protected]>
2025-07-31 13:52 ` Christoph Berg <[email protected]>
2025-07-31 18:22 ` Nathan Bossart <[email protected]>
2025-07-31 19:06 ` Andrew Dunstan <[email protected]>
2025-08-24 01:08 ` Noah Misch <[email protected]>
2025-08-24 16:42 ` Andrew Dunstan <[email protected]>
2025-10-15 17:35 ` Mahendra Singh Thalor <[email protected]>
2025-10-16 10:54 ` Mahendra Singh Thalor <[email protected]>
2025-10-28 06:02 ` Mahendra Singh Thalor <[email protected]>
2025-10-31 09:20 ` Mahendra Singh Thalor <[email protected]>
2025-11-03 06:35 ` Vaibhav Dalvi <[email protected]>
2025-11-03 11:54 ` Mahendra Singh Thalor <[email protected]>
2025-11-04 12:53 ` tushar <[email protected]>
2025-11-04 16:55 ` Andrew Dunstan <[email protected]>
2025-11-05 06:59 ` Vaibhav Dalvi <[email protected]>
2025-11-05 13:16 ` Vaibhav Dalvi <[email protected]>
2025-11-06 05:33 ` Mahendra Singh Thalor <[email protected]>
2025-11-11 05:59 ` Mahendra Singh Thalor <[email protected]>
2025-11-11 15:11 ` Andrew Dunstan <[email protected]>
2025-11-17 17:15 ` Mahendra Singh Thalor <[email protected]>
2025-11-18 10:34 ` Vaibhav Dalvi <[email protected]>
2025-11-27 08:15 ` Mahendra Singh Thalor <[email protected]>
2025-11-27 09:19 ` Mahendra Singh Thalor <[email protected]>
2025-12-01 13:06 ` tushar <[email protected]>
2025-12-01 17:17 ` tushar <[email protected]>
2025-12-02 13:15 ` tushar <[email protected]>
2025-12-08 06:44 ` Mahendra Singh Thalor <[email protected]>
2025-12-08 17:09 ` tushar <[email protected]>
2025-12-08 18:48 ` Mahendra Singh Thalor <[email protected]>
2025-12-10 13:38 ` tushar <[email protected]>
2025-12-11 16:09 ` Mahendra Singh Thalor <[email protected]>
2025-12-12 13:40 ` tushar <[email protected]>
2025-12-12 16:17 ` Mahendra Singh Thalor <[email protected]>
2025-11-17 14:08 ` tushar <[email protected]>
2025-11-03 11:59 ` Vaibhav Dalvi <[email protected]>
2025-07-17 11:11 ` Álvaro Herrera <[email protected]>
2025-07-17 12:52 ` Mahendra Singh Thalor <[email protected]>
2025-07-17 16:09 ` Andrew Dunstan <[email protected]>
2025-07-24 17:50 ` Noah Misch <[email protected]>
2025-07-24 18:02 ` Robert Haas <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox