public inbox for [email protected]
help / color / mirror / Atom feedRe: Non-text mode for pg_dumpall
22+ messages / 6 participants
[nested] [flat]
* Re: Non-text mode for pg_dumpall
@ 2026-01-28 07:34 tushar <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: tushar @ 2026-01-28 07:34 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: jian he <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On Tue, Jan 27, 2026 at 9:11 PM Mahendra Singh Thalor <[email protected]>
wrote:
> On Fri, 23 Jan 2026 at 19:07, tushar <[email protected]>
> wrote:
> >
> >
> >
> > On Fri, Jan 23, 2026 at 12:21 PM tushar <[email protected]>
> wrote:
> >>
> >>
> >> Thanks Mahendra, a minor observation - The pg_restore output shows a
> double slash in the map.dat path (e.g., abc.tar//map.dat).
> >> While it doesn't break the restore, we may want to clean up the path
> joining logic.
> >>
> >> [edb@1a1c15437e7c bin]$ ./pg_restore -Ft -C abc.tar/ -d postgres -p
> 9011 -U ed -v
> >> pg_restore: found database "template1
> >> " (OID: 1) in file "abc.tar//map.dat"
> >> pg_restore: found database "postgres
> >> " (OID: 5) in file "abc.tar//map.dat"
> >>
> >>
> >
> > Please refer to this scenario where - Objects created under template1
> and the postgres database by a specific user are failing during a
> cross-cluster restore.
> > When restoring to a new cluster as a different superuser, pg_restore
> throws the error: ERROR: role "edb" does not exist.
> > It appears the restore is attempting to preserve the original ownership
> of template1 objects even when the target environment lacks those specific
> roles.
> >
> > Steps to reproduce:
> > initdb ( ./initdb -U edb -D data) , start the server , connect to
> postgres and template1 database one by one and create
> > this table ( create table test(n int); )
> > perform pg_dumpall operation ( ./pg_dumpall -Ft -f abc.tar)
> > initdb (./initdb -U xyz) , start the server , create a database ( create
> database abc;)
> > perform pg_restore operation ( ./pg_restore -Ft -C abc.tar/ -d postgres
> -p 9033 -U xyz)
> > --getting an error, table 'test' will be created on 'template1'
> database but failed to create on an another database ( in this case - 'abc'
> database)
> >
> > regards,
>
> Hi,
> Here I am attaching an updated patch for the review and testing.
> Thanks Jian for the reporting rebase issue.
>
>
Thanks Mahendra, getting a regression error during the restore process
after applying this patch.
[edb@1a1c15437e7c bin]$ ./pg_restore -Ft -C abc1.tar/ -d postgres -p 9000
pg_restore: error: could not execute query: ERROR: non-standard string
literals are not supported
Command was: SET standard_conforming_strings = off;
pg_restore: warning: errors ignored on restore: 1
in earlier patches - this was not coming.
regards,
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-18 05:15 Mahendra Singh Thalor <[email protected]>
parent: tushar <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Mahendra Singh Thalor @ 2026-02-18 05:15 UTC (permalink / raw)
To: tushar <[email protected]>; +Cc: jian he <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On Wed, 28 Jan 2026 at 13:04, tushar <[email protected]> wrote:
>
>
>
> On Tue, Jan 27, 2026 at 9:11 PM Mahendra Singh Thalor <[email protected]>
wrote:
>>
>> On Fri, 23 Jan 2026 at 19:07, tushar <[email protected]>
wrote:
>> >
>> >
>> >
>> > On Fri, Jan 23, 2026 at 12:21 PM tushar <[email protected]>
wrote:
>> >>
>> >>
>> >> Thanks Mahendra, a minor observation - The pg_restore output shows
a double slash in the map.dat path (e.g., abc.tar//map.dat).
>> >> While it doesn't break the restore, we may want to clean up the path
joining logic.
>> >>
>> >> [edb@1a1c15437e7c bin]$ ./pg_restore -Ft -C abc.tar/ -d postgres -p
9011 -U ed -v
>> >> pg_restore: found database "template1
>> >> " (OID: 1) in file "abc.tar//map.dat"
>> >> pg_restore: found database "postgres
>> >> " (OID: 5) in file "abc.tar//map.dat"
>> >>
>> >>
>> >
>> > Please refer to this scenario where - Objects created under template1
and the postgres database by a specific user are failing during a
cross-cluster restore.
>> > When restoring to a new cluster as a different superuser, pg_restore
throws the error: ERROR: role "edb" does not exist.
>> > It appears the restore is attempting to preserve the original
ownership of template1 objects even when the target environment lacks those
specific roles.
>> >
>> > Steps to reproduce:
>> > initdb ( ./initdb -U edb -D data) , start the server , connect to
postgres and template1 database one by one and create
>> > this table ( create table test(n int); )
>> > perform pg_dumpall operation ( ./pg_dumpall -Ft -f abc.tar)
>> > initdb (./initdb -U xyz) , start the server , create a database (
create database abc;)
>> > perform pg_restore operation ( ./pg_restore -Ft -C abc.tar/ -d
postgres -p 9033 -U xyz)
>> > --getting an error, table 'test' will be created on 'template1'
database but failed to create on an another database ( in this case - 'abc'
database)
>> >
>> > regards,
>>
>> Hi,
>> Here I am attaching an updated patch for the review and testing.
>> Thanks Jian for the reporting rebase issue.
>>
>
> Thanks Mahendra, getting a regression error during the restore process
after applying this patch.
>
> [edb@1a1c15437e7c bin]$ ./pg_restore -Ft -C abc1.tar/ -d postgres -p 9000
> pg_restore: error: could not execute query: ERROR: non-standard string
literals are not supported
> Command was: SET standard_conforming_strings = off;
> pg_restore: warning: errors ignored on restore: 1
>
> in earlier patches - this was not coming.
>
> regards,
>
Thanks Andrew for some design related feedback.
Thanks Jian for the offline discussions, reviews, testing and delta patches.
Thanks Tushar for the detailed testing.
*Brief about this patch:*
new option to pg_dumpall: --format=d/t/c/p directory/tar/custam/plain
If the user gives a non-text format with pg_dumpall command, then the full
cluster will be dumped and global objects (roles. tablespaces, databases)
will be dumped into toc.glo file in custom format with drop commands and
databases will be dumped into a given archive format one by one with
oid.tar/oid.dmp/oid files/dir.
When restoring, if the user gives -g(globals-only) option, then creating
commands of only global users/tablespaces/databases will be restored. (no
drop commands will be executed)
toc.glo will be executed with -e(exit-on-error=false)
and --transaction-size=0 as some user already created. If the user wants to
restore a single database, they can restore it by a single dump file. For
--clean and -g(globals-only), we added some error cases so that
roles/databases/tablespaces will not be dropped.
Here, I am attaching an updated patch for the review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v16_18022026-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (98.3K, 3-v16_18022026-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
From 0cac227d9c5d75899c055039a0861ce71f734b2e Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Tue, 17 Feb 2026 15:36:20 +0530
Subject: [PATCH] Non text modes for pg_dumpall, correspondingly change
pg_restore
pg_dumpall acquires a new -F/--format option, with the same meanings as
pg_dump. The default is p, meaning plain text. For any other value, a
directory is created containing two files, toc.glo and map.dat. The
first contains commands restoring the global data in custom format, and the second
contains a map from oids to database names in text format. It will also contain a
subdirectory called databases, inside which it will create archives in
the specified format, named using the database oids.
In these casess the -f argument is required.
If pg_restore encounters a directory containing map.dat and toc.glo,
it restores the global settings from toc.glo if exist, and then
restores each database.
pg_restore acquires two new options: -g/--globals-only which suppresses
restoration of any databases, and --exclude-database which inhibits
restoration of particualr database(s) in the same way the same option
works in pg_dumpall.
v23 internal 1
---
doc/src/sgml/ref/pg_dumpall.sgml | 109 ++++-
doc/src/sgml/ref/pg_restore.sgml | 68 ++-
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 10 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 63 ++-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 688 +++++++++++++++++++++-----
src/bin/pg_dump/pg_restore.c | 691 ++++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 50 ++
src/bin/pg_dump/t/007_pg_dumpall.pl | 432 +++++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
14 files changed, 1961 insertions(+), 159 deletions(-)
create mode 100644 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 8834b7ec141..1e4678334fa 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,16 +139,93 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>toc.glo</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces in custom format. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
<listitem>
<para>
Dump only global objects (roles and tablespaces), no databases.
+ Note: <term><option>--globals-only</option></term> can not be used with
+ <term><option>--clean</option></term> with non-text dump format.
</para>
</listitem>
</varlistentry>
@@ -937,9 +1022,16 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
<title>Examples</title>
<para>
To dump all databases:
-
+ If format is given, then dump will be based on format, default plain.
<screen>
<prompt>$</prompt> <userinput>pg_dumpall > db.out</userinput>
+</screen>
+
+<screen>
+<prompt>$</prompt> <userinput>pg_dumpall --format=directory -f db.out</userinput>
+<prompt>$</prompt> <userinput>pg_dumpall --format=custom -f db.out</userinput>
+<prompt>$</prompt> <userinput>pg_dumpall --format=tar -f db.out</userinput>
+<prompt>$</prompt> <userinput>pg_dumpall --format=plain -f db.out</userinput>
</screen>
</para>
@@ -956,6 +1048,15 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
the script will attempt to drop other databases immediately, and that
will fail for the database you are connected to.
</para>
+
+ <para>
+ If dump was taken in non-text format, then use pg_restore to restore all databases.
+<screen>
+<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
+</screen>
+ This will restore all the databases. If user don't want to restore some databases, then use
+ --exclude-pattern to skip those.
+</para>
</refsect1>
<refsect1>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 420a308a7c7..c4f59187048 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,21 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ Note: <term><option>--globals-only</option></term> can not be used with <term><option>--exit-on-error</option></term>,
+ <term><option>--single-transaction</option></term>, <term><option>--clean</option></term> and <term><option>--transaction-size=N</option></term>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -581,6 +607,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 79bd5036841..7c9a475963b 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 56cb2c1f32d..29fa21a1cc0 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 2f8d9799c30..fda912ba0a9 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -313,7 +313,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 7afcc0859c8..faa08cedaa5 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,15 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ /* Skip if no-tablespace is given. */
+ if (ropt->noTablespace && te && te->desc &&
+ (strcmp(te->desc, "TABLESPACE") == 0))
+ continue;
+
+ /* Skip DROP DATABASE/ROLES/TABLESPACE if we didn't specify --clean */
+ if (!ropt->dropSchema && te && te->tag && (strcmp(te->tag, "DROP_GLOBAL") == 0))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1330,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1691,11 +1705,15 @@ archprintf(Archive *AH, const char *fmt,...)
/*******************************
* Stuff below here should be 'private' to the archiver routines
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
*******************************/
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1733,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
@@ -2391,7 +2409,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
/* initialize for backwards compatible string processing */
AH->public.encoding = 0; /* PG_SQL_ASCII */
- AH->public.std_strings = false;
+ AH->public.std_strings = true;
/* sql error handling */
AH->public.exit_on_error = true;
@@ -3027,6 +3045,16 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
return 0;
}
+ /*
+ * Global object TOC entries (e.g., ROLEs or TABLESPACEs) must not be
+ * ignored.
+ */
+ if (strcmp(te->desc, "ROLE") == 0 ||
+ strcmp(te->desc, "ROLE PROPERTIES") == 0 ||
+ strcmp(te->desc, "TABLESPACE") == 0 ||
+ strcmp(te->desc, "DROP_GLOBAL") == 0)
+ return REQ_SCHEMA;
+
/*
* Process exclusions that affect certain classes of TOC entries.
*/
@@ -3062,6 +3090,14 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
if (ropt->no_subscriptions &&
strncmp(te->tag, "SUBSCRIPTION", strlen("SUBSCRIPTION")) == 0)
return 0;
+
+ /*
+ * Global object TOC entries (e.g., ROLEs or TABLESPACEs) are treated as
+ * REQ_SCHEMA; follow this convention for their associated comments.
+ */
+ if (strncmp(te->desc, "ROLE", strlen("ROLE")) == 0 ||
+ strncmp(te->desc, "TABLESPACE", strlen("TABLESPACE")) == 0)
+ return REQ_SCHEMA;
}
/*
@@ -3091,6 +3127,14 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
if (ropt->no_subscriptions &&
strncmp(te->tag, "SUBSCRIPTION", strlen("SUBSCRIPTION")) == 0)
return 0;
+
+ /*
+ * Global object TOC entries (e.g., ROLEs or TABLESPACEs) are treated as
+ * REQ_SCHEMA; follow this convention for their associated SECURITY LABELs.
+ */
+ if (strncmp(te->tag, "ROLE", strlen("ROLE")) == 0 ||
+ strncmp(te->tag, "TABLESPACE", strlen("TABLESPACE")) == 0)
+ return REQ_SCHEMA;
}
/* If it's a subscription, maybe ignore it */
@@ -3865,6 +3909,9 @@ _getObjectDescription(PQExpBuffer buf, const TocEntry *te)
else if (strcmp(type, "CAST") == 0 ||
strcmp(type, "CHECK CONSTRAINT") == 0 ||
strcmp(type, "CONSTRAINT") == 0 ||
+ strcmp(type, "DROP_GLOBAL") == 0 ||
+ strcmp(type, "ROLE PROPERTIES") == 0 ||
+ strcmp(type, "ROLE") == 0 ||
strcmp(type, "DATABASE PROPERTIES") == 0 ||
strcmp(type, "DEFAULT") == 0 ||
strcmp(type, "FK CONSTRAINT") == 0 ||
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 49598304335..d3d0cac98df 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 98389d2034c..ec6fdaa822c 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -30,6 +30,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -67,7 +68,7 @@ static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
static void dumpDatabases(PGconn *conn);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts, char *dbfile);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,6 +77,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -123,6 +126,11 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static ArchiveFormat archDumpFormat = archNull;
+static const CatalogId nilCatalogId = {0, 0};
int
main(int argc, char *argv[])
@@ -148,6 +156,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +206,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *format_name = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -207,6 +217,7 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -244,8 +255,9 @@ main(int argc, char *argv[])
}
pgdumpopts = createPQExpBuffer();
+ InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -273,7 +285,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ format_name = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -313,6 +327,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -434,6 +449,32 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(format_name);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option %s=d|c|t requires option %s",
+ "-F/--format", "-f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
+ /* restrict-key is only supported with --format=plain */
+ if (archDumpFormat != archNull && restrict_key)
+ pg_fatal("option %s can only be used with %s=plain",
+ "--restrict-key", "--format");
+
+ /* --clean and -g/--globals-only cannot be used together in non-text dump */
+ if (archDumpFormat != archNull && output_clean && globals_only)
+ pg_fatal("options %s and %s cannot be used together in non-text dump",
+ "--clean", "-g/--globals-only");
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -494,6 +535,27 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -543,19 +605,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -593,35 +642,111 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
- if (verbose)
- dumpTimestamp("Started on");
+ /* create a archive file for global commands. */
+ if (archDumpFormat != archNull)
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ char global_path[MAXPGPATH];
+ const char *encname;
+
+ /* Set file path for global sql commands. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", filename);
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+ /* Open the output file */
+ fout = CreateArchive(global_path, archCustom, compression_spec,
+ dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC);
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+ ((ArchiveHandle *) fout)->connection = conn;
+ ((ArchiveHandle *) fout)->public.numWorkers = 1;
- /* Replicate encoding and standard_conforming_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release
+ * of our own major version. (See also version check in
+ * pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
+
+ /* Dump default_transaction_read_only. */
+ appendPQExpBufferStr(qry, "SET default_transaction_read_only = off;\n\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "default_transaction_read_only",
+ .description = "default_transaction_read_only",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ resetPQExpBuffer(qry);
+
+ /* Put the correct encoding into the archive */
+ encname = pg_encoding_to_char(encoding);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "client_encoding",
+ .description = "client_encoding",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ resetPQExpBuffer(qry);
+
+ /* Put the correct escape string behavior into the archive. */
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = 'on';\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "standard_conforming_strings",
+ .description = "standard_conforming_strings",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ destroyPQExpBuffer(qry);
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ if (verbose)
+ dumpTimestamp("Started on");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and standard_conforming_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = on;\n");
- fprintf(OPF, "\n");
+ fprintf(OPF, "SET standard_conforming_strings = on;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -630,8 +755,14 @@ main(int argc, char *argv[])
* dependency analysis because databases never depend on each other,
* and tablespaces never depend on each other. Roles could have
* grants to each other, but DROP ROLE will clean those up silently.
+ *
+ * For non-text formats, pg_dumpall unconditionally process --clean
+ * option. In contrast, pg_restore only applies it if the user
+ * explicitly provides the flag. This discrepancy resolves corner cases
+ * where pg_restore requires cleanup instructions that may be missing
+ * from a standard pg_dumpall output.
*/
- if (output_clean)
+ if (output_clean || archDumpFormat != archNull)
{
if (!globals_only && !roles_only && !tablespaces_only)
dropDBs(conn);
@@ -665,28 +796,45 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump
+ * will handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
dumpDatabases(conn);
- PQfinish(conn);
+ if (archDumpFormat == archNull)
+ {
+ PQfinish(conn);
- if (verbose)
- dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+ if (verbose)
+ dumpTimestamp("Completed on");
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (filename)
+ if (filename)
+ {
+ fclose(OPF);
+
+ /* sync the resulting file, errors are not fatal */
+ if (dosync && (archDumpFormat == archNull))
+ (void) fsync_fname(filename, false);
+ }
+ }
+ else
{
- fclose(OPF);
+ RestoreOptions *ropt;
- /* sync the resulting file, errors are not fatal */
- if (dosync)
- (void) fsync_fname(filename, false);
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
+
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
}
exit_nicely(0);
@@ -696,12 +844,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -796,24 +946,45 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
const char *rolename;
+ PQExpBuffer delQry = createPQExpBuffer();
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(rolename));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(rolename));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP ROLE IF EXISTS %s;\n",
+ fmtId(rolename));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "DROP_GLOBAL",
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -823,6 +994,8 @@ static void
dumpRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer comment_buf = createPQExpBuffer();
+ PQExpBuffer seclabel_buf = createPQExpBuffer();
PGresult *res;
int i_oid,
i_rolname,
@@ -894,7 +1067,7 @@ dumpRoles(PGconn *conn)
i_rolcomment = PQfnumber(res, "rolcomment");
i_is_current_user = PQfnumber(res, "is_current_user");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -913,6 +1086,8 @@ dumpRoles(PGconn *conn)
}
resetPQExpBuffer(buf);
+ resetPQExpBuffer(comment_buf);
+ resetPQExpBuffer(seclabel_buf);
if (binary_upgrade)
{
@@ -989,17 +1164,53 @@ dumpRoles(PGconn *conn)
if (!no_comments && !PQgetisnull(res, i, i_rolcomment))
{
- appendPQExpBuffer(buf, "COMMENT ON ROLE %s IS ", fmtId(rolename));
- appendStringLiteralConn(buf, PQgetvalue(res, i, i_rolcomment), conn);
- appendPQExpBufferStr(buf, ";\n");
+ appendPQExpBuffer(comment_buf, "COMMENT ON ROLE %s IS ", fmtId(rolename));
+ appendStringLiteralConn(comment_buf, PQgetvalue(res, i, i_rolcomment), conn);
+ appendPQExpBufferStr(comment_buf, ";\n");
}
if (!no_security_labels)
buildShSecLabels(conn, "pg_authid", auth_oid,
"ROLE", rolename,
- buf);
+ seclabel_buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ {
+ fprintf(OPF, "%s", buf->data);
+ fprintf(OPF, "%s", comment_buf->data);
+
+ if (seclabel_buf->data[0] != '\0')
+ fprintf(OPF, "%s", seclabel_buf->data);
+ }
+ else
+ {
+ char *tag = psprintf("%s %s", "ROLE", fmtId(rolename));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "ROLE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ if (comment_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "COMMENT",
+ .section = SECTION_PRE_DATA,
+ .createStmt = comment_buf->data));
+
+ if (seclabel_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "SECURITY LABEL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = seclabel_buf->data));
+ }
}
/*
@@ -1007,7 +1218,7 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "\n--\n-- User Configurations\n--\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1015,9 +1226,12 @@ dumpRoles(PGconn *conn)
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
+ destroyPQExpBuffer(comment_buf);
+ destroyPQExpBuffer(seclabel_buf);
}
@@ -1031,6 +1245,7 @@ static void
dumpRoleMembership(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer qurybuf = createPQExpBuffer();
PQExpBuffer optbuf = createPQExpBuffer();
PGresult *res;
int start = 0,
@@ -1093,7 +1308,7 @@ dumpRoleMembership(PGconn *conn)
i_inherit_option = PQfnumber(res, "inherit_option");
i_set_option = PQfnumber(res, "set_option");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Role memberships\n--\n\n");
/*
@@ -1229,8 +1444,9 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ resetPQExpBuffer(qurybuf);
+ appendPQExpBuffer(qurybuf, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(qurybuf, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1251,10 +1467,21 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(qurybuf, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(qurybuf, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(qurybuf, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", qurybuf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = fmtId(role),
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qurybuf->data));
}
}
@@ -1265,8 +1492,10 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
+ destroyPQExpBuffer(optbuf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1293,7 +1522,7 @@ dumpRoleGUCPrivs(PGconn *conn)
"FROM pg_catalog.pg_parameter_acl "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1318,14 +1547,25 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = fmtId(parowner),
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1347,21 +1587,45 @@ dropTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
+ PQExpBuffer delQry = createPQExpBuffer();
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
if_exists ? "IF EXISTS " : "",
fmtId(spcname));
+
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(spcname));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP TABLESPACE IF EXISTS %s;\n",
+ fmtId(spcname));
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "DROP_GLOBAL",
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1371,6 +1635,8 @@ static void
dumpTablespaces(PGconn *conn)
{
PGresult *res;
+ PQExpBuffer comment_buf = createPQExpBuffer();
+ PQExpBuffer seclabel_buf = createPQExpBuffer();
int i;
/*
@@ -1387,7 +1653,7 @@ dumpTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1406,6 +1672,9 @@ dumpTablespaces(PGconn *conn)
/* needed for buildACLCommands() */
fspcname = pg_strdup(fmtId(spcname));
+ resetPQExpBuffer(comment_buf);
+ resetPQExpBuffer(seclabel_buf);
+
if (binary_upgrade)
{
appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_tablespace oid\n");
@@ -1447,24 +1716,67 @@ dumpTablespaces(PGconn *conn)
if (!no_comments && spccomment && spccomment[0] != '\0')
{
- appendPQExpBuffer(buf, "COMMENT ON TABLESPACE %s IS ", fspcname);
- appendStringLiteralConn(buf, spccomment, conn);
- appendPQExpBufferStr(buf, ";\n");
+ appendPQExpBuffer(comment_buf, "COMMENT ON TABLESPACE %s IS ", fspcname);
+ appendStringLiteralConn(comment_buf, spccomment, conn);
+ appendPQExpBufferStr(comment_buf, ";\n");
}
if (!no_security_labels)
buildShSecLabels(conn, "pg_tablespace", spcoid,
"TABLESPACE", spcname,
- buf);
+ seclabel_buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ {
+ fprintf(OPF, "%s", buf->data);
+
+ if (comment_buf->data[0] != '\0')
+ fprintf(OPF, "%s", comment_buf->data);
+
+ if (seclabel_buf->data[0] != '\0')
+ fprintf(OPF, "%s", seclabel_buf->data);
+ }
+ else
+ {
+ char *tag = psprintf("%s %s", "TABLESPACE", fmtId(fspcname));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "TABLESPACE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+
+ if (comment_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "COMMENT",
+ .section = SECTION_PRE_DATA,
+ .createStmt = comment_buf->data));
+
+ if (seclabel_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "SECURITY LABEL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = seclabel_buf->data));
+ }
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ destroyPQExpBuffer(comment_buf);
+ destroyPQExpBuffer(seclabel_buf);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1487,12 +1799,13 @@ dropDBs(PGconn *conn)
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY datname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Skip "postgres" and "template1"; dumpDatabases() will deal with
@@ -1503,15 +1816,35 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(dbname));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(dbname));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP DATABASE IF EXISTS %s;\n",
+ fmtId(dbname));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "DROP_GLOBAL",
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1533,7 +1866,7 @@ dumpUserConfig(PGconn *conn, const char *username)
res = executeQuery(conn, buf->data);
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
{
char *sanitized;
@@ -1548,7 +1881,17 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = username,
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
}
PQclear(res);
@@ -1618,6 +1961,9 @@ dumpDatabases(PGconn *conn)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1631,19 +1977,43 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1660,7 +2030,10 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
free(sanitized);
/*
@@ -1675,24 +2048,40 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
+ fprintf(OPF, "\\connect %s\n\n", dbname);
else
- {
create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
- }
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1701,6 +2090,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1710,7 +2103,7 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1719,17 +2112,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s -f %s %s", pg_dump_bin,
+ pgdumpopts->data, dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1874,3 +2286,47 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+/*
+ * createDumpId
+ *
+ * This will return next last used oid.
+ */
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 84b8d410c9e..a53545c47dd 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,12 +41,16 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -54,18 +58,41 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data);
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts);
+
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
- bool data_only = false;
+ bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +116,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +170,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +199,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +226,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global sql commands. */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -321,6 +353,10 @@ main(int argc, char **argv)
opts->restrict_key = pg_strdup(optarg);
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -347,6 +383,14 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option %s cannot be used together with %s",
+ "--exclude-database", "-g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -420,6 +464,31 @@ main(int argc, char **argv)
pg_fatal("options %s and %s cannot be used together",
"-1/--single-transaction", "--transaction-size");
+ if (opts->single_txn && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--single-transaction", "-g/--globals-only");
+
+ if (opts->txn_size && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--transaction-size", "-g/--globals-only");
+
+ if (opts->exit_on_error && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--exit-on-error", "-g/--globals-only");
+
+ if (data_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-a/--data-only", "-g/--globals-only");
+ if (schema_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-s/--schema-only", "-g/--globals-only");
+ if (statistics_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "--statistics-only", "-g/--globals-only");
+ if (with_statistics && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "--statistics", "-g/--globals-only");
+
/*
* -C is not compatible with -1, because we can't create a database inside
* a transaction block.
@@ -485,6 +554,176 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.glo file is present, then restore all the databases from
+ * map.dat, but skip restoring those matching --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "toc.glo")))
+ {
+ char global_path[MAXPGPATH];
+ RestoreOptions *tmpopts = (RestoreOptions *) pg_malloc0(sizeof(RestoreOptions));
+
+ opts->format = archUnknown;
+
+ memcpy(tmpopts, opts, sizeof(RestoreOptions));
+
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-l/--list");
+ if (opts->tocFile)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-L/--use-list");
+
+ if (opts->strict_names)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--strict-names");
+ if (globals_only && opts->dropSchema)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--clean", "-g/--globals-only");
+
+ if (no_schema)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--no-schenma");
+
+ if (data_only)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-a/--data-only");
+
+ if (statistics_only)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--statistics-only");
+
+ if (!(opts->dumpSections & DUMP_PRE_DATA))
+ pg_fatal("option %s cannot exclude %s when restoring a pg_dumpall archive",
+ "--section", "--pre-data");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option %s must be specified when restoring an archive created by pg_dumpall",
+ "-C/--create");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Always restore global objects, even if --exclude-database results in
+ * zero databases to process. If 'globals-only' is set, exit
+ * immediately.
+ */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+
+ n_errors = restore_global_objects(global_path, tmpopts);
+
+ if (globals_only)
+ pg_log_info("database restoring skipped because option %s was specified",
+ "-g/--globals-only");
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else
+ {
+ if (db_exclude_patterns.head != NULL)
+ {
+ simple_string_list_destroy(&db_exclude_patterns);
+ pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
+ "--exclude-database");
+ }
+
+ if (globals_only)
+ pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
+ "-g/--globals-only");
+
+ /* Process if toc.glo file does not exist. */
+ n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ */
+static int
+restore_global_objects(const char *inputFileSpec, RestoreOptions *opts)
+{
+ Archive *AH;
+ int nerror = 0;
+
+ /* Set format as custom so that toc.glo file can be read. */
+ opts->format = archCustom;
+ opts->txn_size = 0;
+
+ AH = OpenArchive(inputFileSpec, opts->format);
+
+ SetArchiveOptions(AH, NULL, opts);
+
+ on_exit_close_archive(AH);
+
+ /* Let the archiver know how noisy to be */
+ AH->verbose = opts->verbose;
+
+ /*
+ * We're talking to the DB directly, don't send comments since they
+ * obscure SQL when displaying errors
+ */
+ ((ArchiveHandle *) AH)->noTocComments = 0;
+
+ AH->exit_on_error = false;
+
+ /* Parallel execution is not supported for global object restoration. */
+ AH->numWorkers = 1;
+
+ ProcessArchiveRestoreOptions(AH);
+ RestoreArchive(AH, false);
+
+ nerror = AH->n_errors;
+
+ /* AH may be freed in CloseArchive? */
+ CloseArchive(AH);
+
+ return nerror;
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -492,13 +731,25 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
+ /*
+ * We're talking to the DB directly, don't send comments since they
+ * obscure SQL when displaying errors
+ */
+ ((ArchiveHandle *) AH)->noTocComments = 0;
+
/*
* Whether to keep submitting sql commands as "pg_restore ... | psql ... "
*/
@@ -514,25 +765,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -550,6 +797,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -566,6 +814,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -601,8 +850,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -707,3 +956,409 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PQExpBuffer db_lit;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+ db_lit = createPQExpBuffer();
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+
+ resetPQExpBuffer(query);
+ resetPQExpBuffer(db_lit);
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if (PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches --exclude-database pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ count_db++;
+ }
+
+ destroyPQExpBuffer(query);
+ destroyPQExpBuffer(db_lit);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+ int len;
+
+
+ /*
+ * If there is no map.dat file in dump, then return from here as there is
+ * no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ len = strlen(dumpdirpath);
+
+ /* Trim slash from directry name. */
+ while (len > 1 && dumpdirpath[len - 1] == '/')
+ {
+ dumpdirpath[len - 1] = '\0';
+ len--;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen <= 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbidname->str, db_oid, map_file_path);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total = 0;
+ char *connected_db = NULL;
+ PGconn *conn = NULL;
+ RestoreOptions *original_opts = (RestoreOptions *) pg_malloc0(sizeof(RestoreOptions));
+ RestoreOptions *tmpopts = (RestoreOptions *) pg_malloc0(sizeof(RestoreOptions));
+
+ memcpy(original_opts, opts, sizeof(RestoreOptions));
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile((char*) inputFileSpec, &dbname_oid_list);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (!conn)
+ {
+ pg_log_error("could not connect to databases \"postgres\" or \"template1\"\n"
+ "Please specify an alternative database.");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with exclude-database patterns. */
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ return 0;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * Since pg_backup_archiver.c may modify RestoreOptions during the
+ * previous restore, we must provide a fresh copy of the original
+ * "opts" for each call to restore_one_database.
+ */
+ memcpy(tmpopts, original_opts, sizeof(RestoreOptions));
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (tmpopts->cparams.override_dbname)
+ {
+ pfree(tmpopts->cparams.override_dbname);
+ tmpopts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (tmpopts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, tmpopts->cparams.pghost,
+ tmpopts->cparams.pgport, tmpopts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ tmpopts->createDB = 0;
+ tmpopts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* We'll have to create it */
+ tmpopts->createDB = 1;
+ tmpopts->cparams.dbname = connected_db;
+ }
+ }
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true);
+
+ n_errors_total += n_errors;
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
index ab9310eb42b..f427fdbfa24 100644
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -244,4 +244,54 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--restrict-key=uu', '-f dumpfile' ],
+ qr/\Qpg_dumpall: error: option --restrict-key can only be used with --format=plain\E/,
+ 'pg_dumpall: --restrict-key can only be used with plain dump format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--globals-only', '--clean', '-f dumpfile' ],
+ qr/\Qpg_dumpall: error: options --clean and -g\/--globals-only cannot be used together in non-text dump\E/,
+ 'pg_dumpall: --clean and -g/--globals-only cannot be used together in non-text dump');
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--data-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options -a\/--data-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options -a/--data-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--schema-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options -s\/--schema-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options -s/--schema-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--statistics-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options --statistics-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options --statistics-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option --exclude-database can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --exclude-database is used in pg_restore with dump of pg_dump'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--globals-only', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option -g\/--globals-only can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --globals-only is used in pg_restore with the dump of pg_dump'
+);
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100644
index 00000000000..d754a80fa60
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,432 @@
+# Copyright (c) 2021-2026, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# Scape tablespace locations on Windows.
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: report an error when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: report an error when \l/--list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: report an error when -L/--use-list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--use-list' => 'use',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -L\/--use-list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When -L/--use-list is used in pg_restore with dump of pg_dumpall');
+
+# test case 4: report an error when --strict-names option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--strict-names',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --strict-names cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --strict-names is used in pg_restore with dump of pg_dumpall');
+
+# test case 5: report an error when --clean and -g\/--globals-only are used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--clean',
+ '--globals-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --clean and -g\/--globals-only cannot be used together when restoring an archive created by pg_dumpall\E/,
+ 'When --clean and -g\/--globals-only are used in pg_restore with dump of pg_dumpall');
+
+# test case 6: report an error when non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+
+$node->stop('fast');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..1a89ef94bec 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -600,6 +600,7 @@ CustomScanMethods
CustomScanState
CycleCtr
DBState
+DbOidName
DCHCacheEntry
DEADLOCK_INFO
DECountItem
--
2.52.0
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-19 20:00 Andrew Dunstan <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Andrew Dunstan @ 2026-02-19 20:00 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; tushar <[email protected]>; +Cc: jian he <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On 2026-02-18 We 12:15 AM, Mahendra Singh Thalor wrote:
> On Wed, 28 Jan 2026 at 13:04, tushar <[email protected]>
> wrote:
> >
> >
> >
> > On Tue, Jan 27, 2026 at 9:11 PM Mahendra Singh Thalor
> <[email protected]> wrote:
> >>
> >> On Fri, 23 Jan 2026 at 19:07, tushar
> <[email protected]> wrote:
> >> >
> >> >
> >> >
> >> > On Fri, Jan 23, 2026 at 12:21 PM tushar
> <[email protected]> wrote:
> >> >>
> >> >>
> >> >> Thanks Mahendra, a minor observation - The pg_restore output
> shows a double slash in the map.dat path (e.g., abc.tar//map.dat).
> >> >> While it doesn't break the restore, we may want to clean up the
> path joining logic.
> >> >>
> >> >> [edb@1a1c15437e7c bin]$ ./pg_restore -Ft -C abc.tar/ -d postgres
> -p 9011 -U ed -v
> >> >> pg_restore: found database "template1
> >> >> " (OID: 1) in file "abc.tar//map.dat"
> >> >> pg_restore: found database "postgres
> >> >> " (OID: 5) in file "abc.tar//map.dat"
> >> >>
> >> >>
> >> >
> >> > Please refer to this scenario where - Objects created under
> template1 and the postgres database by a specific user are failing
> during a cross-cluster restore.
> >> > When restoring to a new cluster as a different superuser,
> pg_restore throws the error: ERROR: role "edb" does not exist.
> >> > It appears the restore is attempting to preserve the original
> ownership of template1 objects even when the target environment lacks
> those specific roles.
> >> >
> >> > Steps to reproduce:
> >> > initdb ( ./initdb -U edb -D data) , start the server , connect to
> postgres and template1 database one by one and create
> >> > this table ( create table test(n int); )
> >> > perform pg_dumpall operation ( ./pg_dumpall -Ft -f abc.tar)
> >> > initdb (./initdb -U xyz) , start the server , create a database (
> create database abc;)
> >> > perform pg_restore operation ( ./pg_restore -Ft -C abc.tar/ -d
> postgres -p 9033 -U xyz)
> >> > --getting an error, table 'test' will be created on 'template1'
> database but failed to create on an another database ( in this case -
> 'abc' database)
> >> >
> >> > regards,
> >>
> >> Hi,
> >> Here I am attaching an updated patch for the review and testing.
> >> Thanks Jian for the reporting rebase issue.
> >>
> >
> > Thanks Mahendra, getting a regression error during the restore
> process after applying this patch.
> >
> > [edb@1a1c15437e7c bin]$ ./pg_restore -Ft -C abc1.tar/ -d postgres
> -p 9000
> > pg_restore: error: could not execute query: ERROR: non-standard
> string literals are not supported
> > Command was: SET standard_conforming_strings = off;
> > pg_restore: warning: errors ignored on restore: 1
> >
> > in earlier patches - this was not coming.
> >
> > regards,
> >
>
> Thanks Andrew for some design related feedback.
>
> Thanks Jian for the offline discussions, reviews, testing and delta
> patches.
>
> Thanks Tushar for the detailed testing.
>
> *Brief about this patch:*
> new option to pg_dumpall: --format=d/t/c/p directory/tar/custam/plain
>
> If the user gives a non-text format with pg_dumpall command, then the
> full cluster will be dumped and global objects (roles. tablespaces,
> databases) will be dumped into toc.glo file in custom format with drop
> commands and databases will be dumped into a given archive format one
> by one with oid.tar/oid.dmp/oid files/dir.
> When restoring, if the user gives -g(globals-only) option, then
> creating commands of only global users/tablespaces/databases will be
> restored. (no drop commands will be executed)
> toc.glo will be executed with -e(exit-on-error=false)
> and --transaction-size=0 as some user already created. If the user
> wants to restore a single database, they can restore it by a single
> dump file. For --clean and -g(globals-only), we added some error cases
> so that roles/databases/tablespaces will not be dropped.
> Here, I am attaching an updated patch for the review and testing.
Here's an update after a round of review. Most of the changes are pretty
minor, but it should get the cfbot all green, with a Windows fix in the
tests.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
Attachments:
[text/x-patch] v17_19022026-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (98.9K, 3-v17_19022026-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 8834b7ec141..49e5c99b09e 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,16 +139,93 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ This option can only be omitted when <option>--format</option> is plain.
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files,
+ <filename>toc.glo</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data (roles and tablespaces) in custom format. The second
+ contains a mapping between database OIDs and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ the <filename>databases</filename> subdirectory, named using the database's OID.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ See <xref linkend="app-pgdump"/> for details on how the
+ various non-plain-text archive formats work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
<listitem>
<para>
Dump only global objects (roles and tablespaces), no databases.
+ Note: <option>--globals-only</option> cannot be used with
+ <option>--clean</option> with non-text dump format.
</para>
</listitem>
</varlistentry>
@@ -936,13 +1021,21 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
<refsect1 id="app-pg-dumpall-ex">
<title>Examples</title>
<para>
- To dump all databases:
-
+ To dump all databases in plain text format (the default):
<screen>
<prompt>$</prompt> <userinput>pg_dumpall > db.out</userinput>
</screen>
</para>
+ <para>
+ To dump all databases using other formats:
+<screen>
+<prompt>$</prompt> <userinput>pg_dumpall --format=directory -f db.out</userinput>
+<prompt>$</prompt> <userinput>pg_dumpall --format=custom -f db.out</userinput>
+<prompt>$</prompt> <userinput>pg_dumpall --format=tar -f db.out</userinput>
+</screen>
+ </para>
+
<para>
To restore database(s) from this file, you can use:
<screen>
@@ -956,6 +1049,16 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
the script will attempt to drop other databases immediately, and that
will fail for the database you are connected to.
</para>
+
+ <para>
+ If the dump was taken in a non-plain-text format, use
+ <application>pg_restore</application> to restore the databases:
+<screen>
+<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
+</screen>
+ This will restore all databases. To restore only some databases, use
+ the <option>--exclude-database</option> option to skip those not wanted.
+ </para>
</refsect1>
<refsect1>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 420a308a7c7..b058326b3c9 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -152,6 +161,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +258,21 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ Note: <option>--globals-only</option> cannot be used with <option>--exit-on-error</option>,
+ <option>--single-transaction</option>, <option>--clean</option>, or <option>--transaction-size</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -581,6 +607,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 79bd5036841..7c9a475963b 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 56cb2c1f32d..29fa21a1cc0 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 2f8d9799c30..fda912ba0a9 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -313,7 +313,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 7afcc0859c8..faa08cedaa5 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,15 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ /* Skip if no-tablespace is given. */
+ if (ropt->noTablespace && te && te->desc &&
+ (strcmp(te->desc, "TABLESPACE") == 0))
+ continue;
+
+ /* Skip DROP DATABASE/ROLES/TABLESPACE if we didn't specify --clean */
+ if (!ropt->dropSchema && te && te->tag && (strcmp(te->tag, "DROP_GLOBAL") == 0))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1330,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1691,11 +1705,15 @@ archprintf(Archive *AH, const char *fmt,...)
/*******************************
* Stuff below here should be 'private' to the archiver routines
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
*******************************/
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1733,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
@@ -2391,7 +2409,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
/* initialize for backwards compatible string processing */
AH->public.encoding = 0; /* PG_SQL_ASCII */
- AH->public.std_strings = false;
+ AH->public.std_strings = true;
/* sql error handling */
AH->public.exit_on_error = true;
@@ -3027,6 +3045,16 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
return 0;
}
+ /*
+ * Global object TOC entries (e.g., ROLEs or TABLESPACEs) must not be
+ * ignored.
+ */
+ if (strcmp(te->desc, "ROLE") == 0 ||
+ strcmp(te->desc, "ROLE PROPERTIES") == 0 ||
+ strcmp(te->desc, "TABLESPACE") == 0 ||
+ strcmp(te->desc, "DROP_GLOBAL") == 0)
+ return REQ_SCHEMA;
+
/*
* Process exclusions that affect certain classes of TOC entries.
*/
@@ -3062,6 +3090,14 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
if (ropt->no_subscriptions &&
strncmp(te->tag, "SUBSCRIPTION", strlen("SUBSCRIPTION")) == 0)
return 0;
+
+ /*
+ * Global object TOC entries (e.g., ROLEs or TABLESPACEs) are treated as
+ * REQ_SCHEMA; follow this convention for their associated comments.
+ */
+ if (strncmp(te->desc, "ROLE", strlen("ROLE")) == 0 ||
+ strncmp(te->desc, "TABLESPACE", strlen("TABLESPACE")) == 0)
+ return REQ_SCHEMA;
}
/*
@@ -3091,6 +3127,14 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
if (ropt->no_subscriptions &&
strncmp(te->tag, "SUBSCRIPTION", strlen("SUBSCRIPTION")) == 0)
return 0;
+
+ /*
+ * Global object TOC entries (e.g., ROLEs or TABLESPACEs) are treated as
+ * REQ_SCHEMA; follow this convention for their associated SECURITY LABELs.
+ */
+ if (strncmp(te->tag, "ROLE", strlen("ROLE")) == 0 ||
+ strncmp(te->tag, "TABLESPACE", strlen("TABLESPACE")) == 0)
+ return REQ_SCHEMA;
}
/* If it's a subscription, maybe ignore it */
@@ -3865,6 +3909,9 @@ _getObjectDescription(PQExpBuffer buf, const TocEntry *te)
else if (strcmp(type, "CAST") == 0 ||
strcmp(type, "CHECK CONSTRAINT") == 0 ||
strcmp(type, "CONSTRAINT") == 0 ||
+ strcmp(type, "DROP_GLOBAL") == 0 ||
+ strcmp(type, "ROLE PROPERTIES") == 0 ||
+ strcmp(type, "ROLE") == 0 ||
strcmp(type, "DATABASE PROPERTIES") == 0 ||
strcmp(type, "DEFAULT") == 0 ||
strcmp(type, "FK CONSTRAINT") == 0 ||
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 49598304335..d3d0cac98df 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 98389d2034c..1e58646ccf8 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -1,13 +1,20 @@
/*-------------------------------------------------------------------------
*
* pg_dumpall.c
+ * pg_dumpall dumps all databases and global objects (roles and
+ * tablespaces) from a PostgreSQL cluster.
+ *
+ * For text format output, globals are written directly and pg_dump is
+ * invoked for each database, with all output going to stdout or a file.
+ *
+ * For non-text formats (custom, directory, tar), a directory is created
+ * containing a toc.glo file with global objects, a map.dat file mapping
+ * database OIDs to names, and a databases/ subdirectory with individual
+ * pg_dump archives for each database.
*
* Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
- * pg_dumpall forces all pg_dump output to be text, since it also outputs
- * text into the same output stream.
- *
* src/bin/pg_dump/pg_dumpall.c
*
*-------------------------------------------------------------------------
@@ -30,6 +37,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -67,7 +75,7 @@ static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
static void dumpDatabases(PGconn *conn);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts, char *dbfile);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,6 +84,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -123,6 +133,11 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static ArchiveFormat archDumpFormat = archNull;
+static const CatalogId nilCatalogId = {0, 0};
int
main(int argc, char *argv[])
@@ -148,6 +163,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +213,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *format_name = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -207,6 +224,7 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -244,8 +262,9 @@ main(int argc, char *argv[])
}
pgdumpopts = createPQExpBuffer();
+ InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -273,7 +292,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ format_name = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -313,6 +334,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -434,6 +456,32 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(format_name);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option %s=d|c|t requires option %s",
+ "-F/--format", "-f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
+ /* restrict-key is only supported with --format=plain */
+ if (archDumpFormat != archNull && restrict_key)
+ pg_fatal("option %s can only be used with %s=plain",
+ "--restrict-key", "--format");
+
+ /* --clean and -g/--globals-only cannot be used together in non-text dump */
+ if (archDumpFormat != archNull && output_clean && globals_only)
+ pg_fatal("options %s and %s cannot be used together in non-text dump",
+ "--clean", "-g/--globals-only");
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -494,6 +542,27 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -543,19 +612,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -593,35 +649,111 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
- if (verbose)
- dumpTimestamp("Started on");
-
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
-
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
-
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
-
- /* Replicate encoding and standard_conforming_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
+ /* create a archive file for global commands. */
+ if (archDumpFormat != archNull)
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ char global_path[MAXPGPATH];
+ const char *encname;
+
+ /* Set file path for global sql commands. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", filename);
+
+ /* Open the output file */
+ fout = CreateArchive(global_path, archCustom, compression_spec,
+ dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC);
+
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
+
+ ((ArchiveHandle *) fout)->connection = conn;
+ ((ArchiveHandle *) fout)->public.numWorkers = 1;
+
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release
+ * of our own major version. (See also version check in
+ * pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
+
+ /* Dump default_transaction_read_only. */
+ appendPQExpBufferStr(qry, "SET default_transaction_read_only = off;\n\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "default_transaction_read_only",
+ .description = "default_transaction_read_only",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ resetPQExpBuffer(qry);
+
+ /* Put the correct encoding into the archive */
+ encname = pg_encoding_to_char(encoding);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "client_encoding",
+ .description = "client_encoding",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ resetPQExpBuffer(qry);
+
+ /* Put the correct escape string behavior into the archive. */
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = 'on';\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "standard_conforming_strings",
+ .description = "standard_conforming_strings",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ destroyPQExpBuffer(qry);
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ if (verbose)
+ dumpTimestamp("Started on");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and standard_conforming_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = on;\n");
- fprintf(OPF, "\n");
+ fprintf(OPF, "SET standard_conforming_strings = on;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -630,8 +762,14 @@ main(int argc, char *argv[])
* dependency analysis because databases never depend on each other,
* and tablespaces never depend on each other. Roles could have
* grants to each other, but DROP ROLE will clean those up silently.
+ *
+ * For non-text formats, pg_dumpall unconditionally process --clean
+ * option. In contrast, pg_restore only applies it if the user
+ * explicitly provides the flag. This discrepancy resolves corner cases
+ * where pg_restore requires cleanup instructions that may be missing
+ * from a standard pg_dumpall output.
*/
- if (output_clean)
+ if (output_clean || archDumpFormat != archNull)
{
if (!globals_only && !roles_only && !tablespaces_only)
dropDBs(conn);
@@ -665,28 +803,45 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump
+ * will handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
dumpDatabases(conn);
- PQfinish(conn);
+ if (archDumpFormat == archNull)
+ {
+ PQfinish(conn);
+
+ if (verbose)
+ dumpTimestamp("Completed on");
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (verbose)
- dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+ if (filename)
+ {
+ fclose(OPF);
- if (filename)
+ /* sync the resulting file, errors are not fatal */
+ if (dosync && (archDumpFormat == archNull))
+ (void) fsync_fname(filename, false);
+ }
+ }
+ else
{
- fclose(OPF);
+ RestoreOptions *ropt;
+
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
- /* sync the resulting file, errors are not fatal */
- if (dosync)
- (void) fsync_fname(filename, false);
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
}
exit_nicely(0);
@@ -696,12 +851,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -796,24 +953,45 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
const char *rolename;
+ PQExpBuffer delQry = createPQExpBuffer();
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(rolename));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(rolename));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP ROLE IF EXISTS %s;\n",
+ fmtId(rolename));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "DROP_GLOBAL",
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -823,6 +1001,8 @@ static void
dumpRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer comment_buf = createPQExpBuffer();
+ PQExpBuffer seclabel_buf = createPQExpBuffer();
PGresult *res;
int i_oid,
i_rolname,
@@ -894,7 +1074,7 @@ dumpRoles(PGconn *conn)
i_rolcomment = PQfnumber(res, "rolcomment");
i_is_current_user = PQfnumber(res, "is_current_user");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -913,6 +1093,8 @@ dumpRoles(PGconn *conn)
}
resetPQExpBuffer(buf);
+ resetPQExpBuffer(comment_buf);
+ resetPQExpBuffer(seclabel_buf);
if (binary_upgrade)
{
@@ -989,17 +1171,53 @@ dumpRoles(PGconn *conn)
if (!no_comments && !PQgetisnull(res, i, i_rolcomment))
{
- appendPQExpBuffer(buf, "COMMENT ON ROLE %s IS ", fmtId(rolename));
- appendStringLiteralConn(buf, PQgetvalue(res, i, i_rolcomment), conn);
- appendPQExpBufferStr(buf, ";\n");
+ appendPQExpBuffer(comment_buf, "COMMENT ON ROLE %s IS ", fmtId(rolename));
+ appendStringLiteralConn(comment_buf, PQgetvalue(res, i, i_rolcomment), conn);
+ appendPQExpBufferStr(comment_buf, ";\n");
}
if (!no_security_labels)
buildShSecLabels(conn, "pg_authid", auth_oid,
"ROLE", rolename,
- buf);
+ seclabel_buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ {
+ fprintf(OPF, "%s", buf->data);
+ fprintf(OPF, "%s", comment_buf->data);
+
+ if (seclabel_buf->data[0] != '\0')
+ fprintf(OPF, "%s", seclabel_buf->data);
+ }
+ else
+ {
+ char *tag = psprintf("%s %s", "ROLE", fmtId(rolename));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "ROLE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ if (comment_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "COMMENT",
+ .section = SECTION_PRE_DATA,
+ .createStmt = comment_buf->data));
+
+ if (seclabel_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "SECURITY LABEL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = seclabel_buf->data));
+ }
}
/*
@@ -1007,7 +1225,7 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "\n--\n-- User Configurations\n--\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1015,9 +1233,12 @@ dumpRoles(PGconn *conn)
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
+ destroyPQExpBuffer(comment_buf);
+ destroyPQExpBuffer(seclabel_buf);
}
@@ -1031,6 +1252,7 @@ static void
dumpRoleMembership(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer querybuf = createPQExpBuffer();
PQExpBuffer optbuf = createPQExpBuffer();
PGresult *res;
int start = 0,
@@ -1093,7 +1315,7 @@ dumpRoleMembership(PGconn *conn)
i_inherit_option = PQfnumber(res, "inherit_option");
i_set_option = PQfnumber(res, "set_option");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Role memberships\n--\n\n");
/*
@@ -1229,8 +1451,9 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ resetPQExpBuffer(querybuf);
+ appendPQExpBuffer(querybuf, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(querybuf, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1251,10 +1474,21 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(querybuf, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(querybuf, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(querybuf, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", querybuf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = fmtId(role),
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = querybuf->data));
}
}
@@ -1265,8 +1499,11 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
+ destroyPQExpBuffer(querybuf);
+ destroyPQExpBuffer(optbuf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1293,7 +1530,7 @@ dumpRoleGUCPrivs(PGconn *conn)
"FROM pg_catalog.pg_parameter_acl "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1318,14 +1555,25 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = fmtId(parowner),
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1347,21 +1595,41 @@ dropTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
+ PQExpBuffer delQry = createPQExpBuffer();
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(spcname));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(spcname));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP TABLESPACE IF EXISTS %s;\n",
+ fmtId(spcname));
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "DROP_GLOBAL",
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1371,6 +1639,8 @@ static void
dumpTablespaces(PGconn *conn)
{
PGresult *res;
+ PQExpBuffer comment_buf = createPQExpBuffer();
+ PQExpBuffer seclabel_buf = createPQExpBuffer();
int i;
/*
@@ -1387,7 +1657,7 @@ dumpTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1406,6 +1676,9 @@ dumpTablespaces(PGconn *conn)
/* needed for buildACLCommands() */
fspcname = pg_strdup(fmtId(spcname));
+ resetPQExpBuffer(comment_buf);
+ resetPQExpBuffer(seclabel_buf);
+
if (binary_upgrade)
{
appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_tablespace oid\n");
@@ -1447,24 +1720,67 @@ dumpTablespaces(PGconn *conn)
if (!no_comments && spccomment && spccomment[0] != '\0')
{
- appendPQExpBuffer(buf, "COMMENT ON TABLESPACE %s IS ", fspcname);
- appendStringLiteralConn(buf, spccomment, conn);
- appendPQExpBufferStr(buf, ";\n");
+ appendPQExpBuffer(comment_buf, "COMMENT ON TABLESPACE %s IS ", fspcname);
+ appendStringLiteralConn(comment_buf, spccomment, conn);
+ appendPQExpBufferStr(comment_buf, ";\n");
}
if (!no_security_labels)
buildShSecLabels(conn, "pg_tablespace", spcoid,
"TABLESPACE", spcname,
- buf);
+ seclabel_buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ {
+ fprintf(OPF, "%s", buf->data);
+
+ if (comment_buf->data[0] != '\0')
+ fprintf(OPF, "%s", comment_buf->data);
+
+ if (seclabel_buf->data[0] != '\0')
+ fprintf(OPF, "%s", seclabel_buf->data);
+ }
+ else
+ {
+ char *tag = psprintf("%s %s", "TABLESPACE", fmtId(fspcname));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "TABLESPACE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+
+ if (comment_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "COMMENT",
+ .section = SECTION_PRE_DATA,
+ .createStmt = comment_buf->data));
+
+ if (seclabel_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "SECURITY LABEL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = seclabel_buf->data));
+ }
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ destroyPQExpBuffer(comment_buf);
+ destroyPQExpBuffer(seclabel_buf);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1487,12 +1803,13 @@ dropDBs(PGconn *conn)
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY datname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Skip "postgres" and "template1"; dumpDatabases() will deal with
@@ -1503,15 +1820,35 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(dbname));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(dbname));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP DATABASE IF EXISTS %s;\n",
+ fmtId(dbname));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "DROP_GLOBAL",
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1533,7 +1870,7 @@ dumpUserConfig(PGconn *conn, const char *username)
res = executeQuery(conn, buf->data);
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
{
char *sanitized;
@@ -1548,7 +1885,17 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = username,
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
}
PQclear(res);
@@ -1618,6 +1965,9 @@ dumpDatabases(PGconn *conn)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1631,19 +1981,43 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1660,7 +2034,10 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
free(sanitized);
/*
@@ -1675,24 +2052,40 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
- else
- {
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
fprintf(OPF, "\\connect %s\n\n", dbname);
- }
+ else
+ create_opts = "";
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1701,6 +2094,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1710,7 +2107,7 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1719,17 +2116,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s -f %s %s", pg_dump_bin,
+ pgdumpopts->data, dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1874,3 +2290,47 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+/*
+ * createDumpId
+ *
+ * Return the next dumpId.
+ */
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 84b8d410c9e..b351b29d2e4 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,12 +41,16 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -54,18 +58,41 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data);
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts);
+
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +116,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +170,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +199,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +226,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global sql commands. */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -321,6 +353,10 @@ main(int argc, char **argv)
opts->restrict_key = pg_strdup(optarg);
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -347,6 +383,14 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option %s cannot be used together with %s",
+ "--exclude-database", "-g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -420,6 +464,31 @@ main(int argc, char **argv)
pg_fatal("options %s and %s cannot be used together",
"-1/--single-transaction", "--transaction-size");
+ if (opts->single_txn && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--single-transaction", "-g/--globals-only");
+
+ if (opts->txn_size && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--transaction-size", "-g/--globals-only");
+
+ if (opts->exit_on_error && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--exit-on-error", "-g/--globals-only");
+
+ if (data_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-a/--data-only", "-g/--globals-only");
+ if (schema_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-s/--schema-only", "-g/--globals-only");
+ if (statistics_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "--statistics-only", "-g/--globals-only");
+ if (with_statistics && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "--statistics", "-g/--globals-only");
+
/*
* -C is not compatible with -1, because we can't create a database inside
* a transaction block.
@@ -485,6 +554,173 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.glo file is present, then restore all the databases from
+ * map.dat, but skip restoring those matching --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "toc.glo")))
+ {
+ char global_path[MAXPGPATH];
+ RestoreOptions *tmpopts = (RestoreOptions *) pg_malloc0(sizeof(RestoreOptions));
+
+ opts->format = archUnknown;
+
+ memcpy(tmpopts, opts, sizeof(RestoreOptions));
+
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-l/--list");
+ if (opts->tocFile)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-L/--use-list");
+
+ if (opts->strict_names)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--strict-names");
+ if (globals_only && opts->dropSchema)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--clean", "-g/--globals-only");
+
+ if (no_schema)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--no-schema");
+
+ if (data_only)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-a/--data-only");
+
+ if (statistics_only)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--statistics-only");
+
+ if (!(opts->dumpSections & DUMP_PRE_DATA))
+ pg_fatal("option %s cannot exclude %s when restoring a pg_dumpall archive",
+ "--section", "--pre-data");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option %s must be specified when restoring an archive created by pg_dumpall",
+ "-C/--create");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Always restore global objects, even if --exclude-database results in
+ * zero databases to process. If 'globals-only' is set, exit
+ * immediately.
+ */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+
+ n_errors = restore_global_objects(global_path, tmpopts);
+
+ if (globals_only)
+ pg_log_info("database restoring skipped because option %s was specified",
+ "-g/--globals-only");
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else
+ {
+ if (db_exclude_patterns.head != NULL)
+ {
+ simple_string_list_destroy(&db_exclude_patterns);
+ pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
+ "--exclude-database");
+ }
+
+ if (globals_only)
+ pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
+ "-g/--globals-only");
+
+ /* Process if toc.glo file does not exist. */
+ n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ */
+static int
+restore_global_objects(const char *inputFileSpec, RestoreOptions *opts)
+{
+ Archive *AH;
+ int nerror = 0;
+
+ /* Set format as custom so that toc.glo file can be read. */
+ opts->format = archCustom;
+ opts->txn_size = 0;
+
+ AH = OpenArchive(inputFileSpec, opts->format);
+
+ SetArchiveOptions(AH, NULL, opts);
+
+ on_exit_close_archive(AH);
+
+ /* Let the archiver know how noisy to be */
+ AH->verbose = opts->verbose;
+
+ /* Don't output TOC entry comments when restoring globals */
+ ((ArchiveHandle *) AH)->noTocComments = 1;
+
+ AH->exit_on_error = false;
+
+ /* Parallel execution is not supported for global object restoration. */
+ AH->numWorkers = 1;
+
+ ProcessArchiveRestoreOptions(AH);
+ RestoreArchive(AH, false);
+
+ nerror = AH->n_errors;
+
+ /* AH may be freed in CloseArchive? */
+ CloseArchive(AH);
+
+ return nerror;
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -492,9 +728,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -514,25 +756,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -550,6 +788,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -566,6 +805,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -601,8 +841,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -707,3 +947,420 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PQExpBuffer db_lit;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+ db_lit = createPQExpBuffer();
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+
+ resetPQExpBuffer(query);
+ resetPQExpBuffer(db_lit);
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if (PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches --exclude-database pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ count_db++;
+ }
+
+ destroyPQExpBuffer(query);
+ destroyPQExpBuffer(db_lit);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+ int len;
+
+
+ /*
+ * If there is no map.dat file in dump, then return from here as there is
+ * no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ len = strlen(dumpdirpath);
+
+ /* Trim slash from directory name. */
+ while (len > 1 && dumpdirpath[len - 1] == '/')
+ {
+ dumpdirpath[len - 1] = '\0';
+ len--;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* Extract dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+ if (p > linebuf.data && *p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Strip trailing newline */
+ if (namelen > 0 && dbname[namelen - 1] == '\n')
+ dbname[--namelen] = '\0';
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen < 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbidname->str, db_oid, map_file_path);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ pfree(linebuf.data);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total = 0;
+ char *connected_db = NULL;
+ PGconn *conn = NULL;
+ RestoreOptions *original_opts = (RestoreOptions *) pg_malloc0(sizeof(RestoreOptions));
+ RestoreOptions *tmpopts = (RestoreOptions *) pg_malloc0(sizeof(RestoreOptions));
+
+ memcpy(original_opts, opts, sizeof(RestoreOptions));
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile((char*) inputFileSpec, &dbname_oid_list);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (!conn)
+ {
+ pg_log_error("could not connect to databases \"postgres\" or \"template1\"\n"
+ "Please specify an alternative database.");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with exclude-database patterns. */
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ pg_free(original_opts);
+ pg_free(tmpopts);
+ return 0;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * Since pg_backup_archiver.c may modify RestoreOptions during the
+ * previous restore, we must provide a fresh copy of the original
+ * "opts" for each call to restore_one_database.
+ */
+ memcpy(tmpopts, original_opts, sizeof(RestoreOptions));
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (tmpopts->cparams.override_dbname)
+ {
+ pfree(tmpopts->cparams.override_dbname);
+ tmpopts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (tmpopts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, tmpopts->cparams.pghost,
+ tmpopts->cparams.pgport, tmpopts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ tmpopts->createDB = 0;
+ tmpopts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* We'll have to create it */
+ tmpopts->createDB = 1;
+ tmpopts->cparams.dbname = connected_db;
+ }
+ }
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true);
+
+ n_errors_total += n_errors;
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ pg_free(original_opts);
+ pg_free(tmpopts);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
index ab9310eb42b..a895bc314b0 100644
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -244,4 +244,59 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--restrict-key=uu', '-f dumpfile' ],
+ qr/\Qpg_dumpall: error: option --restrict-key can only be used with --format=plain\E/,
+ 'pg_dumpall: --restrict-key can only be used with plain dump format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--globals-only', '--clean', '-f', 'dumpfile' ],
+ qr/\Qpg_dumpall: error: options --clean and -g\/--globals-only cannot be used together in non-text dump\E/,
+ 'pg_dumpall: --clean and -g/--globals-only cannot be used together in non-text dump');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd' ],
+ qr/\Qpg_dumpall: error: option -F\/--format=d|c|t requires option -f\/--file\E/,
+ 'pg_dumpall: non-plain format requires --file option');
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--data-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options -a\/--data-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options -a/--data-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--schema-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options -s\/--schema-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options -s/--schema-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--statistics-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options --statistics-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options --statistics-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option --exclude-database can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --exclude-database is used in pg_restore with dump of pg_dump'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--globals-only', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option -g\/--globals-only can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --globals-only is used in pg_restore with the dump of pg_dump'
+);
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100644
index 00000000000..309f42beabb
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,481 @@
+# Copyright (c) 2021-2026, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# escape tablespace locations on Windows.
+my $tablespace2_orig = $tablespace2;
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2_orig';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# test case 1: report an error when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# test case 2: report an error when \l/--list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# test case 3: report an error when -L/--use-list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--use-list' => 'use',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -L\/--use-list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When -L/--use-list is used in pg_restore with dump of pg_dumpall');
+
+# test case 4: report an error when --strict-names option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--strict-names',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --strict-names cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --strict-names is used in pg_restore with dump of pg_dumpall');
+
+# test case 5: report an error when --clean and -g/--globals-only are used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--clean',
+ '--globals-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --clean and -g\/--globals-only cannot be used together when restoring an archive created by pg_dumpall\E/,
+ 'When --clean and -g/--globals-only are used in pg_restore with dump of pg_dumpall'
+);
+
+# test case 6: report an error when non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+# test case 7: report an error when --no-schema is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--no-schema',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --no-schema cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --no-schema is used in pg_restore with dump of pg_dumpall');
+
+# test case 8: report an error when --data-only is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--data-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -a\/--data-only cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --data-only is used in pg_restore with dump of pg_dumpall');
+
+# test case 9: report an error when --statistics-only is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--statistics-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --statistics-only cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --statistics-only is used in pg_restore with dump of pg_dumpall');
+
+# test case 10: report an error when --section excludes pre-data with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--section' => 'post-data',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --section cannot exclude --pre-data when restoring a pg_dumpall archive\E/,
+ 'When --section=post-data is used in pg_restore with dump of pg_dumpall');
+
+$node->stop('fast');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..1a89ef94bec 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -600,6 +600,7 @@ CustomScanMethods
CustomScanState
CycleCtr
DBState
+DbOidName
DCHCacheEntry
DEADLOCK_INFO
DECountItem
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-20 14:10 Mahendra Singh Thalor <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Mahendra Singh Thalor @ 2026-02-20 14:10 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: tushar <[email protected]>; jian he <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On Fri, 20 Feb 2026 at 01:30, Andrew Dunstan <[email protected]> wrote:
>
>
> On 2026-02-18 We 12:15 AM, Mahendra Singh Thalor wrote:
>
> On Wed, 28 Jan 2026 at 13:04, tushar <[email protected]> wrote:
> >
> >
> >
> > On Tue, Jan 27, 2026 at 9:11 PM Mahendra Singh Thalor <[email protected]> wrote:
> >>
> >> On Fri, 23 Jan 2026 at 19:07, tushar <[email protected]> wrote:
> >> >
> >> >
> >> >
> >> > On Fri, Jan 23, 2026 at 12:21 PM tushar <[email protected]> wrote:
> >> >>
> >> >>
> >> >> Thanks Mahendra, a minor observation - The pg_restore output shows a double slash in the map.dat path (e.g., abc.tar//map.dat).
> >> >> While it doesn't break the restore, we may want to clean up the path joining logic.
> >> >>
> >> >> [edb@1a1c15437e7c bin]$ ./pg_restore -Ft -C abc.tar/ -d postgres -p 9011 -U ed -v
> >> >> pg_restore: found database "template1
> >> >> " (OID: 1) in file "abc.tar//map.dat"
> >> >> pg_restore: found database "postgres
> >> >> " (OID: 5) in file "abc.tar//map.dat"
> >> >>
> >> >>
> >> >
> >> > Please refer to this scenario where - Objects created under template1 and the postgres database by a specific user are failing during a cross-cluster restore.
> >> > When restoring to a new cluster as a different superuser, pg_restore throws the error: ERROR: role "edb" does not exist.
> >> > It appears the restore is attempting to preserve the original ownership of template1 objects even when the target environment lacks those specific roles.
> >> >
> >> > Steps to reproduce:
> >> > initdb ( ./initdb -U edb -D data) , start the server , connect to postgres and template1 database one by one and create
> >> > this table ( create table test(n int); )
> >> > perform pg_dumpall operation ( ./pg_dumpall -Ft -f abc.tar)
> >> > initdb (./initdb -U xyz) , start the server , create a database ( create database abc;)
> >> > perform pg_restore operation ( ./pg_restore -Ft -C abc.tar/ -d postgres -p 9033 -U xyz)
> >> > --getting an error, table 'test' will be created on 'template1' database but failed to create on an another database ( in this case - 'abc' database)
> >> >
> >> > regards,
> >>
> >> Hi,
> >> Here I am attaching an updated patch for the review and testing.
> >> Thanks Jian for the reporting rebase issue.
> >>
> >
> > Thanks Mahendra, getting a regression error during the restore process after applying this patch.
> >
> > [edb@1a1c15437e7c bin]$ ./pg_restore -Ft -C abc1.tar/ -d postgres -p 9000
> > pg_restore: error: could not execute query: ERROR: non-standard string literals are not supported
> > Command was: SET standard_conforming_strings = off;
> > pg_restore: warning: errors ignored on restore: 1
> >
> > in earlier patches - this was not coming.
> >
> > regards,
> >
>
> Thanks Andrew for some design related feedback.
>
> Thanks Jian for the offline discussions, reviews, testing and delta patches.
>
> Thanks Tushar for the detailed testing.
>
> Brief about this patch:
> new option to pg_dumpall: --format=d/t/c/p directory/tar/custam/plain
>
> If the user gives a non-text format with pg_dumpall command, then the full cluster will be dumped and global objects (roles. tablespaces, databases) will be dumped into toc.glo file in custom format with drop commands and databases will be dumped into a given archive format one by one with oid.tar/oid.dmp/oid files/dir.
> When restoring, if the user gives -g(globals-only) option, then creating commands of only global users/tablespaces/databases will be restored. (no drop commands will be executed)
> toc.glo will be executed with -e(exit-on-error=false) and --transaction-size=0 as some user already created. If the user wants to restore a single database, they can restore it by a single dump file. For --clean and -g(globals-only), we added some error cases so that roles/databases/tablespaces will not be dropped.
>
> Here, I am attaching an updated patch for the review and testing.
>
>
> Here's an update after a round of review. Most of the changes are pretty minor, but it should get the cfbot all green, with a Windows fix in the tests.
>
>
> cheers
>
>
> andrew
>
>
> --
> Andrew Dunstan
> EDB: https://www.enterprisedb.com
Thanks Andrew for the updated patch.
Updated patch looks good to me. I have verified by some dump/restore testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-21 02:15 jian he <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: jian he @ 2026-02-21 02:15 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
Hi.
RestoreOptions *tmpopts = (RestoreOptions *)
pg_malloc0(sizeof(RestoreOptions));
need change to
RestoreOptions *tmpopts = pg_malloc0_object(RestoreOptions);
+ <para>
+ If the dump was taken in a non-plain-text format, use
+ <application>pg_restore</application> to restore the databases:
+<screen>
+<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
+</screen>
+ This will restore all databases. To restore only some databases, use
+ the <option>--exclude-database</option> option to skip those not wanted.
+ </para>
The change above was added to pg_dumpall.sgml, which seems inappropriate;
it would be more correct to place it in pg_restore.sgml.
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive
made using <application>pg_dumpall</application>.
+ Note: <option>--globals-only</option> cannot be used with
<option>--exit-on-error</option>,
+ <option>--single-transaction</option>,
<option>--clean</option>, or <option>--transaction-size</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<option>--globals-only</option> cannot be used with --data-only,
--schema-only, --statistics-only, --statistics.
We should also mention that.
In doc/src/sgml/ref/pg_restore.sgml
"when restoring from an archive made using pg_dumpall."
It would be better using
"when restoring from a non-plain-text archive made using pg_dumpall."
that would be aligned with pg_dumpall.sgml.
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
<listitem>
<para>
Dump only global objects (roles and tablespaces), no databases.
+ Note: <option>--globals-only</option> cannot be used with
+ <option>--clean</option> with non-text dump format.
</para>
Elsewhere, we use the term “non-plain-text,” so we should use
“non-plain-text” here as well instead of “non-text,” for consistency.
In doc/src/sgml/ref/pg_restore.sgml, We did not mention that many
options cannot be used with pg_restore when performing a
non-plain-text restore.
Like:
"-l/--list"
"-L/--use-list"
"--strict-names"
"--no-schema"
"-a/--data-only"
"--statistics-only"
--section does not include "--pre-data"
pg_restore --clean --format=directory will produce DROP DATABASE will
process global objects,
it will also produce DROP DATABASE when processing each individual database.
To prevent errors during a subsequent restore, we can require
pg_restore --clean option must be used together with --if-exists when
restoring a non-plain-text dump.
--
jian
https://www.enterprisedb.com
Attachments:
[text/x-patch] v17-0001-misc-review-for-v17.patch (7.0K, 2-v17-0001-misc-review-for-v17.patch)
download | inline diff:
From 73079702e513471d35fd0423350503644cf27e0f Mon Sep 17 00:00:00 2001
From: jian he <[email protected]>
Date: Sat, 21 Feb 2026 10:09:56 +0800
Subject: [PATCH v17 1/1] misc review for v17
based on v17_19022026-Non-text-modes-for-pg_dumpall-correspondingly-change.patch
---
doc/src/sgml/ref/pg_dumpall.sgml | 12 +-----------
doc/src/sgml/ref/pg_restore.sgml | 25 ++++++++++++++++++++++---
src/bin/pg_dump/pg_restore.c | 10 +++++++---
src/bin/pg_dump/t/007_pg_dumpall.pl | 1 +
4 files changed, 31 insertions(+), 17 deletions(-)
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 49e5c99b09e..94a674c9501 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -225,7 +225,7 @@ PostgreSQL documentation
<para>
Dump only global objects (roles and tablespaces), no databases.
Note: <option>--globals-only</option> cannot be used with
- <option>--clean</option> with non-text dump format.
+ <option>--clean</option> with non-plain-text dump format.
</para>
</listitem>
</varlistentry>
@@ -1049,16 +1049,6 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
the script will attempt to drop other databases immediately, and that
will fail for the database you are connected to.
</para>
-
- <para>
- If the dump was taken in a non-plain-text format, use
- <application>pg_restore</application> to restore the databases:
-<screen>
-<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
-</screen>
- This will restore all databases. To restore only some databases, use
- the <option>--exclude-database</option> option to skip those not wanted.
- </para>
</refsect1>
<refsect1>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index b058326b3c9..92407705fa9 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -58,6 +58,11 @@ PostgreSQL documentation
When restoring from a dump made by <application>pg_dumpall</application>,
each database will be created and then the restoration will be run in that
database.
+ Not all options are supported when restoring from a dump created by <application>pg_dumpall</application>.
+ The following options are not supported:
+ <option>-l/--list</option>, <option>-L/--use-list</option>, <option>--strict-names</option>, <option>--no-schema</option>,
+ <option>-a/--data-only</option>,<option>--statistics-only</option>
+ In addition, <option>--section</option> must include value<option>--pre-data</option>.
Otherwise, when a database name is not specified, a script containing the SQL
commands necessary to rebuild the database or cluster is created and written
@@ -138,6 +143,8 @@ PostgreSQL documentation
If any of the objects do not exist in the destination database,
ignorable error messages will be reported,
unless <option>--if-exists</option> is also specified.
+ If <option>--clean</option> is specified, then <option>--if-exists</option> must also be specified
+ when restoring multiple databases from an archive created by <application>pg_dumpall</application>.
</para>
</listitem>
</varlistentry>
@@ -266,8 +273,9 @@ PostgreSQL documentation
Restore only global objects (roles and tablespaces), no databases.
</para>
<para>
- This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
- Note: <option>--globals-only</option> cannot be used with <option>--exit-on-error</option>,
+ This option is only relevant when restoring from a non-plain-text archive made using <application>pg_dumpall</application>.
+ Note: <option>--globals-only</option> cannot be used with <option>--data-only</option>, <option>--exit-on-error</option>,
+ <option>--schema-only</option>, <option>--statistics-only</option>, <option>--statistics</option>,
<option>--single-transaction</option>, <option>--clean</option>, or <option>--transaction-size</option>.
</para>
</listitem>
@@ -624,7 +632,7 @@ PostgreSQL documentation
quote the pattern if needed to prevent shell wildcard expansion.
</para>
<para>
- This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ This option is only relevant when restoring from a non-plain-text archive made using <application>pg_dumpall</application>.
</para>
</listitem>
</varlistentry>
@@ -1234,6 +1242,17 @@ CREATE DATABASE foo WITH TEMPLATE template0;
initially empty.
</para>
+ <para>
+ Suppose we have used <application>pg_dumpall</application> non-plain-text
+ format dumped all the database in a cluster to directory <literal>db.dir</literal>,
+ use <application>pg_restore</application> to restore the databases:
+<screen>
+<prompt>$</prompt> <userinput>pg_restore db.dir -d postgres -C</userinput>
+</screen>
+ This will restore all databases. To restore only some databases, use
+ the <option>--exclude-database</option> option to skip those not wanted.
+ </para>
+
<para>
To reorder database items, it is first necessary to dump the table of
contents of the archive:
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index b351b29d2e4..f197ff81f20 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -562,12 +562,16 @@ main(int argc, char **argv)
(file_exists_in_directory(inputFileSpec, "toc.glo")))
{
char global_path[MAXPGPATH];
- RestoreOptions *tmpopts = (RestoreOptions *) pg_malloc0(sizeof(RestoreOptions));
+ RestoreOptions *tmpopts = pg_malloc0_object(RestoreOptions);
opts->format = archUnknown;
memcpy(tmpopts, opts, sizeof(RestoreOptions));
+ if (opts->dropSchema && !if_exists)
+ pg_fatal("option %s must also be specified if option %s is specified when restoring an archive created by pg_dumpall",
+ "--if-exists", "-c/--clean");
+
/*
* Can only use --list or --use-list options with a single database
* dump.
@@ -1182,8 +1186,8 @@ restore_all_databases(const char *inputFileSpec,
int n_errors_total = 0;
char *connected_db = NULL;
PGconn *conn = NULL;
- RestoreOptions *original_opts = (RestoreOptions *) pg_malloc0(sizeof(RestoreOptions));
- RestoreOptions *tmpopts = (RestoreOptions *) pg_malloc0(sizeof(RestoreOptions));
+ RestoreOptions *original_opts = pg_malloc0_object(RestoreOptions);
+ RestoreOptions *tmpopts = pg_malloc0_object(RestoreOptions);
memcpy(original_opts, opts, sizeof(RestoreOptions));
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
index 309f42beabb..84e537c9136 100644
--- a/src/bin/pg_dump/t/007_pg_dumpall.pl
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -409,6 +409,7 @@ $node->command_fails_like(
"$tempdir/format_custom", '-C',
'--format' => 'custom',
'--clean',
+ '--if-exists',
'--globals-only',
'--file' => "$tempdir/error_test.sql",
],
--
2.34.1
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-21 17:05 Andrew Dunstan <[email protected]>
parent: jian he <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Andrew Dunstan @ 2026-02-21 17:05 UTC (permalink / raw)
To: jian he <[email protected]>; Mahendra Singh Thalor <[email protected]>; +Cc: tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On 2026-02-20 Fr 9:15 PM, jian he wrote:
> Hi.
>
> RestoreOptions *tmpopts = (RestoreOptions *)
> pg_malloc0(sizeof(RestoreOptions));
> need change to
> RestoreOptions *tmpopts = pg_malloc0_object(RestoreOptions);
Fixed
>
> + <para>
> + If the dump was taken in a non-plain-text format, use
> + <application>pg_restore</application> to restore the databases:
> +<screen>
> +<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
> +</screen>
> + This will restore all databases. To restore only some databases, use
> + the <option>--exclude-database</option> option to skip those not wanted.
> + </para>
>
> The change above was added to pg_dumpall.sgml, which seems inappropriate;
> it would be more correct to place it in pg_restore.sgml.
I don't agree. The previous paragraph mentions using psql to restore a
text dump, so mentioning pg_restore for a non-text dump seems reasonable.
But I have added a para to the pg_restore docs explaining how it
processes pg_dumpall archives.
>
> + <varlistentry>
> + <term><option>-g</option></term>
> + <term><option>--globals-only</option></term>
> + <listitem>
> + <para>
> + Restore only global objects (roles and tablespaces), no databases.
> + </para>
> + <para>
> + This option is only relevant when restoring from an archive
> made using <application>pg_dumpall</application>.
> + Note: <option>--globals-only</option> cannot be used with
> <option>--exit-on-error</option>,
> + <option>--single-transaction</option>,
> <option>--clean</option>, or <option>--transaction-size</option>.
> + </para>
> + </listitem>
> + </varlistentry>
> +
> <option>--globals-only</option> cannot be used with --data-only,
> --schema-only, --statistics-only, --statistics.
> We should also mention that.
Fixed
>
> In doc/src/sgml/ref/pg_restore.sgml
> "when restoring from an archive made using pg_dumpall."
> It would be better using
> "when restoring from a non-plain-text archive made using pg_dumpall."
> that would be aligned with pg_dumpall.sgml.
>
> <varlistentry>
> <term><option>-g</option></term>
> <term><option>--globals-only</option></term>
> <listitem>
> <para>
> Dump only global objects (roles and tablespaces), no databases.
> + Note: <option>--globals-only</option> cannot be used with
> + <option>--clean</option> with non-text dump format.
> </para>
> Elsewhere, we use the term “non-plain-text,” so we should use
> “non-plain-text” here as well instead of “non-text,” for consistency.
>
> In doc/src/sgml/ref/pg_restore.sgml, We did not mention that many
> options cannot be used with pg_restore when performing a
> non-plain-text restore.
> Like:
> "-l/--list"
> "-L/--use-list"
> "--strict-names"
> "--no-schema"
> "-a/--data-only"
> "--statistics-only"
> --section does not include "--pre-data"
Fixed, but ...
What about options like these?:
n/--schema
N/--exclude-schema
t/--table
T/--trigger
I/--index
P/--function
-filter
We're not currently doing anything about those, but do they make sense
when restoring a pg_dumpall archive?
>
> pg_restore --clean --format=directory will produce DROP DATABASE will
> process global objects,
> it will also produce DROP DATABASE when processing each individual database.
> To prevent errors during a subsequent restore, we can require
> pg_restore --clean option must be used together with --if-exists when
> restoring a non-plain-text dump.
We could. Or we could just turn it on (and document that it will be
turned on) in this case. I'd rather not force people to use lots of flags.
Patch attached with the above noted fixes. It also adds a header comment
to map.dat and has pg_restore ignore comment lines (anything that
doesn't begin with a digit).
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
Attachments:
[text/x-patch] v18-21022026-Non-text-modes-for-pg_dumpall-correspondingly-change.patch (105.4K, 3-v18-21022026-Non-text-modes-for-pg_dumpall-correspondingly-change.patch)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 8834b7ec141..49e5c99b09e 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,16 +139,93 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ This option can only be omitted when <option>--format</option> is plain.
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files,
+ <filename>toc.glo</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data (roles and tablespaces) in custom format. The second
+ contains a mapping between database OIDs and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ the <filename>databases</filename> subdirectory, named using the database's OID.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ See <xref linkend="app-pgdump"/> for details on how the
+ various non-plain-text archive formats work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
<listitem>
<para>
Dump only global objects (roles and tablespaces), no databases.
+ Note: <option>--globals-only</option> cannot be used with
+ <option>--clean</option> with non-text dump format.
</para>
</listitem>
</varlistentry>
@@ -936,13 +1021,21 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
<refsect1 id="app-pg-dumpall-ex">
<title>Examples</title>
<para>
- To dump all databases:
-
+ To dump all databases in plain text format (the default):
<screen>
<prompt>$</prompt> <userinput>pg_dumpall > db.out</userinput>
</screen>
</para>
+ <para>
+ To dump all databases using other formats:
+<screen>
+<prompt>$</prompt> <userinput>pg_dumpall --format=directory -f db.out</userinput>
+<prompt>$</prompt> <userinput>pg_dumpall --format=custom -f db.out</userinput>
+<prompt>$</prompt> <userinput>pg_dumpall --format=tar -f db.out</userinput>
+</screen>
+ </para>
+
<para>
To restore database(s) from this file, you can use:
<screen>
@@ -956,6 +1049,16 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
the script will attempt to drop other databases immediately, and that
will fail for the database you are connected to.
</para>
+
+ <para>
+ If the dump was taken in a non-plain-text format, use
+ <application>pg_restore</application> to restore the databases:
+<screen>
+<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
+</screen>
+ This will restore all databases. To restore only some databases, use
+ the <option>--exclude-database</option> option to skip those not wanted.
+ </para>
</refsect1>
<refsect1>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 420a308a7c7..3d5531cfa96 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,14 +54,34 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
+ <para>
+ A non-plain-text archive made using <application>pg_dumpall</application>
+ is a directory containing a <filename>toc.glo</filename> file with global
+ objects (roles and tablespaces), a <filename>map.dat</filename> file
+ listing the databases, and a subdirectory for each database containing
+ its archive. When restoring such an archive,
+ <application>pg_restore</application> first restores global objects from
+ <filename>toc.glo</filename>, then processes each database listed in
+ <filename>map.dat</filename>. Lines in <filename>map.dat</filename> can
+ be commented out with <literal>#</literal> to skip restoring specific
+ databases.
+ </para>
+
<para>
Obviously, <application>pg_restore</application> cannot restore information
that is not present in the archive file. For instance, if the
@@ -152,6 +174,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from a non-plain-text archive made using <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +271,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from a non-plain-text archive made using <application>pg_dumpall</application>.
+ Note: <option>--globals-only</option> cannot be used with
+ <option>--data-only</option>,
+ <option>--schema-only</option>,
+ <option>--statistics-only</option>,
+ <option>--statistics</option>,
+ <option>--exit-on-error</option>,
+ <option>--single-transaction</option>,
+ <option>--clean</option>, or
+ <option>--transaction-size</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -581,6 +627,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from a non-plain-text archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
@@ -1125,6 +1193,14 @@ CREATE DATABASE foo WITH TEMPLATE template0;
</para>
</listitem>
+ <listitem>
+ <para>
+ When restoring from a non-plain-text archive made using
+ <application>pg_dumpall</application>, the <option>--section</option>
+ option may be used, but must include <option>pre-data</option>.
+ </para>
+ </listitem>
+
</itemizedlist>
</para>
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 79bd5036841..7c9a475963b 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 56cb2c1f32d..29fa21a1cc0 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 2f8d9799c30..fda912ba0a9 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -313,7 +313,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 7afcc0859c8..faa08cedaa5 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,15 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ /* Skip if no-tablespace is given. */
+ if (ropt->noTablespace && te && te->desc &&
+ (strcmp(te->desc, "TABLESPACE") == 0))
+ continue;
+
+ /* Skip DROP DATABASE/ROLES/TABLESPACE if we didn't specify --clean */
+ if (!ropt->dropSchema && te && te->tag && (strcmp(te->tag, "DROP_GLOBAL") == 0))
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1330,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1691,11 +1705,15 @@ archprintf(Archive *AH, const char *fmt,...)
/*******************************
* Stuff below here should be 'private' to the archiver routines
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
*******************************/
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1733,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
@@ -2391,7 +2409,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
/* initialize for backwards compatible string processing */
AH->public.encoding = 0; /* PG_SQL_ASCII */
- AH->public.std_strings = false;
+ AH->public.std_strings = true;
/* sql error handling */
AH->public.exit_on_error = true;
@@ -3027,6 +3045,16 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
return 0;
}
+ /*
+ * Global object TOC entries (e.g., ROLEs or TABLESPACEs) must not be
+ * ignored.
+ */
+ if (strcmp(te->desc, "ROLE") == 0 ||
+ strcmp(te->desc, "ROLE PROPERTIES") == 0 ||
+ strcmp(te->desc, "TABLESPACE") == 0 ||
+ strcmp(te->desc, "DROP_GLOBAL") == 0)
+ return REQ_SCHEMA;
+
/*
* Process exclusions that affect certain classes of TOC entries.
*/
@@ -3062,6 +3090,14 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
if (ropt->no_subscriptions &&
strncmp(te->tag, "SUBSCRIPTION", strlen("SUBSCRIPTION")) == 0)
return 0;
+
+ /*
+ * Global object TOC entries (e.g., ROLEs or TABLESPACEs) are treated as
+ * REQ_SCHEMA; follow this convention for their associated comments.
+ */
+ if (strncmp(te->desc, "ROLE", strlen("ROLE")) == 0 ||
+ strncmp(te->desc, "TABLESPACE", strlen("TABLESPACE")) == 0)
+ return REQ_SCHEMA;
}
/*
@@ -3091,6 +3127,14 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
if (ropt->no_subscriptions &&
strncmp(te->tag, "SUBSCRIPTION", strlen("SUBSCRIPTION")) == 0)
return 0;
+
+ /*
+ * Global object TOC entries (e.g., ROLEs or TABLESPACEs) are treated as
+ * REQ_SCHEMA; follow this convention for their associated SECURITY LABELs.
+ */
+ if (strncmp(te->tag, "ROLE", strlen("ROLE")) == 0 ||
+ strncmp(te->tag, "TABLESPACE", strlen("TABLESPACE")) == 0)
+ return REQ_SCHEMA;
}
/* If it's a subscription, maybe ignore it */
@@ -3865,6 +3909,9 @@ _getObjectDescription(PQExpBuffer buf, const TocEntry *te)
else if (strcmp(type, "CAST") == 0 ||
strcmp(type, "CHECK CONSTRAINT") == 0 ||
strcmp(type, "CONSTRAINT") == 0 ||
+ strcmp(type, "DROP_GLOBAL") == 0 ||
+ strcmp(type, "ROLE PROPERTIES") == 0 ||
+ strcmp(type, "ROLE") == 0 ||
strcmp(type, "DATABASE PROPERTIES") == 0 ||
strcmp(type, "DEFAULT") == 0 ||
strcmp(type, "FK CONSTRAINT") == 0 ||
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 49598304335..d3d0cac98df 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 98389d2034c..5a114d09185 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -1,13 +1,20 @@
/*-------------------------------------------------------------------------
*
* pg_dumpall.c
+ * pg_dumpall dumps all databases and global objects (roles and
+ * tablespaces) from a PostgreSQL cluster.
+ *
+ * For text format output, globals are written directly and pg_dump is
+ * invoked for each database, with all output going to stdout or a file.
+ *
+ * For non-text formats (custom, directory, tar), a directory is created
+ * containing a toc.glo file with global objects, a map.dat file mapping
+ * database OIDs to names, and a databases/ subdirectory with individual
+ * pg_dump archives for each database.
*
* Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
- * pg_dumpall forces all pg_dump output to be text, since it also outputs
- * text into the same output stream.
- *
* src/bin/pg_dump/pg_dumpall.c
*
*-------------------------------------------------------------------------
@@ -30,6 +37,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -67,7 +75,7 @@ static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
static void dumpDatabases(PGconn *conn);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts, char *dbfile);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,6 +84,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -123,6 +133,11 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static ArchiveFormat archDumpFormat = archNull;
+static const CatalogId nilCatalogId = {0, 0};
int
main(int argc, char *argv[])
@@ -148,6 +163,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +213,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *format_name = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -207,6 +224,7 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -244,8 +262,9 @@ main(int argc, char *argv[])
}
pgdumpopts = createPQExpBuffer();
+ InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -273,7 +292,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ format_name = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -313,6 +334,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -434,6 +456,32 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(format_name);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option %s=d|c|t requires option %s",
+ "-F/--format", "-f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
+ /* restrict-key is only supported with --format=plain */
+ if (archDumpFormat != archNull && restrict_key)
+ pg_fatal("option %s can only be used with %s=plain",
+ "--restrict-key", "--format");
+
+ /* --clean and -g/--globals-only cannot be used together in non-text dump */
+ if (archDumpFormat != archNull && output_clean && globals_only)
+ pg_fatal("options %s and %s cannot be used together in non-text dump",
+ "--clean", "-g/--globals-only");
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -494,6 +542,27 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -543,19 +612,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -593,35 +649,111 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
- if (verbose)
- dumpTimestamp("Started on");
-
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
-
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
-
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
-
- /* Replicate encoding and standard_conforming_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
+ /* create a archive file for global commands. */
+ if (archDumpFormat != archNull)
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ char global_path[MAXPGPATH];
+ const char *encname;
+
+ /* Set file path for global sql commands. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", filename);
+
+ /* Open the output file */
+ fout = CreateArchive(global_path, archCustom, compression_spec,
+ dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC);
+
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
+
+ ((ArchiveHandle *) fout)->connection = conn;
+ ((ArchiveHandle *) fout)->public.numWorkers = 1;
+
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release
+ * of our own major version. (See also version check in
+ * pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
+
+ /* Dump default_transaction_read_only. */
+ appendPQExpBufferStr(qry, "SET default_transaction_read_only = off;\n\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "default_transaction_read_only",
+ .description = "default_transaction_read_only",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ resetPQExpBuffer(qry);
+
+ /* Put the correct encoding into the archive */
+ encname = pg_encoding_to_char(encoding);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "client_encoding",
+ .description = "client_encoding",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ resetPQExpBuffer(qry);
+
+ /* Put the correct escape string behavior into the archive. */
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = 'on';\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "standard_conforming_strings",
+ .description = "standard_conforming_strings",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ destroyPQExpBuffer(qry);
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ if (verbose)
+ dumpTimestamp("Started on");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and standard_conforming_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = on;\n");
- fprintf(OPF, "\n");
+ fprintf(OPF, "SET standard_conforming_strings = on;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -630,8 +762,14 @@ main(int argc, char *argv[])
* dependency analysis because databases never depend on each other,
* and tablespaces never depend on each other. Roles could have
* grants to each other, but DROP ROLE will clean those up silently.
+ *
+ * For non-text formats, pg_dumpall unconditionally process --clean
+ * option. In contrast, pg_restore only applies it if the user
+ * explicitly provides the flag. This discrepancy resolves corner cases
+ * where pg_restore requires cleanup instructions that may be missing
+ * from a standard pg_dumpall output.
*/
- if (output_clean)
+ if (output_clean || archDumpFormat != archNull)
{
if (!globals_only && !roles_only && !tablespaces_only)
dropDBs(conn);
@@ -665,28 +803,45 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump
+ * will handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
dumpDatabases(conn);
- PQfinish(conn);
+ if (archDumpFormat == archNull)
+ {
+ PQfinish(conn);
+
+ if (verbose)
+ dumpTimestamp("Completed on");
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (verbose)
- dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+ if (filename)
+ {
+ fclose(OPF);
- if (filename)
+ /* sync the resulting file, errors are not fatal */
+ if (dosync && (archDumpFormat == archNull))
+ (void) fsync_fname(filename, false);
+ }
+ }
+ else
{
- fclose(OPF);
+ RestoreOptions *ropt;
+
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
- /* sync the resulting file, errors are not fatal */
- if (dosync)
- (void) fsync_fname(filename, false);
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
}
exit_nicely(0);
@@ -696,12 +851,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -796,24 +953,45 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
const char *rolename;
+ PQExpBuffer delQry = createPQExpBuffer();
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(rolename));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(rolename));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP ROLE IF EXISTS %s;\n",
+ fmtId(rolename));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "DROP_GLOBAL",
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -823,6 +1001,8 @@ static void
dumpRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer comment_buf = createPQExpBuffer();
+ PQExpBuffer seclabel_buf = createPQExpBuffer();
PGresult *res;
int i_oid,
i_rolname,
@@ -894,7 +1074,7 @@ dumpRoles(PGconn *conn)
i_rolcomment = PQfnumber(res, "rolcomment");
i_is_current_user = PQfnumber(res, "is_current_user");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -913,6 +1093,8 @@ dumpRoles(PGconn *conn)
}
resetPQExpBuffer(buf);
+ resetPQExpBuffer(comment_buf);
+ resetPQExpBuffer(seclabel_buf);
if (binary_upgrade)
{
@@ -989,17 +1171,53 @@ dumpRoles(PGconn *conn)
if (!no_comments && !PQgetisnull(res, i, i_rolcomment))
{
- appendPQExpBuffer(buf, "COMMENT ON ROLE %s IS ", fmtId(rolename));
- appendStringLiteralConn(buf, PQgetvalue(res, i, i_rolcomment), conn);
- appendPQExpBufferStr(buf, ";\n");
+ appendPQExpBuffer(comment_buf, "COMMENT ON ROLE %s IS ", fmtId(rolename));
+ appendStringLiteralConn(comment_buf, PQgetvalue(res, i, i_rolcomment), conn);
+ appendPQExpBufferStr(comment_buf, ";\n");
}
if (!no_security_labels)
buildShSecLabels(conn, "pg_authid", auth_oid,
"ROLE", rolename,
- buf);
+ seclabel_buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ {
+ fprintf(OPF, "%s", buf->data);
+ fprintf(OPF, "%s", comment_buf->data);
+
+ if (seclabel_buf->data[0] != '\0')
+ fprintf(OPF, "%s", seclabel_buf->data);
+ }
+ else
+ {
+ char *tag = psprintf("%s %s", "ROLE", fmtId(rolename));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "ROLE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ if (comment_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "COMMENT",
+ .section = SECTION_PRE_DATA,
+ .createStmt = comment_buf->data));
+
+ if (seclabel_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "SECURITY LABEL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = seclabel_buf->data));
+ }
}
/*
@@ -1007,7 +1225,7 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "\n--\n-- User Configurations\n--\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1015,9 +1233,12 @@ dumpRoles(PGconn *conn)
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
+ destroyPQExpBuffer(comment_buf);
+ destroyPQExpBuffer(seclabel_buf);
}
@@ -1031,6 +1252,7 @@ static void
dumpRoleMembership(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer querybuf = createPQExpBuffer();
PQExpBuffer optbuf = createPQExpBuffer();
PGresult *res;
int start = 0,
@@ -1093,7 +1315,7 @@ dumpRoleMembership(PGconn *conn)
i_inherit_option = PQfnumber(res, "inherit_option");
i_set_option = PQfnumber(res, "set_option");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Role memberships\n--\n\n");
/*
@@ -1229,8 +1451,9 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ resetPQExpBuffer(querybuf);
+ appendPQExpBuffer(querybuf, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(querybuf, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1251,10 +1474,21 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(querybuf, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(querybuf, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(querybuf, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", querybuf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = fmtId(role),
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = querybuf->data));
}
}
@@ -1265,8 +1499,11 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
+ destroyPQExpBuffer(querybuf);
+ destroyPQExpBuffer(optbuf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1293,7 +1530,7 @@ dumpRoleGUCPrivs(PGconn *conn)
"FROM pg_catalog.pg_parameter_acl "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1318,14 +1555,25 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = fmtId(parowner),
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1347,21 +1595,41 @@ dropTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
+ PQExpBuffer delQry = createPQExpBuffer();
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(spcname));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(spcname));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP TABLESPACE IF EXISTS %s;\n",
+ fmtId(spcname));
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "DROP_GLOBAL",
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1371,6 +1639,8 @@ static void
dumpTablespaces(PGconn *conn)
{
PGresult *res;
+ PQExpBuffer comment_buf = createPQExpBuffer();
+ PQExpBuffer seclabel_buf = createPQExpBuffer();
int i;
/*
@@ -1387,7 +1657,7 @@ dumpTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1406,6 +1676,9 @@ dumpTablespaces(PGconn *conn)
/* needed for buildACLCommands() */
fspcname = pg_strdup(fmtId(spcname));
+ resetPQExpBuffer(comment_buf);
+ resetPQExpBuffer(seclabel_buf);
+
if (binary_upgrade)
{
appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_tablespace oid\n");
@@ -1447,24 +1720,67 @@ dumpTablespaces(PGconn *conn)
if (!no_comments && spccomment && spccomment[0] != '\0')
{
- appendPQExpBuffer(buf, "COMMENT ON TABLESPACE %s IS ", fspcname);
- appendStringLiteralConn(buf, spccomment, conn);
- appendPQExpBufferStr(buf, ";\n");
+ appendPQExpBuffer(comment_buf, "COMMENT ON TABLESPACE %s IS ", fspcname);
+ appendStringLiteralConn(comment_buf, spccomment, conn);
+ appendPQExpBufferStr(comment_buf, ";\n");
}
if (!no_security_labels)
buildShSecLabels(conn, "pg_tablespace", spcoid,
"TABLESPACE", spcname,
- buf);
+ seclabel_buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ {
+ fprintf(OPF, "%s", buf->data);
+
+ if (comment_buf->data[0] != '\0')
+ fprintf(OPF, "%s", comment_buf->data);
+
+ if (seclabel_buf->data[0] != '\0')
+ fprintf(OPF, "%s", seclabel_buf->data);
+ }
+ else
+ {
+ char *tag = psprintf("%s %s", "TABLESPACE", fmtId(fspcname));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "TABLESPACE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+
+ if (comment_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "COMMENT",
+ .section = SECTION_PRE_DATA,
+ .createStmt = comment_buf->data));
+
+ if (seclabel_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "SECURITY LABEL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = seclabel_buf->data));
+ }
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ destroyPQExpBuffer(comment_buf);
+ destroyPQExpBuffer(seclabel_buf);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1487,12 +1803,13 @@ dropDBs(PGconn *conn)
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY datname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Skip "postgres" and "template1"; dumpDatabases() will deal with
@@ -1503,15 +1820,35 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(dbname));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(dbname));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP DATABASE IF EXISTS %s;\n",
+ fmtId(dbname));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "DROP_GLOBAL",
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1533,7 +1870,7 @@ dumpUserConfig(PGconn *conn, const char *username)
res = executeQuery(conn, buf->data);
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
{
char *sanitized;
@@ -1548,7 +1885,17 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = username,
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
}
PQclear(res);
@@ -1618,6 +1965,9 @@ dumpDatabases(PGconn *conn)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1631,19 +1981,59 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+ char *map_preamble[] = {
+ "#################################################################",
+ "# map.dat",
+ "#",
+ "# This file maps oids to database names",
+ "#",
+ "# pg_restore will restore all the databases listed here, unless",
+ "# otherwise excluded. You can also inhibit restoration of a",
+ "# database by removing the line or commenting out the line with"
+ "# a # mark.",
+ "#################################################################",
+ NULL
+ };
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ for (char **line = map_preamble; *line; line++)
+ fprintf(map_file, "%s\n", *line);
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1660,7 +2050,10 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
free(sanitized);
/*
@@ -1675,24 +2068,40 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
- else
- {
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
fprintf(OPF, "\\connect %s\n\n", dbname);
- }
+ else
+ create_opts = "";
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1701,6 +2110,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1710,7 +2123,7 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1719,17 +2132,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s -f %s %s", pg_dump_bin,
+ pgdumpopts->data, dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1874,3 +2306,47 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+/*
+ * createDumpId
+ *
+ * Return the next dumpId.
+ */
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 84b8d410c9e..37f30e65421 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,12 +41,16 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -54,18 +58,41 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data);
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts);
+
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +116,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +170,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +199,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +226,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global sql commands. */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -321,6 +353,10 @@ main(int argc, char **argv)
opts->restrict_key = pg_strdup(optarg);
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -347,6 +383,14 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option %s cannot be used together with %s",
+ "--exclude-database", "-g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -420,6 +464,31 @@ main(int argc, char **argv)
pg_fatal("options %s and %s cannot be used together",
"-1/--single-transaction", "--transaction-size");
+ if (opts->single_txn && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--single-transaction", "-g/--globals-only");
+
+ if (opts->txn_size && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--transaction-size", "-g/--globals-only");
+
+ if (opts->exit_on_error && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--exit-on-error", "-g/--globals-only");
+
+ if (data_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-a/--data-only", "-g/--globals-only");
+ if (schema_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-s/--schema-only", "-g/--globals-only");
+ if (statistics_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "--statistics-only", "-g/--globals-only");
+ if (with_statistics && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "--statistics", "-g/--globals-only");
+
/*
* -C is not compatible with -1, because we can't create a database inside
* a transaction block.
@@ -485,6 +554,173 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.glo file is present, then restore all the databases from
+ * map.dat, but skip restoring those matching --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "toc.glo")))
+ {
+ char global_path[MAXPGPATH];
+ RestoreOptions *tmpopts = pg_malloc0_object(RestoreOptions);
+
+ opts->format = archUnknown;
+
+ memcpy(tmpopts, opts, sizeof(RestoreOptions));
+
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-l/--list");
+ if (opts->tocFile)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-L/--use-list");
+
+ if (opts->strict_names)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--strict-names");
+ if (globals_only && opts->dropSchema)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--clean", "-g/--globals-only");
+
+ if (no_schema)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--no-schema");
+
+ if (data_only)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-a/--data-only");
+
+ if (statistics_only)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--statistics-only");
+
+ if (!(opts->dumpSections & DUMP_PRE_DATA))
+ pg_fatal("option %s cannot exclude %s when restoring a pg_dumpall archive",
+ "--section", "--pre-data");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option %s must be specified when restoring an archive created by pg_dumpall",
+ "-C/--create");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Always restore global objects, even if --exclude-database results in
+ * zero databases to process. If 'globals-only' is set, exit
+ * immediately.
+ */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+
+ n_errors = restore_global_objects(global_path, tmpopts);
+
+ if (globals_only)
+ pg_log_info("database restoring skipped because option %s was specified",
+ "-g/--globals-only");
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else
+ {
+ if (db_exclude_patterns.head != NULL)
+ {
+ simple_string_list_destroy(&db_exclude_patterns);
+ pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
+ "--exclude-database");
+ }
+
+ if (globals_only)
+ pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
+ "-g/--globals-only");
+
+ /* Process if toc.glo file does not exist. */
+ n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ */
+static int
+restore_global_objects(const char *inputFileSpec, RestoreOptions *opts)
+{
+ Archive *AH;
+ int nerror = 0;
+
+ /* Set format as custom so that toc.glo file can be read. */
+ opts->format = archCustom;
+ opts->txn_size = 0;
+
+ AH = OpenArchive(inputFileSpec, opts->format);
+
+ SetArchiveOptions(AH, NULL, opts);
+
+ on_exit_close_archive(AH);
+
+ /* Let the archiver know how noisy to be */
+ AH->verbose = opts->verbose;
+
+ /* Don't output TOC entry comments when restoring globals */
+ ((ArchiveHandle *) AH)->noTocComments = 1;
+
+ AH->exit_on_error = false;
+
+ /* Parallel execution is not supported for global object restoration. */
+ AH->numWorkers = 1;
+
+ ProcessArchiveRestoreOptions(AH);
+ RestoreArchive(AH, false);
+
+ nerror = AH->n_errors;
+
+ /* AH may be freed in CloseArchive? */
+ CloseArchive(AH);
+
+ return nerror;
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -492,9 +728,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -514,25 +756,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -550,6 +788,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -566,6 +805,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -601,8 +841,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -707,3 +947,425 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PQExpBuffer db_lit;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+ db_lit = createPQExpBuffer();
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+
+ resetPQExpBuffer(query);
+ resetPQExpBuffer(db_lit);
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if (PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches --exclude-database pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ count_db++;
+ }
+
+ destroyPQExpBuffer(query);
+ destroyPQExpBuffer(db_lit);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+ int len;
+
+
+ /*
+ * If there is no map.dat file in dump, then return from here as there is
+ * no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ len = strlen(dumpdirpath);
+
+ /* Trim slash from directory name. */
+ while (len > 1 && dumpdirpath[len - 1] == '/')
+ {
+ dumpdirpath[len - 1] = '\0';
+ len--;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* look for the dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+
+ /* ignore lines that don't begin with a digit */
+ if (p == linebuf.data)
+ continue;
+
+ if (*p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Strip trailing newline */
+ if (namelen > 0 && dbname[namelen - 1] == '\n')
+ dbname[--namelen] = '\0';
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen < 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbidname->str, db_oid, map_file_path);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ pfree(linebuf.data);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total = 0;
+ char *connected_db = NULL;
+ PGconn *conn = NULL;
+ RestoreOptions *original_opts = pg_malloc0_object(RestoreOptions);
+ RestoreOptions *tmpopts = pg_malloc0_object(RestoreOptions);
+
+ memcpy(original_opts, opts, sizeof(RestoreOptions));
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile((char*) inputFileSpec, &dbname_oid_list);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, then connect to the database to process
+ * it.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (!conn)
+ {
+ pg_log_error("could not connect to databases \"postgres\" or \"template1\"\n"
+ "Please specify an alternative database.");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+ }
+ }
+ }
+
+ /*
+ * filter the db list according to the exclude patterns
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Close the db connection as we are done with exclude-database patterns. */
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ pg_free(original_opts);
+ pg_free(tmpopts);
+ return 0;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * Since pg_backup_archiver.c may modify RestoreOptions during the
+ * previous restore, we must provide a fresh copy of the original
+ * "opts" for each call to restore_one_database.
+ */
+ memcpy(tmpopts, original_opts, sizeof(RestoreOptions));
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (tmpopts->cparams.override_dbname)
+ {
+ pfree(tmpopts->cparams.override_dbname);
+ tmpopts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (tmpopts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, tmpopts->cparams.pghost,
+ tmpopts->cparams.pgport, tmpopts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ tmpopts->createDB = 0;
+ tmpopts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* We'll have to create it */
+ tmpopts->createDB = 1;
+ tmpopts->cparams.dbname = connected_db;
+ }
+ }
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true);
+
+ n_errors_total += n_errors;
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ pg_free(original_opts);
+ pg_free(tmpopts);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
index ab9310eb42b..a895bc314b0 100644
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -244,4 +244,59 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--restrict-key=uu', '-f dumpfile' ],
+ qr/\Qpg_dumpall: error: option --restrict-key can only be used with --format=plain\E/,
+ 'pg_dumpall: --restrict-key can only be used with plain dump format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--globals-only', '--clean', '-f', 'dumpfile' ],
+ qr/\Qpg_dumpall: error: options --clean and -g\/--globals-only cannot be used together in non-text dump\E/,
+ 'pg_dumpall: --clean and -g/--globals-only cannot be used together in non-text dump');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd' ],
+ qr/\Qpg_dumpall: error: option -F\/--format=d|c|t requires option -f\/--file\E/,
+ 'pg_dumpall: non-plain format requires --file option');
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--data-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options -a\/--data-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options -a/--data-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--schema-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options -s\/--schema-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options -s/--schema-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--statistics-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options --statistics-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options --statistics-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option --exclude-database can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --exclude-database is used in pg_restore with dump of pg_dump'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--globals-only', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option -g\/--globals-only can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --globals-only is used in pg_restore with the dump of pg_dump'
+);
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100644
index 00000000000..90fdd2e91b9
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,622 @@
+# Copyright (c) 2021-2026, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# escape tablespace locations on Windows.
+my $tablespace2_orig = $tablespace2;
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2_orig';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# report an error when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# report an error when \l/--list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# report an error when -L/--use-list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--use-list' => 'use',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -L\/--use-list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When -L/--use-list is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --strict-names option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--strict-names',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --strict-names cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --strict-names is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --clean and -g/--globals-only are used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--clean',
+ '--globals-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --clean and -g\/--globals-only cannot be used together when restoring an archive created by pg_dumpall\E/,
+ 'When --clean and -g/--globals-only are used in pg_restore with dump of pg_dumpall'
+);
+
+# report an error when non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+# report an error when --no-schema is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--no-schema',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --no-schema cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --no-schema is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --data-only is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--data-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -a\/--data-only cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --data-only is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --statistics-only is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--statistics-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --statistics-only cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --statistics-only is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --section excludes pre-data with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--section' => 'post-data',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --section cannot exclude --pre-data when restoring a pg_dumpall archive\E/,
+ 'When --section=post-data is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --globals-only and --data-only are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--data-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options -a\/--data-only and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --data-only are used together');
+
+# report an error when --globals-only and --schema-only are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--schema-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options -s\/--schema-only and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --schema-only are used together');
+
+# report an error when --globals-only and --statistics-only are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--statistics-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --statistics-only and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --statistics-only are used together');
+
+# report an error when --globals-only and --statistics are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--statistics',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --statistics and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --statistics are used together');
+
+# report an error when --globals-only and --exit-on-error are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--exit-on-error',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --exit-on-error and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --exit-on-error are used together');
+
+# report an error when --globals-only and --single-transaction are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--single-transaction',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --single-transaction and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --single-transaction are used together');
+
+# report an error when --globals-only and --transaction-size are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--transaction-size' => '100',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --transaction-size and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --transaction-size are used together');
+
+# verify map.dat preamble exists
+my $map_dat_content = slurp_file("$tempdir/format_directory/map.dat");
+like(
+ $map_dat_content,
+ qr/^# map\.dat\n.*# This file maps oids to database names/ms,
+ 'map.dat contains expected preamble');
+
+# verify commenting out a line in map.dat skips that database
+$node->safe_psql($run_db, 'CREATE DATABASE comment_test_db;
+\c comment_test_db
+CREATE TABLE comment_test_table (id int);');
+
+$node->command_ok(
+ [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/comment_test",
+ ],
+ 'pg_dumpall for comment test');
+
+# Modify map.dat to comment out the comment_test_db entry
+my $map_content = slurp_file("$tempdir/comment_test/map.dat");
+$map_content =~ s/^(\d+ comment_test_db)$/# $1/m;
+open(my $fh, '>', "$tempdir/comment_test/map.dat")
+ or die "Cannot open map.dat: $!";
+print $fh $map_content;
+close($fh);
+
+# Create a target node and restore - commented db should be skipped
+my $target_comment = PostgreSQL::Test::Cluster->new("target_comment");
+$target_comment->init;
+$target_comment->start;
+
+$node->command_ok(
+ [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/comment_test_restore.sql",
+ '--host', $target_comment->host,
+ '--port', $target_comment->port,
+ "$tempdir/comment_test",
+ ],
+ 'pg_restore with commented out database in map.dat');
+
+my $restore_output = slurp_file("$tempdir/comment_test_restore.sql");
+unlike(
+ $restore_output,
+ qr/CREATE DATABASE comment_test_db/,
+ 'commented out database in map.dat is not restored');
+
+$node->stop('fast');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..1a89ef94bec 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -600,6 +600,7 @@ CustomScanMethods
CustomScanState
CycleCtr
DBState
+DbOidName
DCHCacheEntry
DEADLOCK_INFO
DECountItem
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-23 08:04 jian he <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: jian he @ 2026-02-23 08:04 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On Sun, Feb 22, 2026 at 1:05 AM Andrew Dunstan <[email protected]> wrote:
>
> What about options like these?:
>
> n/--schema
> N/--exclude-schema
> t/--table
> T/--trigger
> I/--index
> P/--function
> -filter
>
> We're not currently doing anything about those, but do they make sense when restoring a pg_dumpall archive?
>
We should reject these options too, since these options do not make
sense for multiple databases, IMHO.
>
> pg_restore --clean --format=directory will produce DROP DATABASE will
> process global objects,
> it will also produce DROP DATABASE when processing each individual database.
> To prevent errors during a subsequent restore, we can require
> pg_restore --clean option must be used together with --if-exists when
> restoring a non-plain-text dump.
>
> We could. Or we could just turn it on (and document that it will be turned on) in this case. I'd rather not force people to use lots of flags.
>
Turning it on is OK for me.
The attached patch addresses the two issues described above.
--
jian
https://www.enterprisedb.com/
Attachments:
[application/octet-stream] v18-0001-misc-fix-for-v18.no-cfbot (5.3K, 2-v18-0001-misc-fix-for-v18.no-cfbot)
download
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-23 08:58 Mahendra Singh Thalor <[email protected]>
parent: jian he <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Mahendra Singh Thalor @ 2026-02-23 08:58 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On Mon, 23 Feb 2026 at 13:35, jian he <[email protected]> wrote:
>
> On Sun, Feb 22, 2026 at 1:05 AM Andrew Dunstan <[email protected]> wrote:
> >
> > What about options like these?:
> >
> > n/--schema
> > N/--exclude-schema
> > t/--table
> > T/--trigger
> > I/--index
> > P/--function
> > -filter
> >
> > We're not currently doing anything about those, but do they make sense when restoring a pg_dumpall archive?
We can keep these. If we use these options, then all databases will be
created(CREATE DATABASE) and based on n/N/t/T options, objects will be
restored.
Let say customers want to restore tables tb1, tb2, tb5 from the
cluster so only these tables will be restored even if they belong to
different-different databases.
> >
>
> We should reject these options too, since these options do not make
> sense for multiple databases, IMHO.
>
> >
> > pg_restore --clean --format=directory will produce DROP DATABASE will
> > process global objects,
> > it will also produce DROP DATABASE when processing each individual database.
> > To prevent errors during a subsequent restore, we can require
> > pg_restore --clean option must be used together with --if-exists when
> > restoring a non-plain-text dump.
> >
> > We could. Or we could just turn it on (and document that it will be turned on) in this case. I'd rather not force people to use lots of flags.
> >
>
> Turning it on is OK for me.
>
> The attached patch addresses the two issues described above.
>
>
> --
> jian
> https://www.enterprisedb.com/
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-23 20:34 Andrew Dunstan <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Andrew Dunstan @ 2026-02-23 20:34 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; jian he <[email protected]>; +Cc: tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On 2026-02-23 Mo 3:58 AM, Mahendra Singh Thalor wrote:
> On Mon, 23 Feb 2026 at 13:35, jian he <[email protected]> wrote:
>> On Sun, Feb 22, 2026 at 1:05 AM Andrew Dunstan <[email protected]> wrote:
>>> What about options like these?:
>>>
>>> n/--schema
>>> N/--exclude-schema
>>> t/--table
>>> T/--trigger
>>> I/--index
>>> P/--function
>>> -filter
>>>
>>> We're not currently doing anything about those, but do they make sense when restoring a pg_dumpall archive?
> We can keep these. If we use these options, then all databases will be
> created(CREATE DATABASE) and based on n/N/t/T options, objects will be
> restored.
> Let say customers want to restore tables tb1, tb2, tb5 from the
> cluster so only these tables will be restored even if they belong to
> different-different databases.
OK, I left that alone. We can revisit later if it bothers anyone.
>
>> We should reject these options too, since these options do not make
>> sense for multiple databases, IMHO.
>>
>>> pg_restore --clean --format=directory will produce DROP DATABASE will
>>> process global objects,
>>> it will also produce DROP DATABASE when processing each individual database.
>>> To prevent errors during a subsequent restore, we can require
>>> pg_restore --clean option must be used together with --if-exists when
>>> restoring a non-plain-text dump.
>>>
>>> We could. Or we could just turn it on (and document that it will be turned on) in this case. I'd rather not force people to use lots of flags.
>>>
>> Turning it on is OK for me.
>>
>> The attached patch addresses the two issues described above.
>>
>>
OK, here is my commit candidate patch.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
Attachments:
[text/x-patch] v19-0001-Add-non-text-output-formats-to-pg_dumpall.patch (111.7K, 2-v19-0001-Add-non-text-output-formats-to-pg_dumpall.patch)
download | inline diff:
From d115befd898ce373bd7dc747b792ee3461918040 Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email protected]>
Date: Mon, 23 Feb 2026 15:08:55 -0500
Subject: [PATCH v19] Add non-text output formats to pg_dumpall
pg_dumpall can now produce output in custom, directory, or tar formats
in addition to plain text SQL scripts. When using non-text formats,
pg_dumpall creates a directory containing:
- toc.glo: global data (roles and tablespaces) in custom format
- map.dat: mapping between database OIDs and names
- databases/: subdirectory with per-database archives named by OID
pg_restore is extended to handle these pg_dumpall archives, restoring
globals and then each database. The --globals-only and --no-globals
options control which parts are restored.
This enables parallel restore of pg_dumpall output and selective
restoration of individual databases from a cluster-wide backup.
Author: Mahendra Singh Thalor <[email protected]>
Co-Author: Andrew Dunstan <[email protected]>
Reviewed-By: Tushar Ahuja <[email protected]>
Reviewed-By: Jian He <[email protected]>
Reviewed-By: Vaibhav Dalvi <[email protected]>
Reviewed-By: Srinath Reddy <[email protected]>
Discussion: https://postgr.es/m/[email protected]
---
doc/src/sgml/ref/pg_dumpall.sgml | 113 +++-
doc/src/sgml/ref/pg_restore.sgml | 106 +++-
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 14 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 67 ++-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 800 ++++++++++++++++++++++-----
src/bin/pg_dump/pg_restore.c | 703 ++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 55 ++
src/bin/pg_dump/t/007_pg_dumpall.pl | 639 +++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
14 files changed, 2335 insertions(+), 171 deletions(-)
create mode 100644 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 8834b7ec141..49e5c99b09e 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,16 +139,93 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ This option can only be omitted when <option>--format</option> is plain.
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files,
+ <filename>toc.glo</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data (roles and tablespaces) in custom format. The second
+ contains a mapping between database OIDs and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ the <filename>databases</filename> subdirectory, named using the database's OID.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ See <xref linkend="app-pgdump"/> for details on how the
+ various non-plain-text archive formats work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
<listitem>
<para>
Dump only global objects (roles and tablespaces), no databases.
+ Note: <option>--globals-only</option> cannot be used with
+ <option>--clean</option> with non-text dump format.
</para>
</listitem>
</varlistentry>
@@ -936,13 +1021,21 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
<refsect1 id="app-pg-dumpall-ex">
<title>Examples</title>
<para>
- To dump all databases:
-
+ To dump all databases in plain text format (the default):
<screen>
<prompt>$</prompt> <userinput>pg_dumpall > db.out</userinput>
</screen>
</para>
+ <para>
+ To dump all databases using other formats:
+<screen>
+<prompt>$</prompt> <userinput>pg_dumpall --format=directory -f db.out</userinput>
+<prompt>$</prompt> <userinput>pg_dumpall --format=custom -f db.out</userinput>
+<prompt>$</prompt> <userinput>pg_dumpall --format=tar -f db.out</userinput>
+</screen>
+ </para>
+
<para>
To restore database(s) from this file, you can use:
<screen>
@@ -956,6 +1049,16 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
the script will attempt to drop other databases immediately, and that
will fail for the database you are connected to.
</para>
+
+ <para>
+ If the dump was taken in a non-plain-text format, use
+ <application>pg_restore</application> to restore the databases:
+<screen>
+<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
+</screen>
+ This will restore all databases. To restore only some databases, use
+ the <option>--exclude-database</option> option to skip those not wanted.
+ </para>
</refsect1>
<refsect1>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 420a308a7c7..ff50fc23539 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,14 +54,34 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
+ <para>
+ A non-plain-text archive made using <application>pg_dumpall</application>
+ is a directory containing a <filename>toc.glo</filename> file with global
+ objects (roles and tablespaces), a <filename>map.dat</filename> file
+ listing the databases, and a subdirectory for each database containing
+ its archive. When restoring such an archive,
+ <application>pg_restore</application> first restores global objects from
+ <filename>toc.glo</filename>, then processes each database listed in
+ <filename>map.dat</filename>. Lines in <filename>map.dat</filename> can
+ be commented out with <literal>#</literal> to skip restoring specific
+ databases.
+ </para>
+
<para>
Obviously, <application>pg_restore</application> cannot restore information
that is not present in the archive file. For instance, if the
@@ -130,6 +152,12 @@ PostgreSQL documentation
ignorable error messages will be reported,
unless <option>--if-exists</option> is also specified.
</para>
+ <para>
+ When restoring a <application>pg_dumpall</application> archive,
+ <option>--if-exists</option> is implied by <option>--clean</option>,
+ since global objects such as roles and tablespaces may not exist
+ in the target cluster.
+ </para>
</listitem>
</varlistentry>
@@ -152,6 +180,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from a non-plain-text archive made using <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +277,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from a non-plain-text archive made using <application>pg_dumpall</application>.
+ Note: <option>--globals-only</option> cannot be used with
+ <option>--data-only</option>,
+ <option>--schema-only</option>,
+ <option>--statistics-only</option>,
+ <option>--statistics</option>,
+ <option>--exit-on-error</option>,
+ <option>--single-transaction</option>,
+ <option>--clean</option>, or
+ <option>--transaction-size</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -581,6 +633,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from a non-plain-text archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
@@ -669,7 +743,9 @@ PostgreSQL documentation
in <option>--clean</option> mode. This suppresses <quote>does not
exist</quote> errors that might otherwise be reported. This
option is not valid unless <option>--clean</option> is also
- specified.
+ specified. This option is implied when restoring a
+ <application>pg_dumpall</application> archive with
+ <option>--clean</option>.
</para>
</listitem>
</varlistentry>
@@ -1125,6 +1201,14 @@ CREATE DATABASE foo WITH TEMPLATE template0;
</para>
</listitem>
+ <listitem>
+ <para>
+ When restoring from a non-plain-text archive made using
+ <application>pg_dumpall</application>, the <option>--section</option>
+ option may be used, but must include <option>pre-data</option>.
+ </para>
+ </listitem>
+
</itemizedlist>
</para>
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 79bd5036841..7c9a475963b 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 56cb2c1f32d..a28561fbd84 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,20 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * Update the archive handle in the on_exit callback registered by
+ * on_exit_close_archive(). When pg_restore processes a pg_dumpall archive
+ * containing multiple databases, each database is restored from a separate
+ * archive. After closing one archive and opening the next, we update the
+ * shutdown_info to reference the new archive handle so the cleanup callback
+ * will close the correct archive on exit.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 2f8d9799c30..fda912ba0a9 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -313,7 +313,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 7afcc0859c8..df8a69d3b79 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,19 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ /* Skip if no-tablespace is given. */
+ if (ropt->noTablespace && te && te->desc &&
+ (strcmp(te->desc, "TABLESPACE") == 0))
+ continue;
+
+ /*
+ * Skip DROP DATABASE/ROLES/TABLESPACE if we didn't specify
+ * --clean
+ */
+ if (!ropt->dropSchema && te && te->desc &&
+ strcmp(te->desc, "DROP_GLOBAL") == 0)
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1334,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1691,11 +1709,15 @@ archprintf(Archive *AH, const char *fmt,...)
/*******************************
* Stuff below here should be 'private' to the archiver routines
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
*******************************/
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1737,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
@@ -2391,7 +2413,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
/* initialize for backwards compatible string processing */
AH->public.encoding = 0; /* PG_SQL_ASCII */
- AH->public.std_strings = false;
+ AH->public.std_strings = true;
/* sql error handling */
AH->public.exit_on_error = true;
@@ -3027,6 +3049,16 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
return 0;
}
+ /*
+ * Global object TOC entries (e.g., ROLEs or TABLESPACEs) must not be
+ * ignored.
+ */
+ if (strcmp(te->desc, "ROLE") == 0 ||
+ strcmp(te->desc, "ROLE PROPERTIES") == 0 ||
+ strcmp(te->desc, "TABLESPACE") == 0 ||
+ strcmp(te->desc, "DROP_GLOBAL") == 0)
+ return REQ_SCHEMA;
+
/*
* Process exclusions that affect certain classes of TOC entries.
*/
@@ -3062,6 +3094,14 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
if (ropt->no_subscriptions &&
strncmp(te->tag, "SUBSCRIPTION", strlen("SUBSCRIPTION")) == 0)
return 0;
+
+ /*
+ * Comments on global objects (ROLEs or TABLESPACEs) should not be
+ * skipped, since global objects themselves are never skipped.
+ */
+ if (strncmp(te->tag, "ROLE", strlen("ROLE")) == 0 ||
+ strncmp(te->tag, "TABLESPACE", strlen("TABLESPACE")) == 0)
+ return REQ_SCHEMA;
}
/*
@@ -3091,6 +3131,14 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
if (ropt->no_subscriptions &&
strncmp(te->tag, "SUBSCRIPTION", strlen("SUBSCRIPTION")) == 0)
return 0;
+
+ /*
+ * Security labels on global objects (ROLEs or TABLESPACEs) should not
+ * be skipped, since global objects themselves are never skipped.
+ */
+ if (strncmp(te->tag, "ROLE", strlen("ROLE")) == 0 ||
+ strncmp(te->tag, "TABLESPACE", strlen("TABLESPACE")) == 0)
+ return REQ_SCHEMA;
}
/* If it's a subscription, maybe ignore it */
@@ -3865,6 +3913,9 @@ _getObjectDescription(PQExpBuffer buf, const TocEntry *te)
else if (strcmp(type, "CAST") == 0 ||
strcmp(type, "CHECK CONSTRAINT") == 0 ||
strcmp(type, "CONSTRAINT") == 0 ||
+ strcmp(type, "DROP_GLOBAL") == 0 ||
+ strcmp(type, "ROLE PROPERTIES") == 0 ||
+ strcmp(type, "ROLE") == 0 ||
strcmp(type, "DATABASE PROPERTIES") == 0 ||
strcmp(type, "DEFAULT") == 0 ||
strcmp(type, "FK CONSTRAINT") == 0 ||
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 450cec285b3..ce29627050f 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 98389d2034c..92d1214b421 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -1,13 +1,20 @@
/*-------------------------------------------------------------------------
*
* pg_dumpall.c
+ * pg_dumpall dumps all databases and global objects (roles and
+ * tablespaces) from a PostgreSQL cluster.
+ *
+ * For text format output, globals are written directly and pg_dump is
+ * invoked for each database, with all output going to stdout or a file.
+ *
+ * For non-text formats (custom, directory, tar), a directory is created
+ * containing a toc.glo file with global objects, a map.dat file mapping
+ * database OIDs to names, and a databases/ subdirectory with individual
+ * pg_dump archives for each database.
*
* Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
- * pg_dumpall forces all pg_dump output to be text, since it also outputs
- * text into the same output stream.
- *
* src/bin/pg_dump/pg_dumpall.c
*
*-------------------------------------------------------------------------
@@ -30,6 +37,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -67,15 +75,18 @@ static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
static void dumpDatabases(PGconn *conn);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts, char *dbfile);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
static void executeCommand(PGconn *conn, const char *query);
+static void check_for_invalid_global_names(PGconn *conn);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -123,6 +134,11 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static pg_compress_specification compression_spec = {0};
+static int dumpIdVal = 0;
+static ArchiveFormat archDumpFormat = archNull;
+static const CatalogId nilCatalogId = {0, 0};
int
main(int argc, char *argv[])
@@ -148,6 +164,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +214,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *format_name = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -207,6 +225,7 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -244,8 +263,9 @@ main(int argc, char *argv[])
}
pgdumpopts = createPQExpBuffer();
+ InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -273,7 +293,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ format_name = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -313,6 +335,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -434,6 +457,32 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(format_name);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option %s=d|c|t requires option %s",
+ "-F/--format", "-f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
+ /* restrict-key is only supported with --format=plain */
+ if (archDumpFormat != archNull && restrict_key)
+ pg_fatal("option %s can only be used with %s=plain",
+ "--restrict-key", "--format");
+
+ /* --clean and -g/--globals-only cannot be used together in non-text dump */
+ if (archDumpFormat != archNull && output_clean && globals_only)
+ pg_fatal("options %s and %s cannot be used together in non-text dump",
+ "--clean", "-g/--globals-only");
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -494,6 +543,27 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -543,19 +613,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -593,35 +650,120 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
- if (verbose)
- dumpTimestamp("Started on");
-
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
-
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
-
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
-
- /* Replicate encoding and standard_conforming_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = on;\n");
- fprintf(OPF, "\n");
+ /* create a archive file for global commands. */
+ if (archDumpFormat != archNull)
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ char global_path[MAXPGPATH];
+ const char *encname;
+
+ /*
+ * Check that no global object names contain newlines or carriage
+ * returns, which would break the map.dat file format. This is only
+ * needed for servers older than v19, which started prohibiting such
+ * names.
+ */
+ if (server_version < 190000)
+ check_for_invalid_global_names(conn);
+
+ /* Set file path for global sql commands. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", filename);
+
+ /* Open the output file */
+ fout = CreateArchive(global_path, archCustom, compression_spec,
+ dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC);
+
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
+
+ ((ArchiveHandle *) fout)->connection = conn;
+ ((ArchiveHandle *) fout)->public.numWorkers = 1;
+
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release
+ * of our own major version. (See also version check in
+ * pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
+
+ /* Dump default_transaction_read_only. */
+ appendPQExpBufferStr(qry, "SET default_transaction_read_only = off;\n\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "default_transaction_read_only",
+ .description = "default_transaction_read_only",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ resetPQExpBuffer(qry);
+
+ /* Put the correct encoding into the archive */
+ encname = pg_encoding_to_char(encoding);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "client_encoding",
+ .description = "client_encoding",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ resetPQExpBuffer(qry);
+
+ /* Put the correct escape string behavior into the archive. */
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = 'on';\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "standard_conforming_strings",
+ .description = "standard_conforming_strings",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ destroyPQExpBuffer(qry);
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ if (verbose)
+ dumpTimestamp("Started on");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the
+ * dump output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so
+ * whichever database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and standard_conforming_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = on;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -630,8 +772,14 @@ main(int argc, char *argv[])
* dependency analysis because databases never depend on each other,
* and tablespaces never depend on each other. Roles could have
* grants to each other, but DROP ROLE will clean those up silently.
+ *
+ * For non-text formats, pg_dumpall unconditionally process --clean
+ * option. In contrast, pg_restore only applies it if the user
+ * explicitly provides the flag. This discrepancy resolves corner
+ * cases where pg_restore requires cleanup instructions that may be
+ * missing from a standard pg_dumpall output.
*/
- if (output_clean)
+ if (output_clean || archDumpFormat != archNull)
{
if (!globals_only && !roles_only && !tablespaces_only)
dropDBs(conn);
@@ -665,28 +813,45 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump
+ * will handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
dumpDatabases(conn);
- PQfinish(conn);
+ if (archDumpFormat == archNull)
+ {
+ PQfinish(conn);
+
+ if (verbose)
+ dumpTimestamp("Completed on");
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (verbose)
- dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+ if (filename)
+ {
+ fclose(OPF);
- if (filename)
+ /* sync the resulting file, errors are not fatal */
+ if (dosync)
+ (void) fsync_fname(filename, false);
+ }
+ }
+ else
{
- fclose(OPF);
+ RestoreOptions *ropt;
+
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
- /* sync the resulting file, errors are not fatal */
- if (dosync)
- (void) fsync_fname(filename, false);
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
}
exit_nicely(0);
@@ -696,12 +861,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -796,24 +963,45 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
const char *rolename;
+ PQExpBuffer delQry = createPQExpBuffer();
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(rolename));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(rolename));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP ROLE IF EXISTS %s;\n",
+ fmtId(rolename));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = psprintf("ROLE %s", fmtId(rolename)),
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -823,6 +1011,8 @@ static void
dumpRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer comment_buf = createPQExpBuffer();
+ PQExpBuffer seclabel_buf = createPQExpBuffer();
PGresult *res;
int i_oid,
i_rolname,
@@ -894,7 +1084,7 @@ dumpRoles(PGconn *conn)
i_rolcomment = PQfnumber(res, "rolcomment");
i_is_current_user = PQfnumber(res, "is_current_user");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -913,6 +1103,8 @@ dumpRoles(PGconn *conn)
}
resetPQExpBuffer(buf);
+ resetPQExpBuffer(comment_buf);
+ resetPQExpBuffer(seclabel_buf);
if (binary_upgrade)
{
@@ -989,17 +1181,53 @@ dumpRoles(PGconn *conn)
if (!no_comments && !PQgetisnull(res, i, i_rolcomment))
{
- appendPQExpBuffer(buf, "COMMENT ON ROLE %s IS ", fmtId(rolename));
- appendStringLiteralConn(buf, PQgetvalue(res, i, i_rolcomment), conn);
- appendPQExpBufferStr(buf, ";\n");
+ appendPQExpBuffer(comment_buf, "COMMENT ON ROLE %s IS ", fmtId(rolename));
+ appendStringLiteralConn(comment_buf, PQgetvalue(res, i, i_rolcomment), conn);
+ appendPQExpBufferStr(comment_buf, ";\n");
}
if (!no_security_labels)
buildShSecLabels(conn, "pg_authid", auth_oid,
"ROLE", rolename,
- buf);
+ seclabel_buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ {
+ fprintf(OPF, "%s", buf->data);
+ fprintf(OPF, "%s", comment_buf->data);
+
+ if (seclabel_buf->data[0] != '\0')
+ fprintf(OPF, "%s", seclabel_buf->data);
+ }
+ else
+ {
+ char *tag = psprintf("ROLE %s", fmtId(rolename));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "ROLE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ if (comment_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "COMMENT",
+ .section = SECTION_PRE_DATA,
+ .createStmt = comment_buf->data));
+
+ if (seclabel_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "SECURITY LABEL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = seclabel_buf->data));
+ }
}
/*
@@ -1007,7 +1235,7 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "\n--\n-- User Configurations\n--\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1015,9 +1243,12 @@ dumpRoles(PGconn *conn)
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
+ destroyPQExpBuffer(comment_buf);
+ destroyPQExpBuffer(seclabel_buf);
}
@@ -1031,6 +1262,7 @@ static void
dumpRoleMembership(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer querybuf = createPQExpBuffer();
PQExpBuffer optbuf = createPQExpBuffer();
PGresult *res;
int start = 0,
@@ -1093,7 +1325,7 @@ dumpRoleMembership(PGconn *conn)
i_inherit_option = PQfnumber(res, "inherit_option");
i_set_option = PQfnumber(res, "set_option");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Role memberships\n--\n\n");
/*
@@ -1229,8 +1461,9 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ resetPQExpBuffer(querybuf);
+ appendPQExpBuffer(querybuf, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(querybuf, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1251,10 +1484,21 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(querybuf, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(querybuf, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(querybuf, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", querybuf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = psprintf("ROLE %s", fmtId(role)),
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = querybuf->data));
}
}
@@ -1265,8 +1509,11 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
+ destroyPQExpBuffer(querybuf);
+ destroyPQExpBuffer(optbuf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1293,7 +1540,7 @@ dumpRoleGUCPrivs(PGconn *conn)
"FROM pg_catalog.pg_parameter_acl "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1318,14 +1565,25 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = psprintf("ROLE %s", fmtId(parowner)),
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1347,21 +1605,41 @@ dropTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
+ PQExpBuffer delQry = createPQExpBuffer();
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(spcname));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(spcname));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP TABLESPACE IF EXISTS %s;\n",
+ fmtId(spcname));
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = psprintf("TABLESPACE %s", fmtId(spcname)),
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1371,6 +1649,8 @@ static void
dumpTablespaces(PGconn *conn)
{
PGresult *res;
+ PQExpBuffer comment_buf = createPQExpBuffer();
+ PQExpBuffer seclabel_buf = createPQExpBuffer();
int i;
/*
@@ -1387,7 +1667,7 @@ dumpTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1406,6 +1686,9 @@ dumpTablespaces(PGconn *conn)
/* needed for buildACLCommands() */
fspcname = pg_strdup(fmtId(spcname));
+ resetPQExpBuffer(comment_buf);
+ resetPQExpBuffer(seclabel_buf);
+
if (binary_upgrade)
{
appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_tablespace oid\n");
@@ -1447,24 +1730,67 @@ dumpTablespaces(PGconn *conn)
if (!no_comments && spccomment && spccomment[0] != '\0')
{
- appendPQExpBuffer(buf, "COMMENT ON TABLESPACE %s IS ", fspcname);
- appendStringLiteralConn(buf, spccomment, conn);
- appendPQExpBufferStr(buf, ";\n");
+ appendPQExpBuffer(comment_buf, "COMMENT ON TABLESPACE %s IS ", fspcname);
+ appendStringLiteralConn(comment_buf, spccomment, conn);
+ appendPQExpBufferStr(comment_buf, ";\n");
}
if (!no_security_labels)
buildShSecLabels(conn, "pg_tablespace", spcoid,
"TABLESPACE", spcname,
- buf);
+ seclabel_buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ {
+ fprintf(OPF, "%s", buf->data);
+
+ if (comment_buf->data[0] != '\0')
+ fprintf(OPF, "%s", comment_buf->data);
+
+ if (seclabel_buf->data[0] != '\0')
+ fprintf(OPF, "%s", seclabel_buf->data);
+ }
+ else
+ {
+ char *tag = psprintf("TABLESPACE %s", fmtId(fspcname));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "TABLESPACE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+
+ if (comment_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "COMMENT",
+ .section = SECTION_PRE_DATA,
+ .createStmt = comment_buf->data));
+
+ if (seclabel_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "SECURITY LABEL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = seclabel_buf->data));
+ }
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ destroyPQExpBuffer(comment_buf);
+ destroyPQExpBuffer(seclabel_buf);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1487,12 +1813,13 @@ dropDBs(PGconn *conn)
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY datname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Skip "postgres" and "template1"; dumpDatabases() will deal with
@@ -1503,15 +1830,35 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(dbname));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(dbname));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP DATABASE IF EXISTS %s;\n",
+ fmtId(dbname));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = psprintf("DATABASE %s", fmtId(dbname)),
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1533,7 +1880,7 @@ dumpUserConfig(PGconn *conn, const char *username)
res = executeQuery(conn, buf->data);
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
{
char *sanitized;
@@ -1548,7 +1895,17 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = psprintf("ROLE %s", fmtId(username)),
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
}
PQclear(res);
@@ -1618,6 +1975,9 @@ dumpDatabases(PGconn *conn)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1631,19 +1991,59 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+ char *map_preamble[] = {
+ "#################################################################",
+ "# map.dat",
+ "#",
+ "# This file maps oids to database names",
+ "#",
+ "# pg_restore will restore all the databases listed here, unless",
+ "# otherwise excluded. You can also inhibit restoration of a",
+ "# database by removing the line or commenting out the line with"
+ "# a # mark.",
+ "#################################################################",
+ NULL
+ };
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ for (char **line = map_preamble; *line; line++)
+ fprintf(map_file, "%s\n", *line);
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1660,7 +2060,10 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
free(sanitized);
/*
@@ -1675,24 +2078,40 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
- else
- {
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
fprintf(OPF, "\\connect %s\n\n", dbname);
- }
+ else
+ create_opts = "";
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1701,6 +2120,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1710,7 +2133,7 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1719,17 +2142,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s -f %s %s", pg_dump_bin,
+ pgdumpopts->data, dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1803,6 +2245,66 @@ executeCommand(PGconn *conn, const char *query)
}
+/*
+ * check_for_invalid_global_names
+ *
+ * Check that no database, role, or tablespace name contains a newline or
+ * carriage return character. Such characters would break the map.dat file
+ * format used for non-plain-text dumps.
+ */
+static void
+check_for_invalid_global_names(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ PQExpBuffer names;
+ int count = 0;
+
+ res = executeQuery(conn,
+ "SELECT datname AS objname, 'database' AS objtype "
+ "FROM pg_catalog.pg_database "
+ "WHERE datallowconn AND datconnlimit != -2 "
+ "UNION ALL "
+ "SELECT rolname AS objname, 'role' AS objtype "
+ "FROM pg_catalog.pg_roles "
+ "UNION ALL "
+ "SELECT spcname AS objname, 'tablespace' AS objtype "
+ "FROM pg_catalog.pg_tablespace");
+
+ names = createPQExpBuffer();
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ char *objname = PQgetvalue(res, i, 0);
+ char *objtype = PQgetvalue(res, i, 1);
+
+ if (strpbrk(objname, "\n\r"))
+ {
+ appendPQExpBuffer(names, " %s: \"", objtype);
+ for (char *p = objname; *p; p++)
+ {
+ if (*p == '\n')
+ appendPQExpBufferStr(names, "\\n");
+ else if (*p == '\r')
+ appendPQExpBufferStr(names, "\\r");
+ else
+ appendPQExpBufferChar(names, *p);
+ }
+ appendPQExpBufferStr(names, "\"\n");
+ count++;
+ }
+ }
+
+ PQclear(res);
+
+ if (count > 0)
+ pg_fatal("database, role, or tablespace names contain a newline or carriage return character, which is not supported in non-plain-text dumps:\n%s",
+ names->data);
+
+ destroyPQExpBuffer(names);
+}
+
+
/*
* dumpTimestamp
*/
@@ -1874,3 +2376,47 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+/*
+ * createDumpId
+ *
+ * Return the next dumpId.
+ */
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 84b8d410c9e..14d886fc86e 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,12 +41,16 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -54,18 +58,41 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data);
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts);
+
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +116,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +170,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +199,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +226,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global sql commands. */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -321,6 +353,10 @@ main(int argc, char **argv)
opts->restrict_key = pg_strdup(optarg);
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -347,6 +383,14 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option %s cannot be used together with %s",
+ "--exclude-database", "-g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -420,6 +464,31 @@ main(int argc, char **argv)
pg_fatal("options %s and %s cannot be used together",
"-1/--single-transaction", "--transaction-size");
+ if (opts->single_txn && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--single-transaction", "-g/--globals-only");
+
+ if (opts->txn_size && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--transaction-size", "-g/--globals-only");
+
+ if (opts->exit_on_error && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--exit-on-error", "-g/--globals-only");
+
+ if (data_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-a/--data-only", "-g/--globals-only");
+ if (schema_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-s/--schema-only", "-g/--globals-only");
+ if (statistics_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "--statistics-only", "-g/--globals-only");
+ if (with_statistics && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "--statistics", "-g/--globals-only");
+
/*
* -C is not compatible with -1, because we can't create a database inside
* a transaction block.
@@ -485,6 +554,183 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.glo file is present, then restore all the databases from
+ * map.dat, but skip restoring those matching --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "toc.glo")))
+ {
+ char global_path[MAXPGPATH];
+ RestoreOptions *tmpopts = pg_malloc0_object(RestoreOptions);
+
+ opts->format = archUnknown;
+
+ memcpy(tmpopts, opts, sizeof(RestoreOptions));
+
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-l/--list");
+ if (opts->tocFile)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-L/--use-list");
+
+ if (opts->strict_names)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--strict-names");
+ if (globals_only && opts->dropSchema)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--clean", "-g/--globals-only");
+
+ /*
+ * For pg_dumpall archives, --clean implies --if-exists since global
+ * objects may not exist in the target cluster.
+ */
+ if (opts->dropSchema && !opts->if_exists)
+ {
+ opts->if_exists = 1;
+ pg_log_info("--if-exists is implied by --clean for pg_dumpall archives");
+ }
+
+ if (no_schema)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--no-schema");
+
+ if (data_only)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-a/--data-only");
+
+ if (statistics_only)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--statistics-only");
+
+ if (!(opts->dumpSections & DUMP_PRE_DATA))
+ pg_fatal("option %s cannot exclude %s when restoring a pg_dumpall archive",
+ "--section", "--pre-data");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option %s must be specified when restoring an archive created by pg_dumpall",
+ "-C/--create");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Always restore global objects, even if --exclude-database results
+ * in zero databases to process. If 'globals-only' is set, exit
+ * immediately.
+ */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+
+ n_errors = restore_global_objects(global_path, tmpopts);
+
+ if (globals_only)
+ pg_log_info("database restoring skipped because option %s was specified",
+ "-g/--globals-only");
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else
+ {
+ if (db_exclude_patterns.head != NULL)
+ {
+ simple_string_list_destroy(&db_exclude_patterns);
+ pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
+ "--exclude-database");
+ }
+
+ if (globals_only)
+ pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
+ "-g/--globals-only");
+
+ /* Process if toc.glo file does not exist. */
+ n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ */
+static int
+restore_global_objects(const char *inputFileSpec, RestoreOptions *opts)
+{
+ Archive *AH;
+ int nerror = 0;
+
+ /* Set format as custom so that toc.glo file can be read. */
+ opts->format = archCustom;
+ opts->txn_size = 0;
+
+ AH = OpenArchive(inputFileSpec, opts->format);
+
+ SetArchiveOptions(AH, NULL, opts);
+
+ on_exit_close_archive(AH);
+
+ /* Let the archiver know how noisy to be */
+ AH->verbose = opts->verbose;
+
+ /* Don't output TOC entry comments when restoring globals */
+ ((ArchiveHandle *) AH)->noTocComments = 1;
+
+ AH->exit_on_error = false;
+
+ /* Parallel execution is not supported for global object restoration. */
+ AH->numWorkers = 1;
+
+ ProcessArchiveRestoreOptions(AH);
+ RestoreArchive(AH, false);
+
+ nerror = AH->n_errors;
+
+ /* AH may be freed in CloseArchive? */
+ CloseArchive(AH);
+
+ return nerror;
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -492,9 +738,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -514,25 +766,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -550,6 +798,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -566,6 +815,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -601,8 +851,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -707,3 +957,422 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PQExpBuffer db_lit;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+ db_lit = createPQExpBuffer();
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+
+ resetPQExpBuffer(query);
+ resetPQExpBuffer(db_lit);
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if (PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches --exclude-database pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ count_db++;
+ }
+
+ destroyPQExpBuffer(query);
+ destroyPQExpBuffer(db_lit);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+ int len;
+
+
+ /*
+ * If there is no map.dat file in dump, then return from here as there is
+ * no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ len = strlen(dumpdirpath);
+
+ /* Trim slash from directory name. */
+ while (len > 1 && dumpdirpath[len - 1] == '/')
+ {
+ dumpdirpath[len - 1] = '\0';
+ len--;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* look for the dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+
+ /* ignore lines that don't begin with a digit */
+ if (p == linebuf.data)
+ continue;
+
+ if (*p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Strip trailing newline */
+ if (namelen > 0 && dbname[namelen - 1] == '\n')
+ dbname[--namelen] = '\0';
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen < 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbidname->str, db_oid, map_file_path);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ pfree(linebuf.data);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total = 0;
+ char *connected_db = NULL;
+ PGconn *conn = NULL;
+ RestoreOptions *original_opts = pg_malloc0_object(RestoreOptions);
+ RestoreOptions *tmpopts = pg_malloc0_object(RestoreOptions);
+
+ memcpy(original_opts, opts, sizeof(RestoreOptions));
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile((char *) inputFileSpec, &dbname_oid_list);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, connect to the database to process them.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (!conn)
+ {
+ pg_log_error("could not connect to databases \"postgres\" or \"template1\"\n"
+ "Please specify an alternative database.");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+ }
+ }
+
+ /* Filter the db list according to the exclude patterns. */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+ PQfinish(conn);
+ }
+ else
+ num_db_restore = num_total_db;
+
+ /* Exit if no db needs to be restored. */
+ if (num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ pg_free(original_opts);
+ pg_free(tmpopts);
+ return 0;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * Since pg_backup_archiver.c may modify RestoreOptions during the
+ * previous restore, we must provide a fresh copy of the original
+ * "opts" for each call to restore_one_database.
+ */
+ memcpy(tmpopts, original_opts, sizeof(RestoreOptions));
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (tmpopts->cparams.override_dbname)
+ {
+ pfree(tmpopts->cparams.override_dbname);
+ tmpopts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (tmpopts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, tmpopts->cparams.pghost,
+ tmpopts->cparams.pgport, tmpopts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ tmpopts->createDB = 0;
+ tmpopts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* We'll have to create it */
+ tmpopts->createDB = 1;
+ tmpopts->cparams.dbname = connected_db;
+ }
+ }
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true);
+
+ n_errors_total += n_errors;
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ pg_free(original_opts);
+ pg_free(tmpopts);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
index ab9310eb42b..a895bc314b0 100644
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -244,4 +244,59 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--restrict-key=uu', '-f dumpfile' ],
+ qr/\Qpg_dumpall: error: option --restrict-key can only be used with --format=plain\E/,
+ 'pg_dumpall: --restrict-key can only be used with plain dump format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--globals-only', '--clean', '-f', 'dumpfile' ],
+ qr/\Qpg_dumpall: error: options --clean and -g\/--globals-only cannot be used together in non-text dump\E/,
+ 'pg_dumpall: --clean and -g/--globals-only cannot be used together in non-text dump');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd' ],
+ qr/\Qpg_dumpall: error: option -F\/--format=d|c|t requires option -f\/--file\E/,
+ 'pg_dumpall: non-plain format requires --file option');
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--data-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options -a\/--data-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options -a/--data-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--schema-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options -s\/--schema-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options -s/--schema-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--statistics-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options --statistics-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options --statistics-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option --exclude-database can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --exclude-database is used in pg_restore with dump of pg_dump'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--globals-only', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option -g\/--globals-only can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --globals-only is used in pg_restore with the dump of pg_dump'
+);
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100644
index 00000000000..b228e572f43
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,639 @@
+# Copyright (c) 2021-2026, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# escape tablespace locations on Windows.
+my $tablespace2_orig = $tablespace2;
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2_orig';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# report an error when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# report an error when \l/--list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# report an error when -L/--use-list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--use-list' => 'use',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -L\/--use-list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When -L/--use-list is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --strict-names option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--strict-names',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --strict-names cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --strict-names is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --clean and -g/--globals-only are used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--clean',
+ '--globals-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --clean and -g\/--globals-only cannot be used together when restoring an archive created by pg_dumpall\E/,
+ 'When --clean and -g/--globals-only are used in pg_restore with dump of pg_dumpall'
+);
+
+# report an error when non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+# report an error when --no-schema is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--no-schema',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --no-schema cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --no-schema is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --data-only is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--data-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -a\/--data-only cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --data-only is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --statistics-only is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--statistics-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --statistics-only cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --statistics-only is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --section excludes pre-data with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--section' => 'post-data',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --section cannot exclude --pre-data when restoring a pg_dumpall archive\E/,
+ 'When --section=post-data is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --globals-only and --data-only are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--data-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options -a\/--data-only and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --data-only are used together');
+
+# report an error when --globals-only and --schema-only are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--schema-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options -s\/--schema-only and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --schema-only are used together');
+
+# report an error when --globals-only and --statistics-only are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--statistics-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --statistics-only and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --statistics-only are used together');
+
+# report an error when --globals-only and --statistics are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--statistics',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --statistics and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --statistics are used together');
+
+# report an error when --globals-only and --exit-on-error are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--exit-on-error',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --exit-on-error and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --exit-on-error are used together');
+
+# report an error when --globals-only and --single-transaction are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--single-transaction',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --single-transaction and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --single-transaction are used together');
+
+# report an error when --globals-only and --transaction-size are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--transaction-size' => '100',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --transaction-size and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --transaction-size are used together');
+
+# verify map.dat preamble exists
+my $map_dat_content = slurp_file("$tempdir/format_directory/map.dat");
+like(
+ $map_dat_content,
+ qr/^# map\.dat\n.*# This file maps oids to database names/ms,
+ 'map.dat contains expected preamble');
+
+# verify commenting out a line in map.dat skips that database
+$node->safe_psql($run_db, 'CREATE DATABASE comment_test_db;
+\c comment_test_db
+CREATE TABLE comment_test_table (id int);');
+
+$node->command_ok(
+ [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/comment_test",
+ ],
+ 'pg_dumpall for comment test');
+
+# Modify map.dat to comment out the comment_test_db entry
+my $map_content = slurp_file("$tempdir/comment_test/map.dat");
+$map_content =~ s/^(\d+ comment_test_db)$/# $1/m;
+open(my $fh, '>', "$tempdir/comment_test/map.dat")
+ or die "Cannot open map.dat: $!";
+print $fh $map_content;
+close($fh);
+
+# Create a target node and restore - commented db should be skipped
+my $target_comment = PostgreSQL::Test::Cluster->new("target_comment");
+$target_comment->init;
+$target_comment->start;
+
+$node->command_ok(
+ [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/comment_test_restore.sql",
+ '--host', $target_comment->host,
+ '--port', $target_comment->port,
+ "$tempdir/comment_test",
+ ],
+ 'pg_restore with commented out database in map.dat');
+
+my $restore_output = slurp_file("$tempdir/comment_test_restore.sql");
+unlike(
+ $restore_output,
+ qr/CREATE DATABASE comment_test_db/,
+ 'commented out database in map.dat is not restored');
+
+# Test that --clean implies --if-exists for pg_dumpall archives
+$node->command_ok(
+ [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--clean',
+ '--file' => "$tempdir/clean_test.sql",
+ "$tempdir/format_custom",
+ ],
+ 'pg_restore with --clean on pg_dumpall archive');
+
+my $clean_output = slurp_file("$tempdir/clean_test.sql");
+like(
+ $clean_output,
+ qr/DROP ROLE IF EXISTS/,
+ '--clean implies --if-exists: DROP ROLE IF EXISTS in output');
+
+$node->stop('fast');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..1a89ef94bec 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -600,6 +600,7 @@ CustomScanMethods
CustomScanState
CycleCtr
DBState
+DbOidName
DCHCacheEntry
DEADLOCK_INFO
DECountItem
--
2.43.0
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-24 07:21 jian he <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: jian he @ 2026-02-24 07:21 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
Hi.
For v19, the commit message
"""
pg_restore is extended to handle these pg_dumpall archives, restoring
globals and then each database. The --globals-only and --no-globals
options control which parts are restored.
"""
There is no --no-globals option.
In file src/bin/pg_dump/pg_dumpall.c, no need
``
static pg_compress_specification compression_spec = {0};
``
Since compression_spec is only used in an IF branch, we can declare it locally.
The options below are not supported for pg_restore non-text restore,
we can document this.
<option>-a/--data-only</option>,
<option>-l/--list</option>,
<option>-L/--use-list</option>,
<option>--statistics-only</option>,
<option>--strict-names</option>,
<option>--no-schema</option>.
--
jian
https://www.enterprisedb.com/
Attachments:
[application/octet-stream] v19-0001-misc-change-for-v19.no-cfbot (2.3K, 2-v19-0001-misc-change-for-v19.no-cfbot)
download
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-24 10:40 Andrew Dunstan <[email protected]>
parent: jian he <[email protected]>
0 siblings, 3 replies; 22+ messages in thread
From: Andrew Dunstan @ 2026-02-24 10:40 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On 2026-02-24 Tu 2:21 AM, jian he wrote:
> Hi.
>
> For v19, the commit message
> """
> pg_restore is extended to handle these pg_dumpall archives, restoring
> globals and then each database. The --globals-only and --no-globals
> options control which parts are restored.
> """
> There is no --no-globals option.
>
> In file src/bin/pg_dump/pg_dumpall.c, no need
> ``
> static pg_compress_specification compression_spec = {0};
> ``
> Since compression_spec is only used in an IF branch, we can declare it locally.
>
>
> The options below are not supported for pg_restore non-text restore,
> we can document this.
> <option>-a/--data-only</option>,
> <option>-l/--list</option>,
> <option>-L/--use-list</option>,
> <option>--statistics-only</option>,
> <option>--strict-names</option>,
> <option>--no-schema</option>.
>
>
>
Thanks, fixed these.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
Attachments:
[text/x-patch] v20-0001-Add-non-text-output-formats-to-pg_dumpall.patch (112.1K, 2-v20-0001-Add-non-text-output-formats-to-pg_dumpall.patch)
download | inline diff:
From 1c62cef092f6d3a71adc3694898e49d03661ea02 Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email protected]>
Date: Mon, 23 Feb 2026 15:08:55 -0500
Subject: [PATCH v20] Add non-text output formats to pg_dumpall
pg_dumpall can now produce output in custom, directory, or tar formats
in addition to plain text SQL scripts. When using non-text formats,
pg_dumpall creates a directory containing:
- toc.glo: global data (roles and tablespaces) in custom format
- map.dat: mapping between database OIDs and names
- databases/: subdirectory with per-database archives named by OID
pg_restore is extended to handle these pg_dumpall archives, restoring
globals and then each database. The --globals-only option can be used
to restore only the global objects.
This enables parallel restore of pg_dumpall output and selective
restoration of individual databases from a cluster-wide backup.
Author: Mahendra Singh Thalor <[email protected]>
Co-Author: Andrew Dunstan <[email protected]>
Reviewed-By: Tushar Ahuja <[email protected]>
Reviewed-By: Jian He <[email protected]>
Reviewed-By: Vaibhav Dalvi <[email protected]>
Reviewed-By: Srinath Reddy <[email protected]>
Discussion: https://postgr.es/m/[email protected]
---
doc/src/sgml/ref/pg_dumpall.sgml | 113 +++-
doc/src/sgml/ref/pg_restore.sgml | 119 +++-
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/parallel.c | 14 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 67 ++-
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 800 ++++++++++++++++++++++-----
src/bin/pg_dump/pg_restore.c | 703 ++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 55 ++
src/bin/pg_dump/t/007_pg_dumpall.pl | 639 +++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 +
14 files changed, 2348 insertions(+), 171 deletions(-)
create mode 100644 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 8834b7ec141..49e5c99b09e 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,10 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+
+ <refpurpose>
+ export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
+ </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +36,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an SQL script file or an archive. The output contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +55,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -131,16 +139,93 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ This option can only be omitted when <option>--format</option> is plain.
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files,
+ <filename>toc.glo</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data (roles and tablespaces) in custom format. The second
+ contains a mapping between database OIDs and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ the <filename>databases</filename> subdirectory, named using the database's OID.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ See <xref linkend="app-pgdump"/> for details on how the
+ various non-plain-text archive formats work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
<listitem>
<para>
Dump only global objects (roles and tablespaces), no databases.
+ Note: <option>--globals-only</option> cannot be used with
+ <option>--clean</option> with non-text dump format.
</para>
</listitem>
</varlistentry>
@@ -936,13 +1021,21 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
<refsect1 id="app-pg-dumpall-ex">
<title>Examples</title>
<para>
- To dump all databases:
-
+ To dump all databases in plain text format (the default):
<screen>
<prompt>$</prompt> <userinput>pg_dumpall > db.out</userinput>
</screen>
</para>
+ <para>
+ To dump all databases using other formats:
+<screen>
+<prompt>$</prompt> <userinput>pg_dumpall --format=directory -f db.out</userinput>
+<prompt>$</prompt> <userinput>pg_dumpall --format=custom -f db.out</userinput>
+<prompt>$</prompt> <userinput>pg_dumpall --format=tar -f db.out</userinput>
+</screen>
+ </para>
+
<para>
To restore database(s) from this file, you can use:
<screen>
@@ -956,6 +1049,16 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
the script will attempt to drop other databases immediately, and that
will fail for the database you are connected to.
</para>
+
+ <para>
+ If the dump was taken in a non-plain-text format, use
+ <application>pg_restore</application> to restore the databases:
+<screen>
+<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
+</screen>
+ This will restore all databases. To restore only some databases, use
+ the <option>--exclude-database</option> option to skip those not wanted.
+ </para>
</refsect1>
<refsect1>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 420a308a7c7..4a21a089840 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> databases from archives
+ created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,14 +54,34 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by <application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
+ <para>
+ A non-plain-text archive made using <application>pg_dumpall</application>
+ is a directory containing a <filename>toc.glo</filename> file with global
+ objects (roles and tablespaces), a <filename>map.dat</filename> file
+ listing the databases, and a subdirectory for each database containing
+ its archive. When restoring such an archive,
+ <application>pg_restore</application> first restores global objects from
+ <filename>toc.glo</filename>, then processes each database listed in
+ <filename>map.dat</filename>. Lines in <filename>map.dat</filename> can
+ be commented out with <literal>#</literal> to skip restoring specific
+ databases.
+ </para>
+
<para>
Obviously, <application>pg_restore</application> cannot restore information
that is not present in the archive file. For instance, if the
@@ -130,6 +152,12 @@ PostgreSQL documentation
ignorable error messages will be reported,
unless <option>--if-exists</option> is also specified.
</para>
+ <para>
+ When restoring a <application>pg_dumpall</application> archive,
+ <option>--if-exists</option> is implied by <option>--clean</option>,
+ since global objects such as roles and tablespaces may not exist
+ in the target cluster.
+ </para>
</listitem>
</varlistentry>
@@ -152,6 +180,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from a non-plain-text archive made using <application>pg_dumpall</application>.
</para>
<para>
@@ -247,6 +277,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from a non-plain-text archive made using <application>pg_dumpall</application>.
+ Note: <option>--globals-only</option> cannot be used with
+ <option>--data-only</option>,
+ <option>--schema-only</option>,
+ <option>--statistics-only</option>,
+ <option>--statistics</option>,
+ <option>--exit-on-error</option>,
+ <option>--single-transaction</option>,
+ <option>--clean</option>, or
+ <option>--transaction-size</option>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -581,6 +633,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from a non-plain-text archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
@@ -669,7 +743,9 @@ PostgreSQL documentation
in <option>--clean</option> mode. This suppresses <quote>does not
exist</quote> errors that might otherwise be reported. This
option is not valid unless <option>--clean</option> is also
- specified.
+ specified. This option is implied when restoring a
+ <application>pg_dumpall</application> archive with
+ <option>--clean</option>.
</para>
</listitem>
</varlistentry>
@@ -1125,6 +1201,27 @@ CREATE DATABASE foo WITH TEMPLATE template0;
</para>
</listitem>
+ <listitem>
+ <para>
+ When restoring from a non-plain-text archive made using
+ <application>pg_dumpall</application>, the <option>--section</option>
+ option may be used, but must include <option>pre-data</option>.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The following options cannot be used when restoring from a non-plain-text
+ archive made using <application>pg_dumpall</application>:
+ <option>-a/--data-only</option>,
+ <option>-l/--list</option>,
+ <option>-L/--use-list</option>,
+ <option>--no-schema</option>,
+ <option>--statistics-only</option>, and
+ <option>--strict-names</option>.
+ </para>
+ </listitem>
+
</itemizedlist>
</para>
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 79bd5036841..7c9a475963b 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,6 +103,7 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
+ 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 56cb2c1f32d..a28561fbd84 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,20 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * Update the archive handle in the on_exit callback registered by
+ * on_exit_close_archive(). When pg_restore processes a pg_dumpall archive
+ * containing multiple databases, each database is restored from a separate
+ * archive. After closing one archive and opening the next, we update the
+ * shutdown_info to reference the new archive handle so the cleanup callback
+ * will close the correct archive on exit.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 2f8d9799c30..fda912ba0a9 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -313,7 +313,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 7afcc0859c8..df8a69d3b79 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -86,7 +86,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -339,9 +339,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -458,7 +463,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -761,6 +766,19 @@ RestoreArchive(Archive *AHX)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
+ /* Skip if no-tablespace is given. */
+ if (ropt->noTablespace && te && te->desc &&
+ (strcmp(te->desc, "TABLESPACE") == 0))
+ continue;
+
+ /*
+ * Skip DROP DATABASE/ROLES/TABLESPACE if we didn't specify
+ * --clean
+ */
+ if (!ropt->dropSchema && te && te->desc &&
+ strcmp(te->desc, "DROP_GLOBAL") == 0)
+ continue;
+
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1316,7 +1334,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1691,11 +1709,15 @@ archprintf(Archive *AH, const char *fmt,...)
/*******************************
* Stuff below here should be 'private' to the archiver routines
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
*******************************/
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1715,7 +1737,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
@@ -2391,7 +2413,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
/* initialize for backwards compatible string processing */
AH->public.encoding = 0; /* PG_SQL_ASCII */
- AH->public.std_strings = false;
+ AH->public.std_strings = true;
/* sql error handling */
AH->public.exit_on_error = true;
@@ -3027,6 +3049,16 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
return 0;
}
+ /*
+ * Global object TOC entries (e.g., ROLEs or TABLESPACEs) must not be
+ * ignored.
+ */
+ if (strcmp(te->desc, "ROLE") == 0 ||
+ strcmp(te->desc, "ROLE PROPERTIES") == 0 ||
+ strcmp(te->desc, "TABLESPACE") == 0 ||
+ strcmp(te->desc, "DROP_GLOBAL") == 0)
+ return REQ_SCHEMA;
+
/*
* Process exclusions that affect certain classes of TOC entries.
*/
@@ -3062,6 +3094,14 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
if (ropt->no_subscriptions &&
strncmp(te->tag, "SUBSCRIPTION", strlen("SUBSCRIPTION")) == 0)
return 0;
+
+ /*
+ * Comments on global objects (ROLEs or TABLESPACEs) should not be
+ * skipped, since global objects themselves are never skipped.
+ */
+ if (strncmp(te->tag, "ROLE", strlen("ROLE")) == 0 ||
+ strncmp(te->tag, "TABLESPACE", strlen("TABLESPACE")) == 0)
+ return REQ_SCHEMA;
}
/*
@@ -3091,6 +3131,14 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
if (ropt->no_subscriptions &&
strncmp(te->tag, "SUBSCRIPTION", strlen("SUBSCRIPTION")) == 0)
return 0;
+
+ /*
+ * Security labels on global objects (ROLEs or TABLESPACEs) should not
+ * be skipped, since global objects themselves are never skipped.
+ */
+ if (strncmp(te->tag, "ROLE", strlen("ROLE")) == 0 ||
+ strncmp(te->tag, "TABLESPACE", strlen("TABLESPACE")) == 0)
+ return REQ_SCHEMA;
}
/* If it's a subscription, maybe ignore it */
@@ -3865,6 +3913,9 @@ _getObjectDescription(PQExpBuffer buf, const TocEntry *te)
else if (strcmp(type, "CAST") == 0 ||
strcmp(type, "CHECK CONSTRAINT") == 0 ||
strcmp(type, "CONSTRAINT") == 0 ||
+ strcmp(type, "DROP_GLOBAL") == 0 ||
+ strcmp(type, "ROLE PROPERTIES") == 0 ||
+ strcmp(type, "ROLE") == 0 ||
strcmp(type, "DATABASE PROPERTIES") == 0 ||
strcmp(type, "DEFAULT") == 0 ||
strcmp(type, "FK CONSTRAINT") == 0 ||
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 325b53fc9bd..365073b3eae 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,6 +394,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 450cec285b3..ce29627050f 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1292,7 +1292,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 98389d2034c..cbb6150ab40 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -1,13 +1,20 @@
/*-------------------------------------------------------------------------
*
* pg_dumpall.c
+ * pg_dumpall dumps all databases and global objects (roles and
+ * tablespaces) from a PostgreSQL cluster.
+ *
+ * For text format output, globals are written directly and pg_dump is
+ * invoked for each database, with all output going to stdout or a file.
+ *
+ * For non-text formats (custom, directory, tar), a directory is created
+ * containing a toc.glo file with global objects, a map.dat file mapping
+ * database OIDs to names, and a databases/ subdirectory with individual
+ * pg_dump archives for each database.
*
* Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
- * pg_dumpall forces all pg_dump output to be text, since it also outputs
- * text into the same output stream.
- *
* src/bin/pg_dump/pg_dumpall.c
*
*-------------------------------------------------------------------------
@@ -30,6 +37,7 @@
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -67,15 +75,18 @@ static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
static void dumpDatabases(PGconn *conn);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts, char *dbfile);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
static void executeCommand(PGconn *conn, const char *query);
+static void check_for_invalid_global_names(PGconn *conn);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static ArchiveFormat parseDumpFormat(const char *format);
+static int createDumpId(void);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -123,6 +134,10 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
+static Archive *fout = NULL;
+static int dumpIdVal = 0;
+static ArchiveFormat archDumpFormat = archNull;
+static const CatalogId nilCatalogId = {0, 0};
int
main(int argc, char *argv[])
@@ -148,6 +163,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -197,6 +213,7 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ const char *format_name = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -207,6 +224,7 @@ main(int argc, char *argv[])
int c,
ret;
int optindex;
+ DumpOptions dopt;
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -244,8 +262,9 @@ main(int argc, char *argv[])
}
pgdumpopts = createPQExpBuffer();
+ InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -273,7 +292,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ format_name = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -313,6 +334,7 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
+ dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -434,6 +456,32 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(format_name);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option %s=d|c|t requires option %s",
+ "-F/--format", "-f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
+ /* restrict-key is only supported with --format=plain */
+ if (archDumpFormat != archNull && restrict_key)
+ pg_fatal("option %s can only be used with %s=plain",
+ "--restrict-key", "--format");
+
+ /* --clean and -g/--globals-only cannot be used together in non-text dump */
+ if (archDumpFormat != archNull && output_clean && globals_only)
+ pg_fatal("options %s and %s cannot be used together in non-text dump",
+ "--clean", "-g/--globals-only");
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -494,6 +542,27 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory.
+ */
+ if (archDumpFormat != archNull)
+ {
+ Assert(filename);
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -543,19 +612,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -593,35 +649,121 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
- if (verbose)
- dumpTimestamp("Started on");
-
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the dump
- * output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
-
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so whichever
- * database we're connected to at the moment is fine.
- */
-
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
-
- /* Replicate encoding and standard_conforming_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = on;\n");
- fprintf(OPF, "\n");
+ /* create a archive file for global commands. */
+ if (archDumpFormat != archNull)
+ {
+ PQExpBuffer qry = createPQExpBuffer();
+ char global_path[MAXPGPATH];
+ const char *encname;
+ pg_compress_specification compression_spec = {0};
+
+ /*
+ * Check that no global object names contain newlines or carriage
+ * returns, which would break the map.dat file format. This is only
+ * needed for servers older than v19, which started prohibiting such
+ * names.
+ */
+ if (server_version < 190000)
+ check_for_invalid_global_names(conn);
+
+ /* Set file path for global sql commands. */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", filename);
+
+ /* Open the output file */
+ fout = CreateArchive(global_path, archCustom, compression_spec,
+ dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC);
+
+ /* Make dump options accessible right away */
+ SetArchiveOptions(fout, &dopt, NULL);
+
+ ((ArchiveHandle *) fout)->connection = conn;
+ ((ArchiveHandle *) fout)->public.numWorkers = 1;
+
+ /* Register the cleanup hook */
+ on_exit_close_archive(fout);
+
+ /* Let the archiver know how noisy to be */
+ fout->verbose = verbose;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release
+ * of our own major version. (See also version check in
+ * pg_dumpall.c.)
+ */
+ fout->minRemoteVersion = 90200;
+ fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
+ fout->numWorkers = 1;
+
+ /* Dump default_transaction_read_only. */
+ appendPQExpBufferStr(qry, "SET default_transaction_read_only = off;\n\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "default_transaction_read_only",
+ .description = "default_transaction_read_only",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ resetPQExpBuffer(qry);
+
+ /* Put the correct encoding into the archive */
+ encname = pg_encoding_to_char(encoding);
+
+ appendPQExpBufferStr(qry, "SET client_encoding = ");
+ appendStringLiteralAH(qry, encname, fout);
+ appendPQExpBufferStr(qry, ";\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "client_encoding",
+ .description = "client_encoding",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ resetPQExpBuffer(qry);
+
+ /* Put the correct escape string behavior into the archive. */
+ appendPQExpBuffer(qry, "SET standard_conforming_strings = 'on';\n");
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = "standard_conforming_strings",
+ .description = "standard_conforming_strings",
+ .section = SECTION_PRE_DATA,
+ .createStmt = qry->data));
+ destroyPQExpBuffer(qry);
+ }
+ else
+ {
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+
+ if (verbose)
+ dumpTimestamp("Started on");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the
+ * dump output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so
+ * whichever database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and standard_conforming_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = on;\n");
+ fprintf(OPF, "\n");
+ }
if (!data_only && !statistics_only && !no_schema)
{
@@ -630,8 +772,14 @@ main(int argc, char *argv[])
* dependency analysis because databases never depend on each other,
* and tablespaces never depend on each other. Roles could have
* grants to each other, but DROP ROLE will clean those up silently.
+ *
+ * For non-text formats, pg_dumpall unconditionally process --clean
+ * option. In contrast, pg_restore only applies it if the user
+ * explicitly provides the flag. This discrepancy resolves corner
+ * cases where pg_restore requires cleanup instructions that may be
+ * missing from a standard pg_dumpall output.
*/
- if (output_clean)
+ if (output_clean || archDumpFormat != archNull)
{
if (!globals_only && !roles_only && !tablespaces_only)
dropDBs(conn);
@@ -665,28 +813,45 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- /*
- * Exit restricted mode just before dumping the databases. pg_dump will
- * handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ if (archDumpFormat == archNull)
+ {
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump
+ * will handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
+ }
if (!globals_only && !roles_only && !tablespaces_only)
dumpDatabases(conn);
- PQfinish(conn);
+ if (archDumpFormat == archNull)
+ {
+ PQfinish(conn);
+
+ if (verbose)
+ dumpTimestamp("Completed on");
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- if (verbose)
- dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+ if (filename)
+ {
+ fclose(OPF);
- if (filename)
+ /* sync the resulting file, errors are not fatal */
+ if (dosync)
+ (void) fsync_fname(filename, false);
+ }
+ }
+ else
{
- fclose(OPF);
+ RestoreOptions *ropt;
+
+ ropt = NewRestoreOptions();
+ SetArchiveOptions(fout, &dopt, ropt);
- /* sync the resulting file, errors are not fatal */
- if (dosync)
- (void) fsync_fname(filename, false);
+ /* Mark which entries should be output */
+ ProcessArchiveRestoreOptions(fout);
+ CloseArchive(fout);
}
exit_nicely(0);
@@ -696,12 +861,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -796,24 +963,45 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
const char *rolename;
+ PQExpBuffer delQry = createPQExpBuffer();
rolename = PQgetvalue(res, i, i_rolname);
- fprintf(OPF, "DROP ROLE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(rolename));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(rolename));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP ROLE IF EXISTS %s;\n",
+ fmtId(rolename));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = psprintf("ROLE %s", fmtId(rolename)),
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
PQclear(res);
destroyPQExpBuffer(buf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -823,6 +1011,8 @@ static void
dumpRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer comment_buf = createPQExpBuffer();
+ PQExpBuffer seclabel_buf = createPQExpBuffer();
PGresult *res;
int i_oid,
i_rolname,
@@ -894,7 +1084,7 @@ dumpRoles(PGconn *conn)
i_rolcomment = PQfnumber(res, "rolcomment");
i_is_current_user = PQfnumber(res, "is_current_user");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -913,6 +1103,8 @@ dumpRoles(PGconn *conn)
}
resetPQExpBuffer(buf);
+ resetPQExpBuffer(comment_buf);
+ resetPQExpBuffer(seclabel_buf);
if (binary_upgrade)
{
@@ -989,17 +1181,53 @@ dumpRoles(PGconn *conn)
if (!no_comments && !PQgetisnull(res, i, i_rolcomment))
{
- appendPQExpBuffer(buf, "COMMENT ON ROLE %s IS ", fmtId(rolename));
- appendStringLiteralConn(buf, PQgetvalue(res, i, i_rolcomment), conn);
- appendPQExpBufferStr(buf, ";\n");
+ appendPQExpBuffer(comment_buf, "COMMENT ON ROLE %s IS ", fmtId(rolename));
+ appendStringLiteralConn(comment_buf, PQgetvalue(res, i, i_rolcomment), conn);
+ appendPQExpBufferStr(comment_buf, ";\n");
}
if (!no_security_labels)
buildShSecLabels(conn, "pg_authid", auth_oid,
"ROLE", rolename,
- buf);
+ seclabel_buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ {
+ fprintf(OPF, "%s", buf->data);
+ fprintf(OPF, "%s", comment_buf->data);
+
+ if (seclabel_buf->data[0] != '\0')
+ fprintf(OPF, "%s", seclabel_buf->data);
+ }
+ else
+ {
+ char *tag = psprintf("ROLE %s", fmtId(rolename));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "ROLE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+ if (comment_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "COMMENT",
+ .section = SECTION_PRE_DATA,
+ .createStmt = comment_buf->data));
+
+ if (seclabel_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "SECURITY LABEL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = seclabel_buf->data));
+ }
}
/*
@@ -1007,7 +1235,7 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "\n--\n-- User Configurations\n--\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1015,9 +1243,12 @@ dumpRoles(PGconn *conn)
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
+ destroyPQExpBuffer(comment_buf);
+ destroyPQExpBuffer(seclabel_buf);
}
@@ -1031,6 +1262,7 @@ static void
dumpRoleMembership(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
+ PQExpBuffer querybuf = createPQExpBuffer();
PQExpBuffer optbuf = createPQExpBuffer();
PGresult *res;
int start = 0,
@@ -1093,7 +1325,7 @@ dumpRoleMembership(PGconn *conn)
i_inherit_option = PQfnumber(res, "inherit_option");
i_set_option = PQfnumber(res, "set_option");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Role memberships\n--\n\n");
/*
@@ -1229,8 +1461,9 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- fprintf(OPF, "GRANT %s", fmtId(role));
- fprintf(OPF, " TO %s", fmtId(member));
+ resetPQExpBuffer(querybuf);
+ appendPQExpBuffer(querybuf, "GRANT %s", fmtId(role));
+ appendPQExpBuffer(querybuf, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1251,10 +1484,21 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- fprintf(OPF, " WITH %s", optbuf->data);
+ appendPQExpBuffer(querybuf, " WITH %s", optbuf->data);
if (dump_grantors)
- fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
- fprintf(OPF, ";\n");
+ appendPQExpBuffer(querybuf, " GRANTED BY %s", fmtId(grantor));
+ appendPQExpBuffer(querybuf, ";\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", querybuf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = psprintf("ROLE %s", fmtId(role)),
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = querybuf->data));
}
}
@@ -1265,8 +1509,11 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
+ destroyPQExpBuffer(querybuf);
+ destroyPQExpBuffer(optbuf);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1293,7 +1540,7 @@ dumpRoleGUCPrivs(PGconn *conn)
"FROM pg_catalog.pg_parameter_acl "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1318,14 +1565,25 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = psprintf("ROLE %s", fmtId(parowner)),
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1347,21 +1605,41 @@ dropTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
+ PQExpBuffer delQry = createPQExpBuffer();
- fprintf(OPF, "DROP TABLESPACE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(spcname));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(spcname));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP TABLESPACE IF EXISTS %s;\n",
+ fmtId(spcname));
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = psprintf("TABLESPACE %s", fmtId(spcname)),
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
/*
@@ -1371,6 +1649,8 @@ static void
dumpTablespaces(PGconn *conn)
{
PGresult *res;
+ PQExpBuffer comment_buf = createPQExpBuffer();
+ PQExpBuffer seclabel_buf = createPQExpBuffer();
int i;
/*
@@ -1387,7 +1667,7 @@ dumpTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1406,6 +1686,9 @@ dumpTablespaces(PGconn *conn)
/* needed for buildACLCommands() */
fspcname = pg_strdup(fmtId(spcname));
+ resetPQExpBuffer(comment_buf);
+ resetPQExpBuffer(seclabel_buf);
+
if (binary_upgrade)
{
appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_tablespace oid\n");
@@ -1447,24 +1730,67 @@ dumpTablespaces(PGconn *conn)
if (!no_comments && spccomment && spccomment[0] != '\0')
{
- appendPQExpBuffer(buf, "COMMENT ON TABLESPACE %s IS ", fspcname);
- appendStringLiteralConn(buf, spccomment, conn);
- appendPQExpBufferStr(buf, ";\n");
+ appendPQExpBuffer(comment_buf, "COMMENT ON TABLESPACE %s IS ", fspcname);
+ appendStringLiteralConn(comment_buf, spccomment, conn);
+ appendPQExpBufferStr(comment_buf, ";\n");
}
if (!no_security_labels)
buildShSecLabels(conn, "pg_tablespace", spcoid,
"TABLESPACE", spcname,
- buf);
+ seclabel_buf);
- fprintf(OPF, "%s", buf->data);
+ if (archDumpFormat == archNull)
+ {
+ fprintf(OPF, "%s", buf->data);
+
+ if (comment_buf->data[0] != '\0')
+ fprintf(OPF, "%s", comment_buf->data);
+
+ if (seclabel_buf->data[0] != '\0')
+ fprintf(OPF, "%s", seclabel_buf->data);
+ }
+ else
+ {
+ char *tag = psprintf("TABLESPACE %s", fmtId(fspcname));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "TABLESPACE",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
+
+ if (comment_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "COMMENT",
+ .section = SECTION_PRE_DATA,
+ .createStmt = comment_buf->data));
+
+ if (seclabel_buf->data[0] != '\0')
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = tag,
+ .description = "SECURITY LABEL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = seclabel_buf->data));
+ }
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ destroyPQExpBuffer(comment_buf);
+ destroyPQExpBuffer(seclabel_buf);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1487,12 +1813,13 @@ dropDBs(PGconn *conn)
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY datname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ PQExpBuffer delQry = createPQExpBuffer();
/*
* Skip "postgres" and "template1"; dumpDatabases() will deal with
@@ -1503,15 +1830,35 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- fprintf(OPF, "DROP DATABASE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(dbname));
+ if (archDumpFormat == archNull)
+ {
+ appendPQExpBuffer(delQry, "DROP DATABASE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(dbname));
+ fprintf(OPF, "%s", delQry->data);
+ }
+ else
+ {
+ appendPQExpBuffer(delQry, "DROP DATABASE IF EXISTS %s;\n",
+ fmtId(dbname));
+
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = psprintf("DATABASE %s", fmtId(dbname)),
+ .description = "DROP_GLOBAL",
+ .section = SECTION_PRE_DATA,
+ .createStmt = delQry->data));
+ }
+
+ destroyPQExpBuffer(delQry);
}
}
PQclear(res);
- fprintf(OPF, "\n\n");
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "\n\n");
}
@@ -1533,7 +1880,7 @@ dumpUserConfig(PGconn *conn, const char *username)
res = executeQuery(conn, buf->data);
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
{
char *sanitized;
@@ -1548,7 +1895,17 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
- fprintf(OPF, "%s", buf->data);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "%s", buf->data);
+ else
+ ArchiveEntry(fout,
+ nilCatalogId, /* catalog ID */
+ createDumpId(), /* dump ID */
+ ARCHIVE_OPTS(.tag = psprintf("ROLE %s", fmtId(username)),
+ .description = "ROLE PROPERTIES",
+ .section = SECTION_PRE_DATA,
+ .createStmt = buf->data));
}
PQclear(res);
@@ -1618,6 +1975,9 @@ dumpDatabases(PGconn *conn)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1631,19 +1991,59 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
- if (PQntuples(res) > 0)
+ if (PQntuples(res) > 0 && archDumpFormat == archNull)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+ char *map_preamble[] = {
+ "#################################################################",
+ "# map.dat",
+ "#",
+ "# This file maps oids to database names",
+ "#",
+ "# pg_restore will restore all the databases listed here, unless",
+ "# otherwise excluded. You can also inhibit restoration of a",
+ "# database by removing the line or commenting out the line with"
+ "# a # mark.",
+ "#################################################################",
+ NULL
+ };
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, pg_dir_create_mode) != 0)
+ pg_fatal("could not create directory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ for (char **line = map_preamble; *line; line++)
+ fprintf(map_file, "%s\n", *line);
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1660,7 +2060,10 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
+
free(sanitized);
/*
@@ -1675,24 +2078,40 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
- else
- {
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
fprintf(OPF, "\\connect %s\n\n", dbname);
- }
+ else
+ create_opts = "";
}
else
create_opts = "--create";
- if (filename)
+ if (filename && archDumpFormat == archNull)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
+ ret = runPgDump(dbname, create_opts, dbfilepath);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename)
+ if (filename && archDumpFormat == archNull)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -1701,6 +2120,10 @@ dumpDatabases(PGconn *conn)
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1710,7 +2133,7 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1719,17 +2142,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s -f %s %s", pg_dump_bin,
+ pgdumpopts->data, dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1803,6 +2245,66 @@ executeCommand(PGconn *conn, const char *query)
}
+/*
+ * check_for_invalid_global_names
+ *
+ * Check that no database, role, or tablespace name contains a newline or
+ * carriage return character. Such characters would break the map.dat file
+ * format used for non-plain-text dumps.
+ */
+static void
+check_for_invalid_global_names(PGconn *conn)
+{
+ PGresult *res;
+ int i;
+ PQExpBuffer names;
+ int count = 0;
+
+ res = executeQuery(conn,
+ "SELECT datname AS objname, 'database' AS objtype "
+ "FROM pg_catalog.pg_database "
+ "WHERE datallowconn AND datconnlimit != -2 "
+ "UNION ALL "
+ "SELECT rolname AS objname, 'role' AS objtype "
+ "FROM pg_catalog.pg_roles "
+ "UNION ALL "
+ "SELECT spcname AS objname, 'tablespace' AS objtype "
+ "FROM pg_catalog.pg_tablespace");
+
+ names = createPQExpBuffer();
+
+ for (i = 0; i < PQntuples(res); i++)
+ {
+ char *objname = PQgetvalue(res, i, 0);
+ char *objtype = PQgetvalue(res, i, 1);
+
+ if (strpbrk(objname, "\n\r"))
+ {
+ appendPQExpBuffer(names, " %s: \"", objtype);
+ for (char *p = objname; *p; p++)
+ {
+ if (*p == '\n')
+ appendPQExpBufferStr(names, "\\n");
+ else if (*p == '\r')
+ appendPQExpBufferStr(names, "\\r");
+ else
+ appendPQExpBufferChar(names, *p);
+ }
+ appendPQExpBufferStr(names, "\"\n");
+ count++;
+ }
+ }
+
+ PQclear(res);
+
+ if (count > 0)
+ pg_fatal("database, role, or tablespace names contain a newline or carriage return character, which is not supported in non-plain-text dumps:\n%s",
+ names->data);
+
+ destroyPQExpBuffer(names);
+}
+
+
/*
* dumpTimestamp
*/
@@ -1874,3 +2376,47 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
+
+/*
+ * createDumpId
+ *
+ * Return the next dumpId.
+ */
+static int
+createDumpId(void)
+{
+ return ++dumpIdVal;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 84b8d410c9e..14d886fc86e 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,12 +41,16 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -54,18 +58,41 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data);
+static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts);
+
+static int restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(char *dumpdirpath,
+ SimplePtrList *dbname_oid_list);
+
+/*
+ * Stores a database OID and the corresponding name.
+ */
+typedef struct DbOidName
+{
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} DbOidName;
+
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -89,6 +116,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -142,6 +170,7 @@ main(int argc, char **argv)
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
+ {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -170,7 +199,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -197,11 +226,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global sql commands. */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -321,6 +353,10 @@ main(int argc, char **argv)
opts->restrict_key = pg_strdup(optarg);
break;
+ case 7: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
+
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -347,6 +383,14 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option %s cannot be used together with %s",
+ "--exclude-database", "-g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -420,6 +464,31 @@ main(int argc, char **argv)
pg_fatal("options %s and %s cannot be used together",
"-1/--single-transaction", "--transaction-size");
+ if (opts->single_txn && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--single-transaction", "-g/--globals-only");
+
+ if (opts->txn_size && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--transaction-size", "-g/--globals-only");
+
+ if (opts->exit_on_error && globals_only)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--exit-on-error", "-g/--globals-only");
+
+ if (data_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-a/--data-only", "-g/--globals-only");
+ if (schema_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-s/--schema-only", "-g/--globals-only");
+ if (statistics_only && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "--statistics-only", "-g/--globals-only");
+ if (with_statistics && globals_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "--statistics", "-g/--globals-only");
+
/*
* -C is not compatible with -1, because we can't create a database inside
* a transaction block.
@@ -485,6 +554,183 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.glo file is present, then restore all the databases from
+ * map.dat, but skip restoring those matching --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL &&
+ (file_exists_in_directory(inputFileSpec, "toc.glo")))
+ {
+ char global_path[MAXPGPATH];
+ RestoreOptions *tmpopts = pg_malloc0_object(RestoreOptions);
+
+ opts->format = archUnknown;
+
+ memcpy(tmpopts, opts, sizeof(RestoreOptions));
+
+ /*
+ * Can only use --list or --use-list options with a single database
+ * dump.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-l/--list");
+ if (opts->tocFile)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-L/--use-list");
+
+ if (opts->strict_names)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--strict-names");
+ if (globals_only && opts->dropSchema)
+ pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
+ "--clean", "-g/--globals-only");
+
+ /*
+ * For pg_dumpall archives, --clean implies --if-exists since global
+ * objects may not exist in the target cluster.
+ */
+ if (opts->dropSchema && !opts->if_exists)
+ {
+ opts->if_exists = 1;
+ pg_log_info("--if-exists is implied by --clean for pg_dumpall archives");
+ }
+
+ if (no_schema)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--no-schema");
+
+ if (data_only)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "-a/--data-only");
+
+ if (statistics_only)
+ pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
+ "--statistics-only");
+
+ if (!(opts->dumpSections & DUMP_PRE_DATA))
+ pg_fatal("option %s cannot exclude %s when restoring a pg_dumpall archive",
+ "--section", "--pre-data");
+
+ /*
+ * To restore from a pg_dumpall archive, -C (create database) option
+ * must be specified unless we are only restoring globals.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("option %s must be specified when restoring an archive created by pg_dumpall",
+ "-C/--create");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("Individual databases can be restored using their specific archives.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Always restore global objects, even if --exclude-database results
+ * in zero databases to process. If 'globals-only' is set, exit
+ * immediately.
+ */
+ snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
+
+ n_errors = restore_global_objects(global_path, tmpopts);
+
+ if (globals_only)
+ pg_log_info("database restoring skipped because option %s was specified",
+ "-g/--globals-only");
+ else
+ {
+ /* Now restore all the databases from map.dat */
+ n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else
+ {
+ if (db_exclude_patterns.head != NULL)
+ {
+ simple_string_list_destroy(&db_exclude_patterns);
+ pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
+ "--exclude-database");
+ }
+
+ if (globals_only)
+ pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
+ "-g/--globals-only");
+
+ /* Process if toc.glo file does not exist. */
+ n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_global_objects
+ *
+ * This restore all global objects.
+ */
+static int
+restore_global_objects(const char *inputFileSpec, RestoreOptions *opts)
+{
+ Archive *AH;
+ int nerror = 0;
+
+ /* Set format as custom so that toc.glo file can be read. */
+ opts->format = archCustom;
+ opts->txn_size = 0;
+
+ AH = OpenArchive(inputFileSpec, opts->format);
+
+ SetArchiveOptions(AH, NULL, opts);
+
+ on_exit_close_archive(AH);
+
+ /* Let the archiver know how noisy to be */
+ AH->verbose = opts->verbose;
+
+ /* Don't output TOC entry comments when restoring globals */
+ ((ArchiveHandle *) AH)->noTocComments = 1;
+
+ AH->exit_on_error = false;
+
+ /* Parallel execution is not supported for global object restoration. */
+ AH->numWorkers = 1;
+
+ ProcessArchiveRestoreOptions(AH);
+ RestoreArchive(AH, false);
+
+ nerror = AH->n_errors;
+
+ /* AH may be freed in CloseArchive? */
+ CloseArchive(AH);
+
+ return nerror;
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -492,9 +738,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * the previous entry was already in the array and we had closed previous
+ * connection, so we can use the same array slot.
*/
- on_exit_close_archive(AH);
+ if (!append_data)
+ on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -514,25 +766,21 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -550,6 +798,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -566,6 +815,7 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
+ printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -601,8 +851,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
+ "combined and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -707,3 +957,422 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if the file exists in the given directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimplePtrList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PQExpBuffer db_lit;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+ db_lit = createPQExpBuffer();
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ bool skip_db_restore = false;
+
+ resetPQExpBuffer(query);
+ resetPQExpBuffer(db_lit);
+
+ appendStringLiteralConn(db_lit, dbidname->str, conn);
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * If there is an exact match then we don't need to try a pattern
+ * match
+ */
+ if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ /* Otherwise, try a pattern match if there is a connection */
+ else
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, db_lit->data,
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ dbidname->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if (PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database name \"%s\" matches --exclude-database pattern \"%s\"", dbidname->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /*
+ * Mark db to be skipped or increment the counter of dbs to be
+ * restored
+ */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dbidname->str);
+ dbidname->oid = InvalidOid;
+ }
+ else
+ count_db++;
+ }
+
+ destroyPQExpBuffer(query);
+ destroyPQExpBuffer(db_lit);
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(char *dumpdirpath, SimplePtrList *dbname_oid_list)
+{
+ StringInfoData linebuf;
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ int count = 0;
+ int len;
+
+
+ /*
+ * If there is no map.dat file in dump, then return from here as there is
+ * no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
+ return 0;
+ }
+
+ len = strlen(dumpdirpath);
+
+ /* Trim slash from directory name. */
+ while (len > 1 && dumpdirpath[len - 1] == '/')
+ {
+ dumpdirpath[len - 1] = '\0';
+ len--;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open file \"%s\": %m", map_file_path);
+
+ initStringInfo(&linebuf);
+
+ /* Append all the dbname/db_oid combinations to the list. */
+ while (pg_get_line_buf(pfile, &linebuf))
+ {
+ Oid db_oid = InvalidOid;
+ char *dbname;
+ DbOidName *dbidname;
+ int namelen;
+ char *p = linebuf.data;
+
+ /* look for the dboid. */
+ while (isdigit((unsigned char) *p))
+ p++;
+
+ /* ignore lines that don't begin with a digit */
+ if (p == linebuf.data)
+ continue;
+
+ if (*p == ' ')
+ {
+ sscanf(linebuf.data, "%u", &db_oid);
+ p++;
+ }
+
+ /* dbname is the rest of the line */
+ dbname = p;
+ namelen = strlen(dbname);
+
+ /* Strip trailing newline */
+ if (namelen > 0 && dbname[namelen - 1] == '\n')
+ dbname[--namelen] = '\0';
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || namelen < 1)
+ pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
+ count + 1);
+
+ dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
+ dbidname->oid = db_oid;
+ strlcpy(dbidname->str, dbname, namelen + 1);
+
+ pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
+ dbidname->str, db_oid, map_file_path);
+
+ simple_ptr_list_append(dbname_oid_list, dbidname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ pfree(linebuf.data);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(const char *inputFileSpec,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimplePtrList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total = 0;
+ char *connected_db = NULL;
+ PGconn *conn = NULL;
+ RestoreOptions *original_opts = pg_malloc0_object(RestoreOptions);
+ RestoreOptions *tmpopts = pg_malloc0_object(RestoreOptions);
+
+ memcpy(original_opts, opts, sizeof(RestoreOptions));
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile((char *) inputFileSpec, &dbname_oid_list);
+
+ pg_log_info(ngettext("found %d database name in \"%s\"",
+ "found %d database names in \"%s\"",
+ num_total_db),
+ num_total_db, "map.dat");
+
+ /*
+ * If exclude-patterns is given, connect to the database to process them.
+ */
+ if (db_exclude_patterns.head != NULL)
+ {
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "postgres");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect to database \"%s\"", "template1");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (!conn)
+ {
+ pg_log_error("could not connect to databases \"postgres\" or \"template1\"\n"
+ "Please specify an alternative database.");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+ }
+ }
+
+ /* Filter the db list according to the exclude patterns. */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+ PQfinish(conn);
+ }
+ else
+ num_db_restore = num_total_db;
+
+ /* Exit if no db needs to be restored. */
+ if (num_db_restore == 0)
+ {
+ pg_log_info(ngettext("no database needs restoring out of %d database",
+ "no database needs restoring out of %d databases", num_total_db),
+ num_total_db);
+ pg_free(original_opts);
+ pg_free(tmpopts);
+ return 0;
+ }
+
+ pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * We have a list of databases to restore after processing the
+ * exclude-database switch(es). Now we can restore them one by one.
+ */
+ for (SimplePtrListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ DbOidName *dbidname = (DbOidName *) db_cell->ptr;
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (dbidname->oid == InvalidOid)
+ continue;
+
+ /*
+ * Since pg_backup_archiver.c may modify RestoreOptions during the
+ * previous restore, we must provide a fresh copy of the original
+ * "opts" for each call to restore_one_database.
+ */
+ memcpy(tmpopts, original_opts, sizeof(RestoreOptions));
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into an already created database. (used with -d/--dbname option)
+ */
+ if (tmpopts->cparams.override_dbname)
+ {
+ pfree(tmpopts->cparams.override_dbname);
+ tmpopts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
+
+ /*
+ * Look for the database dump file/dir. If there is an {oid}.tar or
+ * {oid}.dmp file, use it. Otherwise try to use a directory called
+ * {oid}
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", dbidname->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (tmpopts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(dbidname->str, NULL, tmpopts->cparams.pghost,
+ tmpopts->cparams.pgport, tmpopts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ tmpopts->createDB = 0;
+ tmpopts->cparams.dbname = dbidname->str;
+ }
+ else
+ {
+ /* We'll have to create it */
+ tmpopts->createDB = 1;
+ tmpopts->cparams.dbname = connected_db;
+ }
+ }
+
+ /* Restore the single database. */
+ n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true);
+
+ n_errors_total += n_errors;
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases is %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_ptr_list_destroy(&dbname_oid_list);
+
+ pg_free(original_opts);
+ pg_free(tmpopts);
+
+ return n_errors_total;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
index ab9310eb42b..a895bc314b0 100644
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -244,4 +244,59 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
+ 'pg_dumpall: unrecognized output format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--restrict-key=uu', '-f dumpfile' ],
+ qr/\Qpg_dumpall: error: option --restrict-key can only be used with --format=plain\E/,
+ 'pg_dumpall: --restrict-key can only be used with plain dump format');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd', '--globals-only', '--clean', '-f', 'dumpfile' ],
+ qr/\Qpg_dumpall: error: options --clean and -g\/--globals-only cannot be used together in non-text dump\E/,
+ 'pg_dumpall: --clean and -g/--globals-only cannot be used together in non-text dump');
+
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'd' ],
+ qr/\Qpg_dumpall: error: option -F\/--format=d|c|t requires option -f\/--file\E/,
+ 'pg_dumpall: non-plain format requires --file option');
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--data-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options -a\/--data-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options -a/--data-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--schema-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options -s\/--schema-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options -s/--schema-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--statistics-only', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: options --statistics-only and -g\/--globals-only cannot be used together\E/,
+ 'pg_restore: error: options --statistics-only and -g/--globals-only cannot be used together'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option --exclude-database can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --exclude-database is used in pg_restore with dump of pg_dump'
+);
+
+command_fails_like(
+ [ 'pg_restore', '--globals-only', '-d', 'xxx', 'dumpdir' ],
+ qr/\Qpg_restore: error: option -g\/--globals-only can be used only when restoring an archive created by pg_dumpall\E/,
+ 'When option --globals-only is used in pg_restore with the dump of pg_dump'
+);
done_testing();
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
new file mode 100644
index 00000000000..b228e572f43
--- /dev/null
+++ b/src/bin/pg_dump/t/007_pg_dumpall.pl
@@ -0,0 +1,639 @@
+# Copyright (c) 2021-2026, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+my $run_db = 'postgres';
+my $sep = $windows_os ? "\\" : "/";
+
+# Tablespace locations used by "restore_tablespace" test case.
+my $tablespace1 = "${tempdir}${sep}tbl1";
+my $tablespace2 = "${tempdir}${sep}tbl2";
+mkdir($tablespace1) || die "mkdir $tablespace1 $!";
+mkdir($tablespace2) || die "mkdir $tablespace2 $!";
+
+# escape tablespace locations on Windows.
+my $tablespace2_orig = $tablespace2;
+$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
+$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
+
+# Where pg_dumpall will be executed.
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init;
+$node->start;
+
+
+###############################################################
+# Definition of the pg_dumpall test cases to run.
+#
+# Each of these test cases are named and those names are used for fail
+# reporting and also to save the dump and restore information needed for the
+# test to assert.
+#
+# The "setup_sql" is a psql valid script that contains SQL commands to execute
+# before of actually execute the tests. The setups are all executed before of
+# any test execution.
+#
+# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
+# "restore_cmd" must have the --file flag to save the restore output so that we
+# can assert on it.
+#
+# The "like" and "unlike" is a regexp that is used to match the pg_restore
+# output. It must have at least one of then filled per test cases but it also
+# can have both. See "excluding_databases" test case for example.
+my %pgdumpall_runs = (
+ restore_roles => {
+ setup_sql => '
+ CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
+ CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_roles.sql",
+ "$tempdir/restore_roles",
+ ],
+ like => qr/
+ \s*\QCREATE ROLE dumpall2;\E
+ \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
+ /xm
+ },
+
+ restore_tablespace => {
+ setup_sql => "
+ CREATE ROLE tap;
+ CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
+ CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_tablespace.sql",
+ "$tempdir/restore_tablespace",
+ ],
+ # Match "E" as optional since it is added on LOCATION when running on
+ # Windows.
+ like => qr/^
+ \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2_orig';\E
+ \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
+ /xm,
+ },
+
+ restore_grants => {
+ setup_sql => "
+ CREATE DATABASE tapgrantsdb;
+ CREATE SCHEMA private;
+ CREATE SEQUENCE serial START 101;
+ CREATE FUNCTION fn() RETURNS void AS \$\$
+ BEGIN
+ END;
+ \$\$ LANGUAGE plpgsql;
+ CREATE ROLE super;
+ CREATE ROLE grant1;
+ CREATE ROLE grant2;
+ CREATE ROLE grant3;
+ CREATE ROLE grant4;
+ CREATE ROLE grant5;
+ CREATE ROLE grant6;
+ CREATE ROLE grant7;
+ CREATE ROLE grant8;
+
+ CREATE TABLE t (id int);
+ INSERT INTO t VALUES (1), (2), (3), (4);
+
+ GRANT SELECT ON TABLE t TO grant1;
+ GRANT INSERT ON TABLE t TO grant2;
+ GRANT ALL PRIVILEGES ON TABLE t to grant3;
+ GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
+ GRANT USAGE, CREATE ON SCHEMA private TO grant5;
+ GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
+ GRANT super TO grant7;
+ GRANT EXECUTE ON FUNCTION fn() TO grant8;
+ ",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/restore_grants.sql",
+ "$tempdir/restore_grants",
+ ],
+ like => qr/^
+ \n\QGRANT ALL ON SCHEMA private TO grant5;\E
+ (.*\n)*
+ \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
+ (.*\n)*
+ \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
+ (.*\n)*
+ \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
+ \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
+ \n\QGRANT ALL ON TABLE public.t TO grant3;\E
+ (.*\n)*
+ \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
+ /xm,
+ },
+
+ excluding_databases => {
+ setup_sql => 'CREATE DATABASE db1;
+ \c db1
+ CREATE TABLE t1 (id int);
+ INSERT INTO t1 VALUES (1), (2), (3), (4);
+ CREATE TABLE t2 (id int);
+ INSERT INTO t2 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db2;
+ \c db2
+ CREATE TABLE t3 (id int);
+ INSERT INTO t3 VALUES (1), (2), (3), (4);
+ CREATE TABLE t4 (id int);
+ INSERT INTO t4 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex3;
+ \c dbex3
+ CREATE TABLE t5 (id int);
+ INSERT INTO t5 VALUES (1), (2), (3), (4);
+ CREATE TABLE t6 (id int);
+ INSERT INTO t6 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE dbex4;
+ \c dbex4
+ CREATE TABLE t7 (id int);
+ INSERT INTO t7 VALUES (1), (2), (3), (4);
+ CREATE TABLE t8 (id int);
+ INSERT INTO t8 VALUES (1), (2), (3), (4);
+
+ CREATE DATABASE db5;
+ \c db5
+ CREATE TABLE t9 (id int);
+ INSERT INTO t9 VALUES (1), (2), (3), (4);
+ CREATE TABLE t10 (id int);
+ INSERT INTO t10 VALUES (1), (2), (3), (4);
+ ',
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases",
+ '--exclude-database' => 'dbex*',
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/excluding_databases.sql",
+ '--exclude-database' => 'db5',
+ "$tempdir/excluding_databases",
+ ],
+ like => qr/^
+ \n\QCREATE DATABASE db1\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t1 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t2 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db2\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t3 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t4 (/xm,
+ unlike => qr/^
+ \n\QCREATE DATABASE db3\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t5 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t6 (\E
+ (.*\n)*
+ \n\QCREATE DATABASE db4\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t7 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t8 (\E
+ \n\QCREATE DATABASE db5\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t9 (\E
+ (.*\n)*
+ \n\QCREATE TABLE public.t10 (\E
+ /xm,
+ },
+
+ format_directory => {
+ setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
+ INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/format_directory.sql",
+ "$tempdir/format_directory",
+ ],
+ like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
+ },
+
+ format_tar => {
+ setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
+ INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'tar',
+ '--file' => "$tempdir/format_tar.sql",
+ "$tempdir/format_tar",
+ ],
+ like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
+ },
+
+ format_custom => {
+ setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
+ INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--file' => "$tempdir/format_custom.sql",
+ "$tempdir/format_custom",
+ ],
+ like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
+ },
+
+ dump_globals_only => {
+ setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
+ INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
+ dump_cmd => [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--globals-only',
+ '--file' => "$tempdir/dump_globals_only",
+ ],
+ restore_cmd => [
+ 'pg_restore', '-C', '--globals-only',
+ '--format' => 'directory',
+ '--file' => "$tempdir/dump_globals_only.sql",
+ "$tempdir/dump_globals_only",
+ ],
+ like => qr/
+ ^\s*\QCREATE ROLE dumpall;\E\s*\n
+ /xm
+ },);
+
+# First execute the setup_sql
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ if ($pgdumpall_runs{$run}->{setup_sql})
+ {
+ $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
+ }
+}
+
+# Execute the tests
+foreach my $run (sort keys %pgdumpall_runs)
+{
+ # Create a new target cluster to pg_restore each test case run so that we
+ # don't need to take care of the cleanup from the target cluster after each
+ # run.
+ my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
+ $target_node->init;
+ $target_node->start;
+
+ # Dumpall from node cluster.
+ $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
+ "$run: pg_dumpall runs");
+
+ # Restore the dump on "target_node" cluster.
+ my @restore_cmd = (
+ @{ $pgdumpall_runs{$run}->{restore_cmd} },
+ '--host', $target_node->host, '--port', $target_node->port);
+
+ my ($stdout, $stderr) = run_command(\@restore_cmd);
+
+ # pg_restore --file output file.
+ my $output_file = slurp_file("$tempdir/${run}.sql");
+
+ if ( !($pgdumpall_runs{$run}->{like})
+ && !($pgdumpall_runs{$run}->{unlike}))
+ {
+ die "missing \"like\" or \"unlike\" in test \"$run\"";
+ }
+
+ if ($pgdumpall_runs{$run}->{like})
+ {
+ like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
+ }
+
+ if ($pgdumpall_runs{$run}->{unlike})
+ {
+ unlike(
+ $output_file,
+ $pgdumpall_runs{$run}->{unlike},
+ "should not dump $run");
+ }
+}
+
+# Some negative test case with dump of pg_dumpall and restore using pg_restore
+# report an error when -C is not used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom",
+ '--format' => 'custom',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
+ 'When -C is not used in pg_restore with dump of pg_dumpall');
+
+# report an error when \l/--list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--list',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --list is used in pg_restore with dump of pg_dumpall');
+
+# report an error when -L/--use-list option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--use-list' => 'use',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -L\/--use-list cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When -L/--use-list is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --strict-names option is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--strict-names',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --strict-names cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --strict-names is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --clean and -g/--globals-only are used in pg_restore with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--clean',
+ '--globals-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --clean and -g\/--globals-only cannot be used together when restoring an archive created by pg_dumpall\E/,
+ 'When --clean and -g/--globals-only are used in pg_restore with dump of pg_dumpall'
+);
+
+# report an error when non-exist database is given with -d option
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '-d' => 'dbpq',
+ ],
+ qr/\QFATAL: database "dbpq" does not exist\E/,
+ 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
+);
+
+# report an error when --no-schema is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--no-schema',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --no-schema cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --no-schema is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --data-only is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--data-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option -a\/--data-only cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --data-only is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --statistics-only is used with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--statistics-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --statistics-only cannot be used when restoring an archive created by pg_dumpall\E/,
+ 'When --statistics-only is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --section excludes pre-data with dump of pg_dumpall
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--section' => 'post-data',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: option --section cannot exclude --pre-data when restoring a pg_dumpall archive\E/,
+ 'When --section=post-data is used in pg_restore with dump of pg_dumpall');
+
+# report an error when --globals-only and --data-only are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--data-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options -a\/--data-only and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --data-only are used together');
+
+# report an error when --globals-only and --schema-only are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--schema-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options -s\/--schema-only and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --schema-only are used together');
+
+# report an error when --globals-only and --statistics-only are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--statistics-only',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --statistics-only and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --statistics-only are used together');
+
+# report an error when --globals-only and --statistics are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--statistics',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --statistics and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --statistics are used together');
+
+# report an error when --globals-only and --exit-on-error are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--exit-on-error',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --exit-on-error and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --exit-on-error are used together');
+
+# report an error when --globals-only and --single-transaction are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--single-transaction',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --single-transaction and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --single-transaction are used together');
+
+# report an error when --globals-only and --transaction-size are used together
+$node->command_fails_like(
+ [
+ 'pg_restore',
+ "$tempdir/format_custom", '-C',
+ '--format' => 'custom',
+ '--globals-only',
+ '--transaction-size' => '100',
+ '--file' => "$tempdir/error_test.sql",
+ ],
+ qr/\Qpg_restore: error: options --transaction-size and -g\/--globals-only cannot be used together\E/,
+ 'When --globals-only and --transaction-size are used together');
+
+# verify map.dat preamble exists
+my $map_dat_content = slurp_file("$tempdir/format_directory/map.dat");
+like(
+ $map_dat_content,
+ qr/^# map\.dat\n.*# This file maps oids to database names/ms,
+ 'map.dat contains expected preamble');
+
+# verify commenting out a line in map.dat skips that database
+$node->safe_psql($run_db, 'CREATE DATABASE comment_test_db;
+\c comment_test_db
+CREATE TABLE comment_test_table (id int);');
+
+$node->command_ok(
+ [
+ 'pg_dumpall',
+ '--format' => 'directory',
+ '--file' => "$tempdir/comment_test",
+ ],
+ 'pg_dumpall for comment test');
+
+# Modify map.dat to comment out the comment_test_db entry
+my $map_content = slurp_file("$tempdir/comment_test/map.dat");
+$map_content =~ s/^(\d+ comment_test_db)$/# $1/m;
+open(my $fh, '>', "$tempdir/comment_test/map.dat")
+ or die "Cannot open map.dat: $!";
+print $fh $map_content;
+close($fh);
+
+# Create a target node and restore - commented db should be skipped
+my $target_comment = PostgreSQL::Test::Cluster->new("target_comment");
+$target_comment->init;
+$target_comment->start;
+
+$node->command_ok(
+ [
+ 'pg_restore', '-C',
+ '--format' => 'directory',
+ '--file' => "$tempdir/comment_test_restore.sql",
+ '--host', $target_comment->host,
+ '--port', $target_comment->port,
+ "$tempdir/comment_test",
+ ],
+ 'pg_restore with commented out database in map.dat');
+
+my $restore_output = slurp_file("$tempdir/comment_test_restore.sql");
+unlike(
+ $restore_output,
+ qr/CREATE DATABASE comment_test_db/,
+ 'commented out database in map.dat is not restored');
+
+# Test that --clean implies --if-exists for pg_dumpall archives
+$node->command_ok(
+ [
+ 'pg_restore', '-C',
+ '--format' => 'custom',
+ '--clean',
+ '--file' => "$tempdir/clean_test.sql",
+ "$tempdir/format_custom",
+ ],
+ 'pg_restore with --clean on pg_dumpall archive');
+
+my $clean_output = slurp_file("$tempdir/clean_test.sql");
+like(
+ $clean_output,
+ qr/DROP ROLE IF EXISTS/,
+ '--clean implies --if-exists: DROP ROLE IF EXISTS in output');
+
+$node->stop('fast');
+
+done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..1a89ef94bec 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -600,6 +600,7 @@ CustomScanMethods
CustomScanState
CycleCtr
DBState
+DbOidName
DCHCacheEntry
DEADLOCK_INFO
DECountItem
--
2.43.0
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-26 14:02 Andrew Dunstan <[email protected]>
parent: Andrew Dunstan <[email protected]>
2 siblings, 2 replies; 22+ messages in thread
From: Andrew Dunstan @ 2026-02-26 14:02 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On 2026-02-24 Tu 5:40 AM, Andrew Dunstan wrote:
>
> On 2026-02-24 Tu 2:21 AM, jian he wrote:
>> Hi.
>>
>> For v19, the commit message
>> """
>> pg_restore is extended to handle these pg_dumpall archives, restoring
>> globals and then each database. The --globals-only and --no-globals
>> options control which parts are restored.
>> """
>> There is no --no-globals option.
>>
>> In file src/bin/pg_dump/pg_dumpall.c, no need
>> ``
>> static pg_compress_specification compression_spec = {0};
>> ``
>> Since compression_spec is only used in an IF branch, we can declare
>> it locally.
>>
>>
>> The options below are not supported for pg_restore non-text restore,
>> we can document this.
>> <option>-a/--data-only</option>,
>> <option>-l/--list</option>,
>> <option>-L/--use-list</option>,
>> <option>--statistics-only</option>,
>> <option>--strict-names</option>,
>> <option>--no-schema</option>.
>>
>>
>>
>
> Thanks, fixed these.
>
>
>
pushed with a slight tweak.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-26 19:11 Mahendra Singh Thalor <[email protected]>
parent: Andrew Dunstan <[email protected]>
1 sibling, 0 replies; 22+ messages in thread
From: Mahendra Singh Thalor @ 2026-02-26 19:11 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: jian he <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On Thu, 26 Feb 2026 at 19:32, Andrew Dunstan <[email protected]> wrote:
>
>
> On 2026-02-24 Tu 5:40 AM, Andrew Dunstan wrote:
> >
> > On 2026-02-24 Tu 2:21 AM, jian he wrote:
> >> Hi.
> >>
> >> For v19, the commit message
> >> """
> >> pg_restore is extended to handle these pg_dumpall archives, restoring
> >> globals and then each database. The --globals-only and --no-globals
> >> options control which parts are restored.
> >> """
> >> There is no --no-globals option.
> >>
> >> In file src/bin/pg_dump/pg_dumpall.c, no need
> >> ``
> >> static pg_compress_specification compression_spec = {0};
> >> ``
> >> Since compression_spec is only used in an IF branch, we can declare
> >> it locally.
> >>
> >>
> >> The options below are not supported for pg_restore non-text restore,
> >> we can document this.
> >> <option>-a/--data-only</option>,
> >> <option>-l/--list</option>,
> >> <option>-L/--use-list</option>,
> >> <option>--statistics-only</option>,
> >> <option>--strict-names</option>,
> >> <option>--no-schema</option>.
> >>
> >>
> >>
> >
> > Thanks, fixed these.
> >
> >
> >
>
>
> pushed with a slight tweak.
>
>
> cheers
>
>
> andrew
>
>
> --
> Andrew Dunstan
> EDB: https://www.enterprisedb.com
>
Thanks Andrew for committing this patch.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-26 21:52 Tom Lane <[email protected]>
parent: Andrew Dunstan <[email protected]>
2 siblings, 1 reply; 22+ messages in thread
From: Tom Lane @ 2026-02-26 21:52 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: jian he <[email protected]>; Mahendra Singh Thalor <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
Andrew Dunstan <[email protected]> writes:
> pushed with a slight tweak.
BF member pollock isn't pleased with 007_pg_dumpall:
026-02-26 20:18:10.389 UTC [14469:1] FATAL: could not create semaphores: No space left on device
2026-02-26 20:18:10.389 UTC [14469:2] DETAIL: Failed system call was semget(668039, 17, 03600).
2026-02-26 20:18:10.389 UTC [14469:3] HINT: This error does *not* mean that you have run out of disk space. It occurs when either the system limit for the maximum number of semaphore sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded. You need to raise the respective kernel parameter. Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its "max_connections" parameter.
It looks to me like this is happening because the script creates a
boatload of postmasters and doesn't bother to shut any of them down
(until that happens implicitly at script end). That seems rather
unfriendly to small BF machines in the first place, as well as for
installations that might try to run multiple TAP scripts in parallel.
It's probably eating an undue amount of disk space, as well.
Is there a reason why the "foreach my $run (sort keys %pgdumpall_runs)"
loop leaves the $target_nodes running, instead of cleaning each one
up at the bottom of the loop?
regards, tom lane
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-02-27 12:12 Andrew Dunstan <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 22+ messages in thread
From: Andrew Dunstan @ 2026-02-27 12:12 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: jian he <[email protected]>; Mahendra Singh Thalor <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On 2026-02-26 Th 4:52 PM, Tom Lane wrote:
> https://www.postgresql.org/message-id/2fb787be-79f2-4161-8ba4-24e8cab019ac%40dunslane.netAndrew Dunstan<[email protected]> writes:
>> pushed with a slight tweak.
> BF member pollock isn't pleased with 007_pg_dumpall:
>
> 026-02-26 20:18:10.389 UTC [14469:1] FATAL: could not create semaphores: No space left on device
> 2026-02-26 20:18:10.389 UTC [14469:2] DETAIL: Failed system call was semget(668039, 17, 03600).
> 2026-02-26 20:18:10.389 UTC [14469:3] HINT: This error does *not* mean that you have run out of disk space. It occurs when either the system limit for the maximum number of semaphore sets (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded. You need to raise the respective kernel parameter. Alternatively, reduce PostgreSQL's consumption of semaphores by reducing its "max_connections" parameter.
>
> It looks to me like this is happening because the script creates a
> boatload of postmasters and doesn't bother to shut any of them down
> (until that happens implicitly at script end). That seems rather
> unfriendly to small BF machines in the first place, as well as for
> installations that might try to run multiple TAP scripts in parallel.
>
> It's probably eating an undue amount of disk space, as well.
>
> Is there a reason why the "foreach my $run (sort keys %pgdumpall_runs)"
> loop leaves the $target_nodes running, instead of cleaning each one
> up at the bottom of the loop?
No, I think it's just an oversight. I'll add this at the bottom of the loop:
+ $target_node->stop;
+ $target_node->clean_node;
More generally, perhaps we should have a DESTROY method in Cluster.pm
that would call the stop method when an object goes out of scope.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-03-03 11:17 Mahendra Singh Thalor <[email protected]>
parent: Andrew Dunstan <[email protected]>
2 siblings, 1 reply; 22+ messages in thread
From: Mahendra Singh Thalor @ 2026-03-03 11:17 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; jian he <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On Tue, 3 Mar 2026 at 14:55, Peter Eisentraut <[email protected]> wrote:
>
> I noticed this cast in the committed code:
>
> > + num_total_db = get_dbname_oid_list_from_mfile((char *) inputFileSpec,
> > &dbname_oid_list);
>
> The cast drops the const qualifier from inputFileSpec.
> get_dbname_oid_list_from_mfile() writes into the space pointed to by its
> argument, so it's really not "const". (And inputFileSpec points into
> argv, so this ends up writing directly into argv.)
>
> Please see if you can clean this up. It might be best if
> get_dbname_oid_list_from_mfile() made a copy of its argument that it can
> write into, and then the argument can be "const".
>
Thanks Peter.
Here, I am attaching a patch to fix this issue.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[text/x-patch] v01_pg_restore-don-t-edit-inputfile-name-instead-use-local.patch (2.1K, 2-v01_pg_restore-don-t-edit-inputfile-name-instead-use-local.patch)
download | inline diff:
From cb744c481ff045a95f11d2a0217aba928ede22db Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Tue, 3 Mar 2026 16:43:33 +0530
Subject: [PATCH] pg_restore: don't edit inputfile name, instead use local copy
In get_dbname_oid_list_from_mfile, we are editing filename by passing is
char *, instead of const char*. Fixed this by adding local copy.
---
src/bin/pg_dump/pg_restore.c | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 14d886fc86e..54c8b9d48b0 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -68,7 +68,7 @@ static int restore_all_databases(const char *inputFileSpec,
static int get_dbnames_list_to_restore(PGconn *conn,
SimplePtrList *dbname_oid_list,
SimpleStringList db_exclude_patterns);
-static int get_dbname_oid_list_from_mfile(char *dumpdirpath,
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpatharg,
SimplePtrList *dbname_oid_list);
/*
@@ -1082,14 +1082,18 @@ get_dbnames_list_to_restore(PGconn *conn,
* Returns, total number of database names in map.dat file.
*/
static int
-get_dbname_oid_list_from_mfile(char *dumpdirpath, SimplePtrList *dbname_oid_list)
+get_dbname_oid_list_from_mfile(const char *dumpdirpatharg, SimplePtrList *dbname_oid_list)
{
StringInfoData linebuf;
FILE *pfile;
char map_file_path[MAXPGPATH];
int count = 0;
int len;
+ char *dumpdirpath;
+ len = strlen(dumpdirpatharg);
+ dumpdirpath = pg_malloc0(len + 1);
+ memcpy(dumpdirpath, dumpdirpatharg, len);
/*
* If there is no map.dat file in dump, then return from here as there is
@@ -1206,7 +1210,7 @@ restore_all_databases(const char *inputFileSpec,
if (opts->cparams.dbname)
connected_db = opts->cparams.dbname;
- num_total_db = get_dbname_oid_list_from_mfile((char *) inputFileSpec, &dbname_oid_list);
+ num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
pg_log_info(ngettext("found %d database name in \"%s\"",
"found %d database names in \"%s\"",
--
2.52.0
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-03-04 20:54 Andrew Dunstan <[email protected]>
parent: Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Andrew Dunstan @ 2026-03-04 20:54 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; Peter Eisentraut <[email protected]>; +Cc: jian he <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On 2026-03-03 Tu 6:17 AM, Mahendra Singh Thalor wrote:
> On Tue, 3 Mar 2026 at 14:55, Peter Eisentraut <[email protected]> wrote:
>> I noticed this cast in the committed code:
>>
>>> + num_total_db = get_dbname_oid_list_from_mfile((char *) inputFileSpec,
>>> &dbname_oid_list);
>> The cast drops the const qualifier from inputFileSpec.
>> get_dbname_oid_list_from_mfile() writes into the space pointed to by its
>> argument, so it's really not "const". (And inputFileSpec points into
>> argv, so this ends up writing directly into argv.)
>>
>> Please see if you can clean this up. It might be best if
>> get_dbname_oid_list_from_mfile() made a copy of its argument that it can
>> write into, and then the argument can be "const".
>>
> Thanks Peter.
>
> Here, I am attaching a patch to fix this issue.
Pushed with a slight tweak.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-03-22 08:50 Andrew Dunstan <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 0 replies; 22+ messages in thread
From: Andrew Dunstan @ 2026-03-22 08:50 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Peter Eisentraut <[email protected]>; jian he <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On Sat, Mar 21, 2026 at 5:10 PM Tom Lane <[email protected]> wrote:
> Andrew Dunstan <[email protected]> writes:
> > Pushed with a slight tweak.
>
> The CF entry for this is still open:
>
> https://commitfest.postgresql.org/patch/5495/
>
> Shouldn't it get closed now?
>
>
>
Yes, sorry, done.
cheers
andrew
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-06-07 00:02 Noah Misch <[email protected]>
parent: Andrew Dunstan <[email protected]>
1 sibling, 1 reply; 22+ messages in thread
From: Noah Misch @ 2026-06-07 00:02 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: jian he <[email protected]>; Mahendra Singh Thalor <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On Thu, Feb 26, 2026 at 09:02:48AM -0500, Andrew Dunstan wrote:
> pushed with a slight tweak.
Having now reviewed commit 763aaa0, I don't think it's ready to remain part of
v19. While some points from my v18 review are now resolved, other points
still seem unresolved. I didn't find discussion of the unresolved points. I
also see new issues.
Wider issues:
> @@ -796,24 +964,45 @@ dropRoles(PGconn *conn)
> + if (archDumpFormat == archNull)
> + {
> + appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
> + if_exists ? "IF EXISTS " : "",
> + fmtId(rolename));
> + fprintf(OPF, "%s", delQry->data);
> + }
> + else
> + {
> + appendPQExpBuffer(delQry, "DROP ROLE IF EXISTS %s;\n",
> + fmtId(rolename));
> +
> + ArchiveEntry(fout,
> + nilCatalogId, /* catalog ID */
> + createDumpId(), /* dump ID */
> + ARCHIVE_OPTS(.tag = psprintf("ROLE %s", fmtId(rolename)),
> + .description = "DROP_GLOBAL",
> + .section = SECTION_PRE_DATA,
> + .createStmt = delQry->data));
> + }
pg_dumpall.c should call ArchiveEntry() in the same way pg_dump.c does. In
pg_dump.c, per-object-class support code calls ArchiveEntry() unconditionally,
and the object-independent infra of pg_backup_archiver.c deals with the
difference between plain and non-plain formats.
There should be one appendPQExpBuffer(delQry, "DROP ROLE ..."), not one for
plain format and another for non-plain formats. Having two creates excess
risk of format-specific bugs, something pg_dump.c has long avoided well. (I'm
echoing my postgr.es/m/[email protected] review. I wrote,
"The strength of the archiver architecture shows in how rarely new features
need format-specific logic and how rarely format-specific bugs get reported."
That holds for the way pg_dump.c uses the archiver, but it doesn't hold for
the way pg_dumpall.c now uses the archiver.)
Separately, DROP should be in dropStmt, not in createStmt. This likely
entails refactoring to merge dumpRoles() and dropRoles() into one function,
per the style of pg_dump.c.
> + /*
> + * For pg_dumpall archives, --clean implies --if-exists since global
> + * objects may not exist in the target cluster.
> + */
> + if (opts->dropSchema && !opts->if_exists)
> + {
> + opts->if_exists = 1;
> + pg_log_info("--if-exists is implied by --clean for pg_dumpall archives");
> + }
The last comment of postgr.es/m/[email protected] disagreed
with this decision, so that remains unresolved.
> + /* If database is already created, then don't set createDB flag. */
> + if (tmpopts->cparams.dbname)
> + {
> + PGconn *test_conn;
> +
> + test_conn = ConnectDatabase(dbidname->str, NULL, tmpopts->cparams.pghost,
> + tmpopts->cparams.pgport, tmpopts->cparams.username, TRI_DEFAULT,
> + false, progname, NULL, NULL, NULL, NULL);
> + if (test_conn)
> + {
> + PQfinish(test_conn);
> +
> + /* Use already created database for connection. */
> + tmpopts->createDB = 0;
> + tmpopts->cparams.dbname = dbidname->str;
> + }
> + else
> + {
> + /* We'll have to create it */
> + tmpopts->createDB = 1;
> + tmpopts->cparams.dbname = connected_db;
> + }
postgr.es/m/[email protected] called for this to change.
None of this is meant to say the feature is impossible. But I don’t think this
commit is at the point where post-commit fixups are the right workflow. I
recommend reverting, posting a new version, and letting commitfest review
finish.
Localized issues:
There's new code to write map and globals files, but I don't see corresponding
code to fsync those files, like we fsync a plain-format dump.
> @@ -3027,6 +3049,16 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
> return 0;
> }
>
> + /*
> + * Global object TOC entries (e.g., ROLEs or TABLESPACEs) must not be
> + * ignored.
> + */
> + if (strcmp(te->desc, "ROLE") == 0 ||
> + strcmp(te->desc, "ROLE PROPERTIES") == 0 ||
> + strcmp(te->desc, "TABLESPACE") == 0 ||
> + strcmp(te->desc, "DROP_GLOBAL") == 0)
> + return REQ_SCHEMA;
> +
If I delete this addition, no src/bin/pg_dump test fails. Would you explain
the rationale for this addition?
> + if (comment_buf->data[0] != '\0')
> + ArchiveEntry(fout,
> + nilCatalogId, /* catalog ID */
> + createDumpId(), /* dump ID */
> + ARCHIVE_OPTS(.tag = tag,
> + .description = "COMMENT",
> + .section = SECTION_PRE_DATA,
> + .createStmt = comment_buf->data));
> +
> + if (seclabel_buf->data[0] != '\0')
> + ArchiveEntry(fout,
> + nilCatalogId, /* catalog ID */
> + createDumpId(), /* dump ID */
> + ARCHIVE_OPTS(.tag = tag,
> + .description = "SECURITY LABEL",
> + .section = SECTION_PRE_DATA,
> + .createStmt = seclabel_buf->data));
COMMENT and SECURITY LABEL should use .deps and .nDeps to record a dependency
on the main object. Representative example from pg_dump.c:
ArchiveEntry(fout, nilCatalogId, createDumpId(),
ARCHIVE_OPTS(.tag = target->data,
.namespace = tbinfo->dobj.namespace->dobj.name,
.owner = tbinfo->rolname,
.description = "SECURITY LABEL",
.section = SECTION_NONE,
.createStmt = query->data,
.deps = &(tbinfo->dobj.dumpId),
.nDeps = 1));
This probably has no user-visible consequences, since "/* Parallel execution
is not supported for global object restoration. */". Still, best to follow
the standard.
> + /*
> + * If this is not a plain format dump, then append dboid and dbname to
> + * the map.dat file.
> + */
The first six lines inside the block aren't for the purpose described here.
The one line that is for this purpose has its own comment. Please delete this
comment.
> + if (archDumpFormat != archNull)
> + {
> + if (archDumpFormat == archCustom)
> + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
> + else if (archDumpFormat == archTar)
> + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
> + else
> + snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
My review in postgr.es/m/[email protected] called for this
to change. I'm finding no discussion of the rationale for not changing it.
> +
> + /* Put one line entry for dboid and dbname in map file. */
> + fprintf(map_file, "%s %s\n", oid, dbname);
> + }
> +runPgDump(const char *dbname, const char *create_opts, char *dbfile)
> + if (archDumpFormat != archNull)
> + {
> + printfPQExpBuffer(&cmd, "\"%s\" %s -f %s %s", pg_dump_bin,
> + pgdumpopts->data, dbfile, create_opts);
A different "-f" argument is already in pgdumpopts. That works, but it's
untidy.
> + pg_fatal("option %s cannot exclude %s when restoring a pg_dumpall archive",
> + "--section", "--pre-data");
s/--pre-data/pre-data/
> + /*
> + * Always restore global objects, even if --exclude-database results
> + * in zero databases to process. If 'globals-only' is set, exit
> + * immediately.
I think this comment is out of date, because the code doesn't have an exit:
> + */
> + snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
> +
> + n_errors = restore_global_objects(global_path, tmpopts);
> +
> + if (globals_only)
> + pg_log_info("database restoring skipped because option %s was specified",
> + "-g/--globals-only");
> + else
> + {
> + /* Now restore all the databases from map.dat */
> + n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns,
> + opts, numWorkers);
> + }
> +
> + /* Free db pattern list. */
> + simple_string_list_destroy(&db_exclude_patterns);
> + }
> + /* Don't output TOC entry comments when restoring globals */
> + ((ArchiveHandle *) AH)->noTocComments = 1;
Why not?
> +static int
> +get_dbnames_list_to_restore(PGconn *conn,
> + SimplePtrList *dbname_oid_list,
> + SimpleStringList db_exclude_patterns)
> +{
...
> + if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
> + skip_db_restore = true;
> + /* Otherwise, try a pattern match if there is a connection */
The code assumes a connection unconditionally (which seems right to me):
> + else
> + {
> + int dotcnt;
> +
> + appendPQExpBufferStr(query, "SELECT 1 ");
> + processSQLNamePattern(conn, query, pat_cell->val, false,
> + false, NULL, db_lit->data,
> + NULL, NULL, NULL, &dotcnt);
> +
> + if (dotcnt > 0)
> + {
> + pg_log_error("improper qualified name (too many dotted names): %s",
> + dbidname->str);
> + PQfinish(conn);
> + exit_nicely(1);
> + }
> +
> + res = executeQuery(conn, query->data);
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-06-15 20:11 Andrew Dunstan <[email protected]>
parent: Noah Misch <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Andrew Dunstan @ 2026-06-15 20:11 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: jian he <[email protected]>; Mahendra Singh Thalor <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On 2026-06-06 Sa 8:02 PM, Noah Misch wrote:
> On Thu, Feb 26, 2026 at 09:02:48AM -0500, Andrew Dunstan wrote:
>> pushed with a slight tweak.
> Having now reviewed commit 763aaa0, I don't think it's ready to remain part of
> v19. While some points from my v18 review are now resolved, other points
> still seem unresolved. I didn't find discussion of the unresolved points. I
> also see new issues.
OK, here's a reversion path. It's a bit messy, and I didn't touch the
release notes, but apart from that I think it does the right thing.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
Attachments:
[text/x-patch] 0001-Revert-non-text-pg_dumpall.patch (127.3K, 2-0001-Revert-non-text-pg_dumpall.patch)
download | inline diff:
From fe6da46afb016d9dcdf33051051856ae630bb74d Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email protected]>
Date: Mon, 15 Jun 2026 15:58:04 -0400
Subject: [PATCH] Revert non-text output formats for pg_dumpall
This reverts the non-text (custom/directory/tar) output format support
for pg_dumpall added by 763aaa06f03 and its feature-specific follow-ups,
in line with Noah Misch's post-commit review which recommends reverting
and finishing the work through the commitfest.
Scope is deliberately minimal: only the feature itself is removed.
Independent improvements that merely touched the same files, or that were
committed alongside the feature but do not depend on its design, are
preserved.
Reverted (the feature):
763aaa06f03 Add non-text output formats to pg_dumpall
d6d9b96b404 Clean up nodes that are no longer of use in 007_pgdumpall.pl
01c729e0c7a Fix casting away const-ness in pg_restore.c
c7572cd48d3 Improve writing map.dat preamble
3c19983cc08 pg_restore: add --no-globals option to skip globals
abff4492d02 Fix options listing of pg_restore --no-globals
bb53b8d359d Fix small memory leak in get_dbname_oid_list_from_mfile()
a793677e57b pg_restore: Remove dead code in restore_all_databases()
a198c26dede pg_dumpall: simplify coding of dropDBs()
ec80215c033 pg_restore: Remove unnecessary strlen() calls in options parsing
Preserved (independent of the feature):
b2898baaf7e the check_mut_excl_opts() helper in src/fe_utils/option_utils.c
and its use in pg_dump
7c8280eeb58 pg_dump's conflicting-option refactor (and tests 002/005)
be0d0b457cb pg_dumpall's rejection of --clean together with --data-only
(re-expressed directly, since pg_dumpall.c is otherwise
returned to its pre-feature state)
74b4438a70b the dangling-grantor-OID GRANT fix (back-patched through 16)
273d26b75e7, d4cb9c37765 independent pg_restore.sgml clarifications
Because the feature restructured pg_dumpall.c and pg_restore.c (pg_restore's
main() was split into restore_one_database() plus a dispatcher) and
interleaved its option checks with the conflicting-option refactor in the
same regions, the cosmetic check_mut_excl_opts() reflow of those two files'
option blocks is inseparable from the feature and comes out with it; the
behavior is unchanged. The reusable helper and pg_dump's use of it are
unaffected.
Discussion: https://postgr.es/m/[email protected]
---
doc/src/sgml/ref/pg_dumpall.sgml | 135 +--
doc/src/sgml/ref/pg_restore.sgml | 129 +--
src/bin/pg_dump/meson.build | 1 -
src/bin/pg_dump/parallel.c | 14 -
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 67 +-
src/bin/pg_dump/pg_backup_archiver.h | 1 -
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 863 ++++----------------
src/bin/pg_dump/pg_restore.c | 776 ++----------------
src/bin/pg_dump/t/001_basic.pl | 112 +--
src/bin/pg_dump/t/005_pg_dump_filterfile.pl | 4 +-
src/bin/pg_dump/t/007_pg_dumpall.pl | 661 ---------------
src/tools/pgindent/typedefs.list | 1 -
15 files changed, 275 insertions(+), 2495 deletions(-)
delete mode 100644 src/bin/pg_dump/t/007_pg_dumpall.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 51c70198091..8834b7ec141 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,10 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
-
- <refpurpose>
- export a <productname>PostgreSQL</productname> database cluster as an SQL script or to other formats
- </refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -36,33 +33,14 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into an SQL script file or an archive. It does this by
+ of a cluster into one script file. The script file contains
+ <acronym>SQL</acronym> commands that can be used as input to <xref
+ linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
- The output contains <acronym>SQL</acronym> commands that can be used
- as input to <xref linkend="app-psql"/> or <xref linkend="app-pgrestore"/>
- to restore the databases.
<application>pg_dumpall</application> also dumps global objects
that are common to all databases, namely database roles, tablespaces,
and privilege grants for configuration parameters.
(<application>pg_dump</application> does not save these objects.)
- The only parts of a database cluster's state that
- are <emphasis>not</emphasis> included in the default output
- of <application>pg_dumpall</application> are the configuration files
- and any database parameter setting changes made with
- <xref linkend="sql-altersystem"/>.
- </para>
-
- <para>
- If the output format is a
- plain text SQL script, it will be written to the standard output. Use the
- <option>-f</option>/<option>--file</option> option or shell operators to
- redirect it into a file.
- </para>
-
- <para>
- If another output format is selected, the archive will be placed in a
- directory named using the <option>-f</option>/<option>--file</option>
- option, which is required in this case.
</para>
<para>
@@ -73,6 +51,12 @@ PostgreSQL documentation
allowed to add roles and create databases.
</para>
+ <para>
+ The SQL script will be written to the standard output. Use the
+ <option>-f</option>/<option>--file</option> option or shell operators to
+ redirect it into a file.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -147,93 +131,16 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- This option can only be omitted when <option>--format</option> is plain.
</para>
</listitem>
</varlistentry>
- <varlistentry>
- <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
- <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
- <listitem>
- <para>
- Specify the format of dump files. In plain format, all the dump data is
- sent in a single text stream. This is the default.
-
- In all other modes, <application>pg_dumpall</application> first creates two files,
- <filename>toc.glo</filename> and <filename>map.dat</filename>, in the directory
- specified by <option>--file</option>.
- The first file contains global data (roles and tablespaces) in custom format. The second
- contains a mapping between database OIDs and names. These files are used by
- <application>pg_restore</application>. Data for individual databases is placed in
- the <filename>databases</filename> subdirectory, named using the database's OID.
-
- <variablelist>
- <varlistentry>
- <term><literal>d</literal></term>
- <term><literal>directory</literal></term>
- <listitem>
- <para>
- Output directory-format archives for each database,
- suitable for input into pg_restore. The directory
- will have database <type>oid</type> as its name.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><literal>p</literal></term>
- <term><literal>plain</literal></term>
- <listitem>
- <para>
- Output a plain-text SQL script file (the default).
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><literal>c</literal></term>
- <term><literal>custom</literal></term>
- <listitem>
- <para>
- Output a custom-format archive for each database,
- suitable for input into pg_restore. The archive
- will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
- <type>oid</type> of the database.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term><literal>t</literal></term>
- <term><literal>tar</literal></term>
- <listitem>
- <para>
- Output a tar-format archive for each database,
- suitable for input into pg_restore. The archive
- will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
- <type>oid</type> of the database.
- </para>
- </listitem>
- </varlistentry>
-
- </variablelist>
-
- See <xref linkend="app-pgdump"/> for details on how the
- various non-plain-text archive formats work.
-
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry>
<term><option>-g</option></term>
<term><option>--globals-only</option></term>
<listitem>
<para>
Dump only global objects (roles and tablespaces), no databases.
- Note: <option>--globals-only</option> cannot be used with
- <option>--clean</option> with non-text dump format.
</para>
</listitem>
</varlistentry>
@@ -1029,21 +936,13 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
<refsect1 id="app-pg-dumpall-ex">
<title>Examples</title>
<para>
- To dump all databases in plain text format (the default):
+ To dump all databases:
+
<screen>
<prompt>$</prompt> <userinput>pg_dumpall > db.out</userinput>
</screen>
</para>
- <para>
- To dump all databases using other formats:
-<screen>
-<prompt>$</prompt> <userinput>pg_dumpall --format=directory -f db.out</userinput>
-<prompt>$</prompt> <userinput>pg_dumpall --format=custom -f db.out</userinput>
-<prompt>$</prompt> <userinput>pg_dumpall --format=tar -f db.out</userinput>
-</screen>
- </para>
-
<para>
To restore database(s) from this file, you can use:
<screen>
@@ -1057,16 +956,6 @@ exclude database <replaceable class="parameter">PATTERN</replaceable>
the script will attempt to drop other databases immediately, and that
will fail for the database you are connected to.
</para>
-
- <para>
- If the dump was taken in a non-plain-text format, use
- <application>pg_restore</application> to restore the databases:
-<screen>
-<prompt>$</prompt> <userinput>pg_restore db.out -d postgres -C</userinput>
-</screen>
- This will restore all databases. To restore only some databases, use
- the <option>--exclude-database</option> option to skip those not wanted.
- </para>
</refsect1>
<refsect1>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 5e77ddd556f..b6c5299c36e 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,9 +18,8 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore <productname>PostgreSQL</productname> databases from archives
- created by <application>pg_dump</application> or
- <application>pg_dumpall</application>
+ restore a <productname>PostgreSQL</productname> database from an
+ archive file created by <application>pg_dump</application>
</refpurpose>
</refnamediv>
@@ -39,14 +38,13 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database or cluster from an archive
- created by <xref linkend="app-pgdump"/> or
- <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database from an archive
+ created by <xref linkend="app-pgdump"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database or cluster to the state it was in at the time it was saved. The
- archives also allow <application>pg_restore</application> to
+ database to the state it was in at the time it was saved. The
+ archive files also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive formats are designed to be
+ prior to being restored. The archive files are designed to be
portable across architectures.
</para>
@@ -54,34 +52,14 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database.
- When restoring from a dump made by <application>pg_dumpall</application>,
- each database will be created and then the restoration will be run in that
- database.
-
- Otherwise, when a database name is not specified, a script containing the SQL
- commands necessary to rebuild the database or cluster is created and written
+ the database. Otherwise, a script containing the SQL
+ commands necessary to rebuild the database is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application> or
- <application>pg_dumpall</application>.
-
+ the plain text output format of <application>pg_dump</application>.
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
- <para>
- A non-plain-text archive made using <application>pg_dumpall</application>
- is a directory containing a <filename>toc.glo</filename> file with global
- objects (roles and tablespaces), a <filename>map.dat</filename> file
- listing the databases, and a subdirectory for each database containing
- its archive. When restoring such an archive,
- <application>pg_restore</application> first restores global objects from
- <filename>toc.glo</filename>, then processes each database listed in
- <filename>map.dat</filename>. Lines in <filename>map.dat</filename> can
- be commented out with <literal>#</literal> to skip restoring specific
- databases.
- </para>
-
<para>
Obviously, <application>pg_restore</application> cannot restore information
that is not present in the archive file. For instance, if the
@@ -152,12 +130,6 @@ PostgreSQL documentation
ignorable error messages will be reported,
unless <option>--if-exists</option> is also specified.
</para>
- <para>
- When restoring a <application>pg_dumpall</application> archive,
- <option>--if-exists</option> is implied by <option>--clean</option>,
- since global objects such as roles and tablespaces may not exist
- in the target cluster.
- </para>
</listitem>
</varlistentry>
@@ -180,8 +152,6 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
- <option>--create</option> is required when restoring multiple databases
- from a non-plain-text archive made using <application>pg_dumpall</application>.
</para>
<para>
@@ -282,29 +252,6 @@ PostgreSQL documentation
</listitem>
</varlistentry>
- <varlistentry>
- <term><option>-g</option></term>
- <term><option>--globals-only</option></term>
- <listitem>
- <para>
- Restore only global objects (roles and tablespaces), no databases.
- </para>
- <para>
- This option is only relevant when restoring from a non-plain-text archive made using <application>pg_dumpall</application>.
- Note: <option>--globals-only</option> cannot be used with
- <option>--data-only</option>,
- <option>--schema-only</option>,
- <option>--statistics-only</option>,
- <option>--statistics</option>,
- <option>--no-globals</option>,
- <option>--exit-on-error</option>,
- <option>--single-transaction</option>,
- <option>--clean</option>, or
- <option>--transaction-size</option>.
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
@@ -641,28 +588,6 @@ PostgreSQL documentation
</listitem>
</varlistentry>
- <varlistentry>
- <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
- <listitem>
- <para>
- Do not restore databases whose name matches
- <replaceable class="parameter">pattern</replaceable>.
- Multiple patterns can be excluded by writing multiple
- <option>--exclude-database</option> switches. The
- <replaceable class="parameter">pattern</replaceable> parameter is
- interpreted as a pattern according to the same rules used by
- <application>psql</application>'s <literal>\d</literal>
- commands (see <xref linkend="app-psql-patterns"/>),
- so multiple databases can also be excluded by writing wildcard
- characters in the pattern. When using wildcards, be careful to
- quote the pattern if needed to prevent shell wildcard expansion.
- </para>
- <para>
- This option is only relevant when restoring from a non-plain-text archive made using <application>pg_dumpall</application>.
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
@@ -751,9 +676,7 @@ PostgreSQL documentation
in <option>--clean</option> mode. This suppresses <quote>does not
exist</quote> errors that might otherwise be reported. This
option is not valid unless <option>--clean</option> is also
- specified. This option is implied when restoring a
- <application>pg_dumpall</application> archive with
- <option>--clean</option>.
+ specified.
</para>
</listitem>
</varlistentry>
@@ -800,21 +723,6 @@ PostgreSQL documentation
</listitem>
</varlistentry>
- <varlistentry>
- <term><option>--no-globals</option></term>
- <listitem>
- <para>
- Do not restore global objects (roles and tablespaces). When
- <option>-C</option>/<option>--create</option> is not specified,
- databases that do not already exist on the target server are skipped.
- </para>
- <para>
- This option is only relevant when restoring from a non-plain-text
- archive made using <application>pg_dumpall</application>.
- </para>
- </listitem>
- </varlistentry>
-
<varlistentry>
<term><option>--no-policies</option></term>
<listitem>
@@ -1240,21 +1148,6 @@ CREATE DATABASE foo WITH TEMPLATE template0;
</para>
</listitem>
- <listitem>
- <para>
- The following options cannot be used when restoring from a non-plain-text
- archive made using <application>pg_dumpall</application>:
- <option>-a/--data-only</option>,
- <option>-l/--list</option>,
- <option>-L/--use-list</option>,
- <option>--no-schema</option>,
- <option>--statistics-only</option>, and
- <option>--strict-names</option>.
- Also, if the <option>--section</option> option is used, it must
- include <option>pre-data</option>.
- </para>
- </listitem>
-
</itemizedlist>
</para>
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 7c9a475963b..79bd5036841 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -103,7 +103,6 @@ tests += {
't/004_pg_dump_parallel.pl',
't/005_pg_dump_filterfile.pl',
't/006_pg_dump_compress.pl',
- 't/007_pg_dumpall.pl',
't/010_dump_connstr.pl',
],
},
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index a7bed5ecccf..7e2e9f958ea 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,20 +333,6 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
-/*
- * Update the archive handle in the on_exit callback registered by
- * on_exit_close_archive(). When pg_restore processes a pg_dumpall archive
- * containing multiple databases, each database is restored from a separate
- * archive. After closing one archive and opening the next, we update the
- * shutdown_info to reference the new archive handle so the cleanup callback
- * will close the correct archive on exit.
- */
-void
-replace_on_exit_close_archive(Archive *AHX)
-{
- shutdown_info.AHX = AHX;
-}
-
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 28e7ff6fa16..c7bdda1deed 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -313,7 +313,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX, bool append_data);
+extern void RestoreArchive(Archive *AHX);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 2fd773ad84f..6d2f2ea8b69 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -87,7 +87,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec, bool append_data);
+ const pg_compress_specification compression_spec);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -340,14 +340,9 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/*
- * RestoreArchive
- *
- * If append_data is set, then append data into file as we are restoring dump
- * of multiple databases which was taken by pg_dumpall.
- */
+/* Public */
void
-RestoreArchive(Archive *AHX, bool append_data)
+RestoreArchive(Archive *AHX)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -464,7 +459,7 @@ RestoreArchive(Archive *AHX, bool append_data)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
+ SetOutput(AH, ropt->filename, ropt->compression_spec);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -767,19 +762,6 @@ RestoreArchive(Archive *AHX, bool append_data)
if ((te->reqs & (REQ_SCHEMA | REQ_DATA | REQ_STATS)) == 0)
continue; /* ignore if not to be dumped at all */
- /* Skip if no-tablespace is given. */
- if (ropt->noTablespace && te && te->desc &&
- (strcmp(te->desc, "TABLESPACE") == 0))
- continue;
-
- /*
- * Skip DROP DATABASE/ROLES/TABLESPACE if we didn't specify
- * --clean
- */
- if (!ropt->dropSchema && te && te->desc &&
- strcmp(te->desc, "DROP_GLOBAL") == 0)
- continue;
-
switch (_tocEntryRestorePass(te))
{
case RESTORE_PASS_MAIN:
@@ -1335,7 +1317,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec, false);
+ SetOutput(AH, ropt->filename, out_compression_spec);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1710,15 +1692,11 @@ archprintf(Archive *AH, const char *fmt, ...)
/*******************************
* Stuff below here should be 'private' to the archiver routines
- *
- * If append_data is set, then append data into file as we are restoring dump
- * of multiple databases which was taken by pg_dumpall.
*******************************/
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec,
- bool append_data)
+ const pg_compress_specification compression_spec)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1738,7 +1716,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (append_data || AH->mode == archModeAppend)
+ if (AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
@@ -2414,7 +2392,7 @@ _allocAH(const char *FileSpec, const ArchiveFormat fmt,
/* initialize for backwards compatible string processing */
AH->public.encoding = 0; /* PG_SQL_ASCII */
- AH->public.std_strings = true;
+ AH->public.std_strings = false;
/* sql error handling */
AH->public.exit_on_error = true;
@@ -3050,16 +3028,6 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
return 0;
}
- /*
- * Global object TOC entries (e.g., ROLEs or TABLESPACEs) must not be
- * ignored.
- */
- if (strcmp(te->desc, "ROLE") == 0 ||
- strcmp(te->desc, "ROLE PROPERTIES") == 0 ||
- strcmp(te->desc, "TABLESPACE") == 0 ||
- strcmp(te->desc, "DROP_GLOBAL") == 0)
- return REQ_SCHEMA;
-
/*
* Process exclusions that affect certain classes of TOC entries.
*/
@@ -3095,14 +3063,6 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
if (ropt->no_subscriptions &&
strncmp(te->tag, "SUBSCRIPTION", strlen("SUBSCRIPTION")) == 0)
return 0;
-
- /*
- * Comments on global objects (ROLEs or TABLESPACEs) should not be
- * skipped, since global objects themselves are never skipped.
- */
- if (strncmp(te->tag, "ROLE", strlen("ROLE")) == 0 ||
- strncmp(te->tag, "TABLESPACE", strlen("TABLESPACE")) == 0)
- return REQ_SCHEMA;
}
/*
@@ -3132,14 +3092,6 @@ _tocEntryRequired(TocEntry *te, teSection curSection, ArchiveHandle *AH)
if (ropt->no_subscriptions &&
strncmp(te->tag, "SUBSCRIPTION", strlen("SUBSCRIPTION")) == 0)
return 0;
-
- /*
- * Security labels on global objects (ROLEs or TABLESPACEs) should not
- * be skipped, since global objects themselves are never skipped.
- */
- if (strncmp(te->tag, "ROLE", strlen("ROLE")) == 0 ||
- strncmp(te->tag, "TABLESPACE", strlen("TABLESPACE")) == 0)
- return REQ_SCHEMA;
}
/* If it's a subscription, maybe ignore it */
@@ -3915,9 +3867,6 @@ _getObjectDescription(PQExpBuffer buf, const TocEntry *te)
else if (strcmp(type, "CAST") == 0 ||
strcmp(type, "CHECK CONSTRAINT") == 0 ||
strcmp(type, "CONSTRAINT") == 0 ||
- strcmp(type, "DROP_GLOBAL") == 0 ||
- strcmp(type, "ROLE PROPERTIES") == 0 ||
- strcmp(type, "ROLE") == 0 ||
strcmp(type, "DATABASE PROPERTIES") == 0 ||
strcmp(type, "DEFAULT") == 0 ||
strcmp(type, "FK CONSTRAINT") == 0 ||
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index 1218bf6a6a1..c1528d78853 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -394,7 +394,6 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
-extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt, ...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index ae3a7c80cbf..55b1e4e85d0 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH, false);
+ RestoreArchive((Archive *) AH);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index a0f7f8e2168..c56437d6057 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1279,7 +1279,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout, false);
+ RestoreArchive(fout);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index c1f43113c53..b9653f0aefe 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -1,20 +1,13 @@
/*-------------------------------------------------------------------------
*
* pg_dumpall.c
- * pg_dumpall dumps all databases and global objects (roles and
- * tablespaces) from a PostgreSQL cluster.
- *
- * For text format output, globals are written directly and pg_dump is
- * invoked for each database, with all output going to stdout or a file.
- *
- * For non-text formats (custom, directory, tar), a directory is created
- * containing a toc.glo file with global objects, a map.dat file mapping
- * database OIDs to names, and a databases/ subdirectory with individual
- * pg_dump archives for each database.
*
* Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
+ * pg_dumpall forces all pg_dump output to be text, since it also outputs
+ * text into the same output stream.
+ *
* src/bin/pg_dump/pg_dumpall.c
*
*-------------------------------------------------------------------------
@@ -34,11 +27,9 @@
#include "common/string.h"
#include "connectdb.h"
#include "dumputils.h"
-#include "fe_utils/option_utils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
-#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -76,19 +67,15 @@ static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
static void dumpDatabases(PGconn *conn);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts, char *dbfile);
+static int runPgDump(const char *dbname, const char *create_opts);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
static void executeCommand(PGconn *conn, const char *query);
-static void check_for_invalid_global_names(PGconn *conn,
- SimpleStringList *excluded_names);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
-static ArchiveFormat parseDumpFormat(const char *format);
-static int createDumpId(void);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -136,10 +123,6 @@ static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
static char *restrict_key;
-static Archive *fout = NULL;
-static int dumpIdVal = 0;
-static ArchiveFormat archDumpFormat = archNull;
-static const CatalogId nilCatalogId = {0, 0};
int
main(int argc, char *argv[])
@@ -165,7 +148,6 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
- {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -215,19 +197,16 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
- const char *format_name = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
bool roles_only = false;
- bool schema_only = false;
bool tablespaces_only = false;
PGconn *conn;
int encoding;
int c,
ret;
int optindex;
- DumpOptions dopt;
pg_logging_init(argv[0]);
pg_logging_set_level(PG_LOG_WARNING);
@@ -265,9 +244,8 @@ main(int argc, char *argv[])
}
pgdumpopts = createPQExpBuffer();
- InitDumpOptions(&dopt);
- while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -295,9 +273,7 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
- case 'F':
- format_name = pg_strdup(optarg);
- break;
+
case 'g':
globals_only = true;
break;
@@ -323,7 +299,6 @@ main(int argc, char *argv[])
break;
case 's':
- schema_only = true;
appendPQExpBufferStr(pgdumpopts, " -s");
break;
@@ -338,7 +313,6 @@ main(int argc, char *argv[])
case 'U':
pguser = pg_strdup(optarg);
- dopt.cparams.username = pg_strdup(optarg);
break;
case 'v':
@@ -421,74 +395,50 @@ main(int argc, char *argv[])
exit_nicely(1);
}
- /* --exclude-database is incompatible with global *-only options */
- check_mut_excl_opts(database_exclude_patterns.head, "--exclude-database",
- globals_only, "-g/--globals-only",
- roles_only, "-r/--roles-only",
- tablespaces_only, "-t/--tablespaces-only");
+ if (database_exclude_patterns.head != NULL &&
+ (globals_only || roles_only || tablespaces_only))
+ {
+ pg_log_error("option %s cannot be used together with %s, %s, or %s",
+ "--exclude-database",
+ "-g/--globals-only", "-r/--roles-only", "-t/--tablespaces-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
- /* *-only options are incompatible with each other */
- check_mut_excl_opts(data_only, "-a/--data-only",
- globals_only, "-g/--globals-only",
- roles_only, "-r/--roles-only",
- schema_only, "-s/--schema-only",
- statistics_only, "--statistics-only",
- tablespaces_only, "-t/--tablespaces-only");
+ /* Make sure the user hasn't specified a mix of globals-only options */
+ if (globals_only && roles_only)
+ {
+ pg_log_error("options %s and %s cannot be used together",
+ "-g/--globals-only", "-r/--roles-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
- /* --no-* and *-only for same thing are incompatible */
- check_mut_excl_opts(data_only, "-a/--data-only",
- no_data, "--no-data");
- check_mut_excl_opts(schema_only, "-s/--schema-only",
- no_schema, "--no-schema");
- check_mut_excl_opts(statistics_only, "--statistics-only",
- no_statistics, "--no-statistics");
-
- /* --statistics and --no-statistics are incompatible */
- check_mut_excl_opts(with_statistics, "--statistics",
- no_statistics, "--no-statistics");
-
- /* --statistics is incompatible with *-only (except --statistics-only) */
- check_mut_excl_opts(with_statistics, "--statistics",
- data_only, "-a/--data-only",
- globals_only, "-g/--globals-only",
- roles_only, "-r/--roles-only",
- schema_only, "-s/--schema-only",
- tablespaces_only, "-t/--tablespaces-only");
-
- /* --clean and --data-only are incompatible */
- check_mut_excl_opts(output_clean, "-c/--clean",
- data_only, "-a/--data-only");
+ if (globals_only && tablespaces_only)
+ {
+ pg_log_error("options %s and %s cannot be used together",
+ "-g/--globals-only", "-t/--tablespaces-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
if (if_exists && !output_clean)
pg_fatal("option %s requires option %s",
"--if-exists", "-c/--clean");
- /* Get format for dump. */
- archDumpFormat = parseDumpFormat(format_name);
+ /* --clean and --data-only are incompatible */
+ if (output_clean && data_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-c/--clean", "-a/--data-only");
- /*
- * If a non-plain format is specified, a file name is also required as the
- * path to the main directory.
- */
- if (archDumpFormat != archNull &&
- (!filename || strcmp(filename, "") == 0))
+ if (roles_only && tablespaces_only)
{
- pg_log_error("option %s=d|c|t requires option %s",
- "-F/--format", "-f/--file");
+ pg_log_error("options %s and %s cannot be used together",
+ "-r/--roles-only", "-t/--tablespaces-only");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit_nicely(1);
}
- /* restrict-key is only supported with --format=plain */
- if (archDumpFormat != archNull && restrict_key)
- pg_fatal("option %s can only be used with %s=plain",
- "--restrict-key", "--format");
-
- /* --clean and -g/--globals-only cannot be used together in non-text dump */
- if (archDumpFormat != archNull && output_clean && globals_only)
- pg_fatal("options %s and %s cannot be used together in non-text dump",
- "--clean", "-g/--globals-only");
-
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -549,27 +499,6 @@ main(int argc, char *argv[])
if (sequence_data)
appendPQExpBufferStr(pgdumpopts, " --sequence-data");
- /*
- * Open the output file if required, otherwise use stdout. If required,
- * then create new directory.
- */
- if (archDumpFormat != archNull)
- {
- Assert(filename);
-
- /* Create new directory or accept the empty existing directory. */
- create_or_open_dir(filename);
- }
- else if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* If you don't provide a restrict key, one will be appointed for you.
*/
@@ -619,6 +548,19 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
+ /*
+ * Open the output file if required, otherwise use stdout
+ */
+ if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* Set the client encoding if requested.
*/
@@ -656,121 +598,35 @@ main(int argc, char *argv[])
if (quote_all_identifiers)
executeCommand(conn, "SET quote_all_identifiers = true");
- /* create a archive file for global commands. */
- if (archDumpFormat != archNull)
- {
- PQExpBuffer qry = createPQExpBuffer();
- char global_path[MAXPGPATH];
- const char *encname;
- pg_compress_specification compression_spec = {0};
-
- /*
- * Check that no global object names contain newlines or carriage
- * returns, which would break the map.dat file format. This is only
- * needed for servers older than v19, which started prohibiting such
- * names.
- */
- if (server_version < 190000)
- check_for_invalid_global_names(conn, &database_exclude_names);
-
- /* Set file path for global sql commands. */
- snprintf(global_path, MAXPGPATH, "%s/toc.glo", filename);
-
- /* Open the output file */
- fout = CreateArchive(global_path, archCustom, compression_spec,
- dosync, archModeWrite, NULL, DATA_DIR_SYNC_METHOD_FSYNC);
-
- /* Make dump options accessible right away */
- SetArchiveOptions(fout, &dopt, NULL);
-
- ((ArchiveHandle *) fout)->connection = conn;
- ((ArchiveHandle *) fout)->public.numWorkers = 1;
-
- /* Register the cleanup hook */
- on_exit_close_archive(fout);
-
- /* Let the archiver know how noisy to be */
- fout->verbose = verbose;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release
- * of our own major version. (See also version check in
- * pg_dumpall.c.)
- */
- fout->minRemoteVersion = 90200;
- fout->maxRemoteVersion = (PG_VERSION_NUM / 100) * 100 + 99;
- fout->numWorkers = 1;
-
- /* Dump default_transaction_read_only. */
- appendPQExpBufferStr(qry, "SET default_transaction_read_only = off;\n\n");
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = "default_transaction_read_only",
- .description = "default_transaction_read_only",
- .section = SECTION_PRE_DATA,
- .createStmt = qry->data));
- resetPQExpBuffer(qry);
-
- /* Put the correct encoding into the archive */
- encname = pg_encoding_to_char(encoding);
-
- appendPQExpBufferStr(qry, "SET client_encoding = ");
- appendStringLiteralAH(qry, encname, fout);
- appendPQExpBufferStr(qry, ";\n");
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = "client_encoding",
- .description = "client_encoding",
- .section = SECTION_PRE_DATA,
- .createStmt = qry->data));
- resetPQExpBuffer(qry);
-
- /* Put the correct escape string behavior into the archive. */
- appendPQExpBufferStr(qry, "SET standard_conforming_strings = 'on';\n");
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = "standard_conforming_strings",
- .description = "standard_conforming_strings",
- .section = SECTION_PRE_DATA,
- .createStmt = qry->data));
- destroyPQExpBuffer(qry);
- }
- else
- {
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
-
- if (verbose)
- dumpTimestamp("Started on");
-
- /*
- * Enter restricted mode to block any unexpected psql meta-commands. A
- * malicious source might try to inject a variety of things via bogus
- * responses to queries. While we cannot prevent such sources from
- * affecting the destination at restore time, we can block psql
- * meta-commands so that the client machine that runs psql with the
- * dump output remains unaffected.
- */
- fprintf(OPF, "\\restrict %s\n\n", restrict_key);
-
- /*
- * We used to emit \connect postgres here, but that served no purpose
- * other than to break things for installations without a postgres
- * database. Everything we're restoring here is a global, so
- * whichever database we're connected to at the moment is fine.
- */
-
- /* Restore will need to write to the target cluster */
- fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
-
- /* Replicate encoding and standard_conforming_strings in output */
- fprintf(OPF, "SET client_encoding = '%s';\n",
- pg_encoding_to_char(encoding));
- fprintf(OPF, "SET standard_conforming_strings = on;\n");
- fprintf(OPF, "\n");
- }
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump\n--\n\n");
+ if (verbose)
+ dumpTimestamp("Started on");
+
+ /*
+ * Enter restricted mode to block any unexpected psql meta-commands. A
+ * malicious source might try to inject a variety of things via bogus
+ * responses to queries. While we cannot prevent such sources from
+ * affecting the destination at restore time, we can block psql
+ * meta-commands so that the client machine that runs psql with the dump
+ * output remains unaffected.
+ */
+ fprintf(OPF, "\\restrict %s\n\n", restrict_key);
+
+ /*
+ * We used to emit \connect postgres here, but that served no purpose
+ * other than to break things for installations without a postgres
+ * database. Everything we're restoring here is a global, so whichever
+ * database we're connected to at the moment is fine.
+ */
+
+ /* Restore will need to write to the target cluster */
+ fprintf(OPF, "SET default_transaction_read_only = off;\n\n");
+
+ /* Replicate encoding and standard_conforming_strings in output */
+ fprintf(OPF, "SET client_encoding = '%s';\n",
+ pg_encoding_to_char(encoding));
+ fprintf(OPF, "SET standard_conforming_strings = on;\n");
+ fprintf(OPF, "\n");
if (!data_only && !statistics_only && !no_schema)
{
@@ -779,14 +635,8 @@ main(int argc, char *argv[])
* dependency analysis because databases never depend on each other,
* and tablespaces never depend on each other. Roles could have
* grants to each other, but DROP ROLE will clean those up silently.
- *
- * For non-text formats, pg_dumpall unconditionally process --clean
- * option. In contrast, pg_restore only applies it if the user
- * explicitly provides the flag. This discrepancy resolves corner
- * cases where pg_restore requires cleanup instructions that may be
- * missing from a standard pg_dumpall output.
*/
- if (output_clean || archDumpFormat != archNull)
+ if (output_clean)
{
if (!globals_only && !roles_only && !tablespaces_only)
dropDBs(conn);
@@ -820,45 +670,28 @@ main(int argc, char *argv[])
dumpTablespaces(conn);
}
- if (archDumpFormat == archNull)
- {
- /*
- * Exit restricted mode just before dumping the databases. pg_dump
- * will handle entering restricted mode again as appropriate.
- */
- fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
- }
+ /*
+ * Exit restricted mode just before dumping the databases. pg_dump will
+ * handle entering restricted mode again as appropriate.
+ */
+ fprintf(OPF, "\\unrestrict %s\n\n", restrict_key);
if (!globals_only && !roles_only && !tablespaces_only)
dumpDatabases(conn);
- if (archDumpFormat == archNull)
- {
- PQfinish(conn);
-
- if (verbose)
- dumpTimestamp("Completed on");
- fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
+ PQfinish(conn);
- if (filename)
- {
- fclose(OPF);
+ if (verbose)
+ dumpTimestamp("Completed on");
+ fprintf(OPF, "--\n-- PostgreSQL database cluster dump complete\n--\n\n");
- /* sync the resulting file, errors are not fatal */
- if (dosync)
- (void) fsync_fname(filename, false);
- }
- }
- else
+ if (filename)
{
- RestoreOptions *ropt;
-
- ropt = NewRestoreOptions();
- SetArchiveOptions(fout, &dopt, ropt);
+ fclose(OPF);
- /* Mark which entries should be output */
- ProcessArchiveRestoreOptions(fout);
- CloseArchive(fout);
+ /* sync the resulting file, errors are not fatal */
+ if (dosync)
+ (void) fsync_fname(filename, false);
}
exit_nicely(0);
@@ -868,14 +701,12 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s exports a PostgreSQL database cluster as an SQL script or to other formats.\n\n"), progname);
+ printf(_("%s exports a PostgreSQL database cluster as an SQL script.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
- printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
- " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -970,45 +801,24 @@ dropRoles(PGconn *conn)
i_rolname = PQfnumber(res, "rolname");
- if (PQntuples(res) > 0 && archDumpFormat == archNull)
+ if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Drop roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
const char *rolename;
- PQExpBuffer delQry = createPQExpBuffer();
rolename = PQgetvalue(res, i, i_rolname);
- if (archDumpFormat == archNull)
- {
- appendPQExpBuffer(delQry, "DROP ROLE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(rolename));
- fprintf(OPF, "%s", delQry->data);
- }
- else
- {
- appendPQExpBuffer(delQry, "DROP ROLE IF EXISTS %s;\n",
- fmtId(rolename));
-
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = psprintf("ROLE %s", fmtId(rolename)),
- .description = "DROP_GLOBAL",
- .section = SECTION_PRE_DATA,
- .createStmt = delQry->data));
- }
-
- destroyPQExpBuffer(delQry);
+ fprintf(OPF, "DROP ROLE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(rolename));
}
PQclear(res);
destroyPQExpBuffer(buf);
- if (archDumpFormat == archNull)
- fprintf(OPF, "\n\n");
+ fprintf(OPF, "\n\n");
}
/*
@@ -1018,8 +828,6 @@ static void
dumpRoles(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
- PQExpBuffer comment_buf = createPQExpBuffer();
- PQExpBuffer seclabel_buf = createPQExpBuffer();
PGresult *res;
int i_oid,
i_rolname,
@@ -1091,7 +899,7 @@ dumpRoles(PGconn *conn)
i_rolcomment = PQfnumber(res, "rolcomment");
i_is_current_user = PQfnumber(res, "is_current_user");
- if (PQntuples(res) > 0 && archDumpFormat == archNull)
+ if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Roles\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1110,8 +918,6 @@ dumpRoles(PGconn *conn)
}
resetPQExpBuffer(buf);
- resetPQExpBuffer(comment_buf);
- resetPQExpBuffer(seclabel_buf);
if (binary_upgrade)
{
@@ -1188,53 +994,17 @@ dumpRoles(PGconn *conn)
if (!no_comments && !PQgetisnull(res, i, i_rolcomment))
{
- appendPQExpBuffer(comment_buf, "COMMENT ON ROLE %s IS ", fmtId(rolename));
- appendStringLiteralConn(comment_buf, PQgetvalue(res, i, i_rolcomment), conn);
- appendPQExpBufferStr(comment_buf, ";\n");
+ appendPQExpBuffer(buf, "COMMENT ON ROLE %s IS ", fmtId(rolename));
+ appendStringLiteralConn(buf, PQgetvalue(res, i, i_rolcomment), conn);
+ appendPQExpBufferStr(buf, ";\n");
}
if (!no_security_labels)
buildShSecLabels(conn, "pg_authid", auth_oid,
"ROLE", rolename,
- seclabel_buf);
+ buf);
- if (archDumpFormat == archNull)
- {
- fprintf(OPF, "%s", buf->data);
- fprintf(OPF, "%s", comment_buf->data);
-
- if (seclabel_buf->data[0] != '\0')
- fprintf(OPF, "%s", seclabel_buf->data);
- }
- else
- {
- char *tag = psprintf("ROLE %s", fmtId(rolename));
-
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = tag,
- .description = "ROLE",
- .section = SECTION_PRE_DATA,
- .createStmt = buf->data));
- if (comment_buf->data[0] != '\0')
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = tag,
- .description = "COMMENT",
- .section = SECTION_PRE_DATA,
- .createStmt = comment_buf->data));
-
- if (seclabel_buf->data[0] != '\0')
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = tag,
- .description = "SECURITY LABEL",
- .section = SECTION_PRE_DATA,
- .createStmt = seclabel_buf->data));
- }
+ fprintf(OPF, "%s", buf->data);
}
/*
@@ -1242,7 +1012,7 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0 && archDumpFormat == archNull)
+ if (PQntuples(res) > 0)
fprintf(OPF, "\n--\n-- User Configurations\n--\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1250,12 +1020,9 @@ dumpRoles(PGconn *conn)
PQclear(res);
- if (archDumpFormat == archNull)
- fprintf(OPF, "\n\n");
+ fprintf(OPF, "\n\n");
destroyPQExpBuffer(buf);
- destroyPQExpBuffer(comment_buf);
- destroyPQExpBuffer(seclabel_buf);
}
@@ -1269,7 +1036,6 @@ static void
dumpRoleMembership(PGconn *conn)
{
PQExpBuffer buf = createPQExpBuffer();
- PQExpBuffer querybuf = createPQExpBuffer();
PQExpBuffer optbuf = createPQExpBuffer();
PGresult *res;
int start = 0,
@@ -1332,7 +1098,7 @@ dumpRoleMembership(PGconn *conn)
i_inherit_option = PQfnumber(res, "inherit_option");
i_set_option = PQfnumber(res, "set_option");
- if (PQntuples(res) > 0 && archDumpFormat == archNull)
+ if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Role memberships\n--\n\n");
/*
@@ -1479,9 +1245,8 @@ dumpRoleMembership(PGconn *conn)
/* Generate the actual GRANT statement. */
resetPQExpBuffer(optbuf);
- resetPQExpBuffer(querybuf);
- appendPQExpBuffer(querybuf, "GRANT %s", fmtId(role));
- appendPQExpBuffer(querybuf, " TO %s", fmtId(member));
+ fprintf(OPF, "GRANT %s", fmtId(role));
+ fprintf(OPF, " TO %s", fmtId(member));
if (*admin_option == 't')
appendPQExpBufferStr(optbuf, "ADMIN OPTION");
if (dump_grant_options)
@@ -1502,21 +1267,10 @@ dumpRoleMembership(PGconn *conn)
appendPQExpBufferStr(optbuf, "SET FALSE");
}
if (optbuf->data[0] != '\0')
- appendPQExpBuffer(querybuf, " WITH %s", optbuf->data);
+ fprintf(OPF, " WITH %s", optbuf->data);
if (dump_this_grantor)
- appendPQExpBuffer(querybuf, " GRANTED BY %s", fmtId(grantor));
- appendPQExpBufferStr(querybuf, ";\n");
-
- if (archDumpFormat == archNull)
- fprintf(OPF, "%s", querybuf->data);
- else
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = psprintf("ROLE %s", fmtId(role)),
- .description = "ROLE PROPERTIES",
- .section = SECTION_PRE_DATA,
- .createStmt = querybuf->data));
+ fprintf(OPF, " GRANTED BY %s", fmtId(grantor));
+ fprintf(OPF, ";\n");
}
}
@@ -1527,11 +1281,8 @@ dumpRoleMembership(PGconn *conn)
PQclear(res);
destroyPQExpBuffer(buf);
- destroyPQExpBuffer(querybuf);
- destroyPQExpBuffer(optbuf);
- if (archDumpFormat == archNull)
- fprintf(OPF, "\n\n");
+ fprintf(OPF, "\n\n");
}
@@ -1558,7 +1309,7 @@ dumpRoleGUCPrivs(PGconn *conn)
"FROM pg_catalog.pg_parameter_acl "
"ORDER BY 1");
- if (PQntuples(res) > 0 && archDumpFormat == archNull)
+ if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Role privileges on configuration parameters\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1583,25 +1334,14 @@ dumpRoleGUCPrivs(PGconn *conn)
exit_nicely(1);
}
- if (archDumpFormat == archNull)
- fprintf(OPF, "%s", buf->data);
- else
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = psprintf("ROLE %s", fmtId(parowner)),
- .description = "ROLE PROPERTIES",
- .section = SECTION_PRE_DATA,
- .createStmt = buf->data));
+ fprintf(OPF, "%s", buf->data);
free(fparname);
destroyPQExpBuffer(buf);
}
PQclear(res);
-
- if (archDumpFormat == archNull)
- fprintf(OPF, "\n\n");
+ fprintf(OPF, "\n\n");
}
@@ -1623,41 +1363,21 @@ dropTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0 && archDumpFormat == archNull)
+ if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Drop tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
{
char *spcname = PQgetvalue(res, i, 0);
- PQExpBuffer delQry = createPQExpBuffer();
- if (archDumpFormat == archNull)
- {
- appendPQExpBuffer(delQry, "DROP TABLESPACE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(spcname));
- fprintf(OPF, "%s", delQry->data);
- }
- else
- {
- appendPQExpBuffer(delQry, "DROP TABLESPACE IF EXISTS %s;\n",
- fmtId(spcname));
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = psprintf("TABLESPACE %s", fmtId(spcname)),
- .description = "DROP_GLOBAL",
- .section = SECTION_PRE_DATA,
- .createStmt = delQry->data));
- }
-
- destroyPQExpBuffer(delQry);
+ fprintf(OPF, "DROP TABLESPACE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(spcname));
}
PQclear(res);
- if (archDumpFormat == archNull)
- fprintf(OPF, "\n\n");
+ fprintf(OPF, "\n\n");
}
/*
@@ -1667,8 +1387,6 @@ static void
dumpTablespaces(PGconn *conn)
{
PGresult *res;
- PQExpBuffer comment_buf = createPQExpBuffer();
- PQExpBuffer seclabel_buf = createPQExpBuffer();
int i;
/*
@@ -1685,7 +1403,7 @@ dumpTablespaces(PGconn *conn)
"WHERE spcname !~ '^pg_' "
"ORDER BY 1");
- if (PQntuples(res) > 0 && archDumpFormat == archNull)
+ if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Tablespaces\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1704,9 +1422,6 @@ dumpTablespaces(PGconn *conn)
/* needed for buildACLCommands() */
fspcname = pg_strdup(fmtId(spcname));
- resetPQExpBuffer(comment_buf);
- resetPQExpBuffer(seclabel_buf);
-
if (binary_upgrade)
{
appendPQExpBufferStr(buf, "\n-- For binary upgrade, must preserve pg_tablespace oid\n");
@@ -1748,67 +1463,24 @@ dumpTablespaces(PGconn *conn)
if (!no_comments && spccomment && spccomment[0] != '\0')
{
- appendPQExpBuffer(comment_buf, "COMMENT ON TABLESPACE %s IS ", fspcname);
- appendStringLiteralConn(comment_buf, spccomment, conn);
- appendPQExpBufferStr(comment_buf, ";\n");
+ appendPQExpBuffer(buf, "COMMENT ON TABLESPACE %s IS ", fspcname);
+ appendStringLiteralConn(buf, spccomment, conn);
+ appendPQExpBufferStr(buf, ";\n");
}
if (!no_security_labels)
buildShSecLabels(conn, "pg_tablespace", spcoid,
"TABLESPACE", spcname,
- seclabel_buf);
+ buf);
- if (archDumpFormat == archNull)
- {
- fprintf(OPF, "%s", buf->data);
-
- if (comment_buf->data[0] != '\0')
- fprintf(OPF, "%s", comment_buf->data);
-
- if (seclabel_buf->data[0] != '\0')
- fprintf(OPF, "%s", seclabel_buf->data);
- }
- else
- {
- char *tag = psprintf("TABLESPACE %s", fmtId(fspcname));
-
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = tag,
- .description = "TABLESPACE",
- .section = SECTION_PRE_DATA,
- .createStmt = buf->data));
-
- if (comment_buf->data[0] != '\0')
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = tag,
- .description = "COMMENT",
- .section = SECTION_PRE_DATA,
- .createStmt = comment_buf->data));
-
- if (seclabel_buf->data[0] != '\0')
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = tag,
- .description = "SECURITY LABEL",
- .section = SECTION_PRE_DATA,
- .createStmt = seclabel_buf->data));
- }
+ fprintf(OPF, "%s", buf->data);
free(fspcname);
destroyPQExpBuffer(buf);
}
PQclear(res);
- destroyPQExpBuffer(comment_buf);
- destroyPQExpBuffer(seclabel_buf);
-
- if (archDumpFormat == archNull)
- fprintf(OPF, "\n\n");
+ fprintf(OPF, "\n\n");
}
@@ -1831,7 +1503,7 @@ dropDBs(PGconn *conn)
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY datname");
- if (PQntuples(res) > 0 && archDumpFormat == archNull)
+ if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Drop databases (except postgres and template1)\n--\n\n");
for (i = 0; i < PQntuples(res); i++)
@@ -1847,33 +1519,15 @@ dropDBs(PGconn *conn)
strcmp(dbname, "template0") != 0 &&
strcmp(dbname, "postgres") != 0)
{
- if (archDumpFormat == archNull)
- {
- fprintf(OPF, "DROP DATABASE %s%s;\n",
- if_exists ? "IF EXISTS " : "",
- fmtId(dbname));
- }
- else
- {
- char *stmt = psprintf("DROP DATABASE IF EXISTS %s;\n",
- fmtId(dbname));
-
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = psprintf("DATABASE %s", fmtId(dbname)),
- .description = "DROP_GLOBAL",
- .section = SECTION_PRE_DATA,
- .createStmt = stmt));
- pg_free(stmt);
- }
+ fprintf(OPF, "DROP DATABASE %s%s;\n",
+ if_exists ? "IF EXISTS " : "",
+ fmtId(dbname));
}
}
PQclear(res);
- if (archDumpFormat == archNull)
- fprintf(OPF, "\n\n");
+ fprintf(OPF, "\n\n");
}
@@ -1895,7 +1549,7 @@ dumpUserConfig(PGconn *conn, const char *username)
res = executeQuery(conn, buf->data);
- if (PQntuples(res) > 0 && archDumpFormat == archNull)
+ if (PQntuples(res) > 0)
{
char *sanitized;
@@ -1910,17 +1564,7 @@ dumpUserConfig(PGconn *conn, const char *username)
makeAlterConfigCommand(conn, PQgetvalue(res, i, 0),
"ROLE", username, NULL, NULL,
buf);
-
- if (archDumpFormat == archNull)
- fprintf(OPF, "%s", buf->data);
- else
- ArchiveEntry(fout,
- nilCatalogId, /* catalog ID */
- createDumpId(), /* dump ID */
- ARCHIVE_OPTS(.tag = psprintf("ROLE %s", fmtId(username)),
- .description = "ROLE PROPERTIES",
- .section = SECTION_PRE_DATA,
- .createStmt = buf->data));
+ fprintf(OPF, "%s", buf->data);
}
PQclear(res);
@@ -1990,9 +1634,6 @@ dumpDatabases(PGconn *conn)
{
PGresult *res;
int i;
- char db_subdir[MAXPGPATH];
- char dbfilepath[MAXPGPATH];
- FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -2006,55 +1647,19 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname, oid "
+ "SELECT datname "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
- if (PQntuples(res) > 0 && archDumpFormat == archNull)
+ if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
- /*
- * If directory/tar/custom format is specified, create a subdirectory
- * under the main directory and each database dump file or subdirectory
- * will be created in that subdirectory by pg_dump.
- */
- if (archDumpFormat != archNull)
- {
- char map_file_path[MAXPGPATH];
-
- snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
-
- /* Create a subdirectory with 'databases' name under main directory. */
- if (mkdir(db_subdir, pg_dir_create_mode) != 0)
- pg_fatal("could not create directory \"%s\": %m", db_subdir);
-
- snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
-
- /* Create a map file (to store dboid and dbname) */
- map_file = fopen(map_file_path, PG_BINARY_W);
- if (!map_file)
- pg_fatal("could not open file \"%s\": %m", map_file_path);
-
- fprintf(map_file,
- "#################################################################\n"
- "# map.dat\n"
- "#\n"
- "# This file maps oids to database names\n"
- "#\n"
- "# pg_restore will restore all the databases listed here, unless\n"
- "# otherwise excluded. You can also inhibit restoration of a\n"
- "# database by removing the line or commenting out the line with\n"
- "# a # mark.\n"
- "#################################################################\n");
- }
-
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
char *sanitized;
- char *oid = PQgetvalue(res, i, 1);
- const char *create_opts = "";
+ const char *create_opts;
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -2071,10 +1676,7 @@ dumpDatabases(PGconn *conn)
pg_log_info("dumping database \"%s\"", dbname);
sanitized = sanitize_line(dbname, true);
-
- if (archDumpFormat == archNull)
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
-
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", sanitized);
free(sanitized);
/*
@@ -2089,40 +1691,24 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
- /* Since pg_dump won't emit a \connect command, we must */
- else if (archDumpFormat == archNull)
- fprintf(OPF, "\\connect %s\n\n", dbname);
else
+ {
create_opts = "";
+ /* Since pg_dump won't emit a \connect command, we must */
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
}
else
create_opts = "--create";
- if (filename && archDumpFormat == archNull)
+ if (filename)
fclose(OPF);
- /*
- * If this is not a plain format dump, then append dboid and dbname to
- * the map.dat file.
- */
- if (archDumpFormat != archNull)
- {
- if (archDumpFormat == archCustom)
- snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
- else if (archDumpFormat == archTar)
- snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
- else
- snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
-
- /* Put one line entry for dboid and dbname in map file. */
- fprintf(map_file, "%s %s\n", oid, dbname);
- }
-
- ret = runPgDump(dbname, create_opts, dbfilepath);
+ ret = runPgDump(dbname, create_opts);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
- if (filename && archDumpFormat == archNull)
+ if (filename)
{
OPF = fopen(filename, PG_BINARY_A);
if (!OPF)
@@ -2131,10 +1717,6 @@ dumpDatabases(PGconn *conn)
}
}
- /* Close map file */
- if (archDumpFormat != archNull)
- fclose(map_file);
-
PQclear(res);
}
@@ -2144,7 +1726,7 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts, char *dbfile)
+runPgDump(const char *dbname, const char *create_opts)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -2153,36 +1735,17 @@ runPgDump(const char *dbname, const char *create_opts, char *dbfile)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
/*
- * If this is not a plain format dump, then append file name and dump
- * format to the pg_dump command to get archive dump.
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
*/
- if (archDumpFormat != archNull)
- {
- printfPQExpBuffer(&cmd, "\"%s\" %s -f %s %s", pg_dump_bin,
- pgdumpopts->data, dbfile, create_opts);
-
- if (archDumpFormat == archDirectory)
- appendPQExpBufferStr(&cmd, " --format=directory ");
- else if (archDumpFormat == archCustom)
- appendPQExpBufferStr(&cmd, " --format=custom ");
- else if (archDumpFormat == archTar)
- appendPQExpBufferStr(&cmd, " --format=tar ");
- }
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
else
- {
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
- /*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
- */
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
- else
- appendPQExpBufferStr(&cmd, " -Fp ");
- }
+ appendPQExpBufferStr(&cmd, " -Fp ");
/*
* Append the database name to the already-constructed stem of connection
@@ -2256,76 +1819,6 @@ executeCommand(PGconn *conn, const char *query)
}
-/*
- * check_for_invalid_global_names
- *
- * Check that no database, role, or tablespace name contains a newline or
- * carriage return character. Such characters in database names would break
- * the map.dat file format used for non-plain-text dumps. Role and tablespace
- * names are also checked because such characters were forbidden starting in
- * v19.
- *
- * Excluded databases are skipped since they won't appear in map.dat.
- */
-static void
-check_for_invalid_global_names(PGconn *conn,
- SimpleStringList *excluded_names)
-{
- PGresult *res;
- int i;
- PQExpBuffer names;
- int count = 0;
-
- res = executeQuery(conn,
- "SELECT datname AS objname, 'database' AS objtype "
- "FROM pg_catalog.pg_database "
- "WHERE datallowconn AND datconnlimit != -2 "
- "UNION ALL "
- "SELECT rolname AS objname, 'role' AS objtype "
- "FROM pg_catalog.pg_roles "
- "UNION ALL "
- "SELECT spcname AS objname, 'tablespace' AS objtype "
- "FROM pg_catalog.pg_tablespace");
-
- names = createPQExpBuffer();
-
- for (i = 0; i < PQntuples(res); i++)
- {
- char *objname = PQgetvalue(res, i, 0);
- char *objtype = PQgetvalue(res, i, 1);
-
- /* Skip excluded databases since they won't be in map.dat */
- if (strcmp(objtype, "database") == 0 &&
- simple_string_list_member(excluded_names, objname))
- continue;
-
- if (strpbrk(objname, "\n\r"))
- {
- appendPQExpBuffer(names, " %s: \"", objtype);
- for (char *p = objname; *p; p++)
- {
- if (*p == '\n')
- appendPQExpBufferStr(names, "\\n");
- else if (*p == '\r')
- appendPQExpBufferStr(names, "\\r");
- else
- appendPQExpBufferChar(names, *p);
- }
- appendPQExpBufferStr(names, "\"\n");
- count++;
- }
- }
-
- PQclear(res);
-
- if (count > 0)
- pg_fatal("database, role, or tablespace names contain a newline or carriage return character, which is not supported in non-plain-text dumps:\n%s",
- names->data);
-
- destroyPQExpBuffer(names);
-}
-
-
/*
* dumpTimestamp
*/
@@ -2397,45 +1890,3 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
-
-/*
- * parseDumpFormat
- *
- * This will validate dump formats.
- */
-static ArchiveFormat
-parseDumpFormat(const char *format)
-{
- if (pg_strcasecmp(format, "c") == 0)
- return archCustom;
- else if (pg_strcasecmp(format, "custom") == 0)
- return archCustom;
- else if (pg_strcasecmp(format, "d") == 0)
- return archDirectory;
- else if (pg_strcasecmp(format, "directory") == 0)
- return archDirectory;
- else if (pg_strcasecmp(format, "p") == 0)
- return archNull;
- else if (pg_strcasecmp(format, "plain") == 0)
- return archNull;
- else if (pg_strcasecmp(format, "t") == 0)
- return archTar;
- else if (pg_strcasecmp(format, "tar") == 0)
- return archTar;
- else
- pg_fatal("unrecognized output format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
- format);
-
- return archUnknown;
-}
-
-/*
- * createDumpId
- *
- * Return the next dumpId.
- */
-static int
-createDumpId(void)
-{
- return ++dumpIdVal;
-}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 48fdcb0fae1..84b8d410c9e 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump/pg_dumpall using the archiver
+ * from a backup archive created by pg_dump using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,16 +41,12 @@
#include "postgres_fe.h"
#include <ctype.h>
-#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
-#include "common/string.h"
-#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/option_utils.h"
-#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -58,41 +54,18 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
-static bool file_exists_in_directory(const char *dir, const char *filename);
-static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
- int numWorkers, bool append_data);
-static int restore_global_objects(const char *inputFileSpec, RestoreOptions *opts);
-
-static int restore_all_databases(const char *inputFileSpec,
- SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
-static int get_dbnames_list_to_restore(PGconn *conn,
- SimplePtrList *dbname_oid_list,
- SimpleStringList db_exclude_patterns);
-static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
- SimplePtrList *dbname_oid_list);
-
-/*
- * Stores a database OID and the corresponding name.
- */
-typedef struct DbOidName
-{
- Oid oid;
- char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
-} DbOidName;
-
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
+ int exit_code;
int numWorkers = 1;
+ Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
- int n_errors = 0;
- bool globals_only = false;
- SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -107,7 +80,6 @@ main(int argc, char **argv)
static int no_schema = 0;
static int no_security_labels = 0;
static int no_statistics = 0;
- static int no_globals = 0;
static int no_subscriptions = 0;
static int strict_names = 0;
static int statistics_only = 0;
@@ -117,7 +89,6 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
- {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -165,14 +136,12 @@ main(int argc, char **argv)
{"no-publications", no_argument, &no_publications, 1},
{"no-schema", no_argument, &no_schema, 1},
{"no-security-labels", no_argument, &no_security_labels, 1},
- {"no-globals", no_argument, &no_globals, 1},
{"no-subscriptions", no_argument, &no_subscriptions, 1},
{"no-statistics", no_argument, &no_statistics, 1},
{"statistics", no_argument, &with_statistics, 1},
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
{"restrict-key", required_argument, NULL, 6},
- {"exclude-database", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -201,7 +170,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -225,15 +194,14 @@ main(int argc, char **argv)
opts->filename = pg_strdup(optarg);
break;
case 'F':
- opts->formatName = pg_strdup(optarg);
- break;
- case 'g':
- /* restore only global sql commands. */
- globals_only = true;
+ if (strlen(optarg) != 0)
+ opts->formatName = pg_strdup(optarg);
break;
case 'h':
- opts->cparams.pghost = pg_strdup(optarg);
+ if (strlen(optarg) != 0)
+ opts->cparams.pghost = pg_strdup(optarg);
break;
+
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -262,7 +230,8 @@ main(int argc, char **argv)
break;
case 'p':
- opts->cparams.pgport = pg_strdup(optarg);
+ if (strlen(optarg) != 0)
+ opts->cparams.pgport = pg_strdup(optarg);
break;
case 'R':
/* no-op, still accepted for backwards compatibility */
@@ -352,10 +321,6 @@ main(int argc, char **argv)
opts->restrict_key = pg_strdup(optarg);
break;
- case 7: /* database patterns to skip */
- simple_string_list_append(&db_exclude_patterns, optarg);
- break;
-
default:
/* getopt_long already emitted a complaint */
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -382,20 +347,23 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
- /* --exclude-database and --globals-only are incompatible */
- check_mut_excl_opts(db_exclude_patterns.head, "--exclude-database",
- globals_only, "-g/--globals-only");
-
/* Should get at most one of -d and -f, else user is confused */
- check_mut_excl_opts(opts->cparams.dbname, "-d/--dbname",
- opts->filename, "-f/--file");
-
- /* --dbname and --restrict-key are incompatible */
- check_mut_excl_opts(opts->cparams.dbname, "-d/--dbname",
- opts->restrict_key, "--restrict-key");
-
if (opts->cparams.dbname)
+ {
+ if (opts->filename)
+ {
+ pg_log_error("options %s and %s cannot be used together",
+ "-d/--dbname", "-f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
+ if (opts->restrict_key)
+ pg_fatal("options %s and %s cannot be used together",
+ "-d/--dbname", "--restrict-key");
+
opts->useDB = 1;
+ }
else
{
/*
@@ -409,54 +377,56 @@ main(int argc, char **argv)
pg_fatal("invalid restrict key");
}
- /* *-only options are incompatible with each other */
- check_mut_excl_opts(data_only, "-a/--data-only",
- globals_only, "-g/--globals-only",
- schema_only, "-s/--schema-only",
- statistics_only, "--statistics-only");
+ /* reject conflicting "-only" options */
+ if (data_only && schema_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-s/--schema-only", "-a/--data-only");
+ if (schema_only && statistics_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-s/--schema-only", "--statistics-only");
+ if (data_only && statistics_only)
+ pg_fatal("options %s and %s cannot be used together",
+ "-a/--data-only", "--statistics-only");
- /* --no-* and *-only for same thing are incompatible */
- check_mut_excl_opts(data_only, "-a/--data-only",
- no_data, "--no-data");
- check_mut_excl_opts(globals_only, "-g/--globals-only",
- no_globals, "--no-globals");
- check_mut_excl_opts(schema_only, "-s/--schema-only",
- no_schema, "--no-schema");
- check_mut_excl_opts(statistics_only, "--statistics-only",
- no_statistics, "--no-statistics");
+ /* reject conflicting "-only" and "no-" options */
+ if (data_only && no_data)
+ pg_fatal("options %s and %s cannot be used together",
+ "-a/--data-only", "--no-data");
+ if (schema_only && no_schema)
+ pg_fatal("options %s and %s cannot be used together",
+ "-s/--schema-only", "--no-schema");
+ if (statistics_only && no_statistics)
+ pg_fatal("options %s and %s cannot be used together",
+ "--statistics-only", "--no-statistics");
- /* --statistics and --no-statistics are incompatible */
- check_mut_excl_opts(with_statistics, "--statistics",
- no_statistics, "--no-statistics");
+ /* reject conflicting "no-" options */
+ if (with_statistics && no_statistics)
+ pg_fatal("options %s and %s cannot be used together",
+ "--statistics", "--no-statistics");
- /* --statistics is incompatible with *-only (except --statistics-only) */
- check_mut_excl_opts(with_statistics, "--statistics",
- data_only, "-a/--data-only",
- globals_only, "-g/--globals-only",
- schema_only, "-s/--schema-only");
+ /* reject conflicting "only-" options */
+ if (data_only && with_statistics)
+ pg_fatal("options %s and %s cannot be used together",
+ "-a/--data-only", "--statistics");
+ if (schema_only && with_statistics)
+ pg_fatal("options %s and %s cannot be used together",
+ "-s/--schema-only", "--statistics");
- /* --clean and --data-only are incompatible */
- check_mut_excl_opts(opts->dropSchema, "-c/--clean",
- data_only, "-a/--data-only");
+ if (data_only && opts->dropSchema)
+ pg_fatal("options %s and %s cannot be used together",
+ "-c/--clean", "-a/--data-only");
- /*
- * --globals-only, --single-transaction, and --transaction-size are
- * incompatible.
- */
- check_mut_excl_opts(globals_only, "-g/--globals-only",
- opts->single_txn, "-1/--single-transaction",
- opts->txn_size, "--transaction-size");
-
- /* --exit-on-error and --globals-only are incompatible */
- check_mut_excl_opts(opts->exit_on_error, "--exit-on-error",
- globals_only, "-g/--globals-only");
+ if (opts->single_txn && opts->txn_size > 0)
+ pg_fatal("options %s and %s cannot be used together",
+ "-1/--single-transaction", "--transaction-size");
/*
* -C is not compatible with -1, because we can't create a database inside
* a transaction block.
*/
- check_mut_excl_opts(opts->createDB, "-C/--create",
- opts->single_txn, "-1/--single-transaction");
+ if (opts->createDB && opts->single_txn)
+ pg_fatal("options %s and %s cannot be used together",
+ "-C/--create", "-1/--single-transaction");
/* Can't do single-txn mode with multiple connections */
if (opts->single_txn && numWorkers > 1)
@@ -515,187 +485,6 @@ main(int argc, char **argv)
opts->formatName);
}
- /*
- * If toc.glo file is present, then restore all the databases from
- * map.dat, but skip restoring those matching --exclude-database patterns.
- */
- if (inputFileSpec != NULL &&
- (file_exists_in_directory(inputFileSpec, "toc.glo")))
- {
- char global_path[MAXPGPATH];
- RestoreOptions *tmpopts = pg_malloc0_object(RestoreOptions);
-
- opts->format = archUnknown;
-
- memcpy(tmpopts, opts, sizeof(RestoreOptions));
-
- /*
- * Can only use --list or --use-list options with a single database
- * dump.
- */
- if (opts->tocSummary)
- pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
- "-l/--list");
- if (opts->tocFile)
- pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
- "-L/--use-list");
-
- if (opts->strict_names)
- pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
- "--strict-names");
- if (globals_only && opts->dropSchema)
- pg_fatal("options %s and %s cannot be used together when restoring an archive created by pg_dumpall",
- "--clean", "-g/--globals-only");
-
- /*
- * For pg_dumpall archives, --clean implies --if-exists since global
- * objects may not exist in the target cluster.
- */
- if (opts->dropSchema && !opts->if_exists)
- {
- opts->if_exists = 1;
- pg_log_info("--if-exists is implied by --clean for pg_dumpall archives");
- }
-
- if (no_schema)
- pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
- "--no-schema");
-
- if (data_only)
- pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
- "-a/--data-only");
-
- if (statistics_only)
- pg_fatal("option %s cannot be used when restoring an archive created by pg_dumpall",
- "--statistics-only");
-
- if (!(opts->dumpSections & DUMP_PRE_DATA))
- pg_fatal("option %s cannot exclude %s when restoring a pg_dumpall archive",
- "--section", "--pre-data");
-
- /*
- * To restore from a pg_dumpall archive, -C (create database) option
- * must be specified unless we are only restoring globals or we are
- * skipping globals.
- */
- if (!no_globals && !globals_only && opts->createDB != 1)
- {
- pg_log_error("option %s must be specified when restoring an archive created by pg_dumpall",
- "-C/--create");
- pg_log_error_hint("Try \"%s --help\" for more information.", progname);
- pg_log_error_hint("Individual databases can be restored using their specific archives.");
- exit_nicely(1);
- }
-
- /*
- * Restore global objects, even if --exclude-database results in zero
- * databases to process. If 'globals-only' is set, exit immediately.
- */
- snprintf(global_path, MAXPGPATH, "%s/toc.glo", inputFileSpec);
-
- if (!no_globals)
- n_errors = restore_global_objects(global_path, tmpopts);
- else
- pg_log_info("skipping restore of global objects because %s was specified",
- "--no-globals");
-
- if (globals_only)
- pg_log_info("database restoring skipped because option %s was specified",
- "-g/--globals-only");
- else
- {
- /* Now restore all the databases from map.dat */
- n_errors = n_errors + restore_all_databases(inputFileSpec, db_exclude_patterns,
- opts, numWorkers);
- }
-
- /* Free db pattern list. */
- simple_string_list_destroy(&db_exclude_patterns);
- }
- else
- {
- if (db_exclude_patterns.head != NULL)
- {
- simple_string_list_destroy(&db_exclude_patterns);
- pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
- "--exclude-database");
- }
-
- if (globals_only)
- pg_fatal("option %s can be used only when restoring an archive created by pg_dumpall",
- "-g/--globals-only");
-
- /* Process if toc.glo file does not exist. */
- n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false);
- }
-
- /* Done, print a summary of ignored errors during restore. */
- if (n_errors)
- {
- pg_log_warning("errors ignored on restore: %d", n_errors);
- return 1;
- }
-
- return 0;
-}
-
-/*
- * restore_global_objects
- *
- * This restore all global objects.
- */
-static int
-restore_global_objects(const char *inputFileSpec, RestoreOptions *opts)
-{
- Archive *AH;
- int nerror = 0;
-
- /* Set format as custom so that toc.glo file can be read. */
- opts->format = archCustom;
- opts->txn_size = 0;
-
- AH = OpenArchive(inputFileSpec, opts->format);
-
- SetArchiveOptions(AH, NULL, opts);
-
- on_exit_close_archive(AH);
-
- /* Let the archiver know how noisy to be */
- AH->verbose = opts->verbose;
-
- /* Don't output TOC entry comments when restoring globals */
- ((ArchiveHandle *) AH)->noTocComments = 1;
-
- AH->exit_on_error = false;
-
- /* Parallel execution is not supported for global object restoration. */
- AH->numWorkers = 1;
-
- ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH, false);
-
- nerror = AH->n_errors;
-
- /* AH may be freed in CloseArchive? */
- CloseArchive(AH);
-
- return nerror;
-}
-
-/*
- * restore_one_database
- *
- * This will restore one database using toc.dat file.
- *
- * returns the number of errors while doing restore.
- */
-static int
-restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
- int numWorkers, bool append_data)
-{
- Archive *AH;
- int n_errors;
-
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -703,15 +492,9 @@ restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op. If we are
- * restoring multiple databases, then only update AH handle for cleanup as
- * the previous entry was already in the array and we had closed previous
- * connection, so we can use the same array slot.
+ * it's still NULL, the cleanup function will just be a no-op.
*/
- if (!append_data)
- on_exit_close_archive(AH);
- else
- replace_on_exit_close_archive(AH);
+ on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -731,21 +514,25 @@ restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH, append_data);
+ RestoreArchive(AH);
}
- n_errors = AH->n_errors;
+ /* done, print a summary of ignored errors */
+ if (AH->n_errors)
+ pg_log_warning("errors ignored on restore: %d", AH->n_errors);
/* AH may be freed in CloseArchive? */
+ exit_code = AH->n_errors ? 1 : 0;
+
CloseArchive(AH);
- return n_errors;
+ return exit_code;
}
static void
usage(const char *progname)
{
- printf(_("%s restores PostgreSQL databases from archives created by pg_dump or pg_dumpall.\n\n"), progname);
+ printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -763,7 +550,6 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
- printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -780,7 +566,6 @@ usage(const char *progname)
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --enable-row-security enable row security\n"));
- printf(_(" --exclude-database=PATTERN do not restore the specified database(s)\n"));
printf(_(" --filter=FILENAME restore or skip objects based on expressions\n"
" in FILENAME\n"));
printf(_(" --if-exists use IF EXISTS when dropping objects\n"));
@@ -788,7 +573,6 @@ usage(const char *progname)
printf(_(" --no-data do not restore data\n"));
printf(_(" --no-data-for-failed-tables do not restore data of tables that could not be\n"
" created\n"));
- printf(_(" --no-globals do not restore global objects (roles and tablespaces)\n"));
printf(_(" --no-policies do not restore row security policies\n"));
printf(_(" --no-publications do not restore publications\n"));
printf(_(" --no-schema do not restore schema\n"));
@@ -817,8 +601,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be\n"
- "combined and specified multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
+ "multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -923,409 +707,3 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
-
-/*
- * file_exists_in_directory
- *
- * Returns true if the file exists in the given directory.
- */
-static bool
-file_exists_in_directory(const char *dir, const char *filename)
-{
- struct stat st;
- char buf[MAXPGPATH];
-
- if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
- pg_fatal("directory name too long: \"%s\"", dir);
-
- return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
-}
-
-/*
- * get_dbnames_list_to_restore
- *
- * This will mark for skipping any entries from dbname_oid_list that pattern match an
- * entry in the db_exclude_patterns list.
- *
- * Returns the number of database to be restored.
- *
- */
-static int
-get_dbnames_list_to_restore(PGconn *conn,
- SimplePtrList *dbname_oid_list,
- SimpleStringList db_exclude_patterns)
-{
- int count_db = 0;
- PQExpBuffer query;
- PQExpBuffer db_lit;
- PGresult *res;
-
- query = createPQExpBuffer();
- db_lit = createPQExpBuffer();
-
- /*
- * Process one by one all dbnames and if specified to skip restoring, then
- * remove dbname from list.
- */
- for (SimplePtrListCell *db_cell = dbname_oid_list->head;
- db_cell; db_cell = db_cell->next)
- {
- DbOidName *dbidname = (DbOidName *) db_cell->ptr;
- bool skip_db_restore = false;
-
- resetPQExpBuffer(query);
- resetPQExpBuffer(db_lit);
-
- appendStringLiteralConn(db_lit, dbidname->str, conn);
-
- for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
- {
- /*
- * If there is an exact match then we don't need to try a pattern
- * match
- */
- if (pg_strcasecmp(dbidname->str, pat_cell->val) == 0)
- skip_db_restore = true;
- /* Otherwise, try a pattern match if there is a connection */
- else
- {
- int dotcnt;
-
- appendPQExpBufferStr(query, "SELECT 1 ");
- processSQLNamePattern(conn, query, pat_cell->val, false,
- false, NULL, db_lit->data,
- NULL, NULL, NULL, &dotcnt);
-
- if (dotcnt > 0)
- {
- pg_log_error("improper qualified name (too many dotted names): %s",
- dbidname->str);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- res = executeQuery(conn, query->data);
-
- if (PQntuples(res))
- {
- skip_db_restore = true;
- pg_log_info("database name \"%s\" matches --exclude-database pattern \"%s\"", dbidname->str, pat_cell->val);
- }
-
- PQclear(res);
- resetPQExpBuffer(query);
- }
-
- if (skip_db_restore)
- break;
- }
-
- /*
- * Mark db to be skipped or increment the counter of dbs to be
- * restored
- */
- if (skip_db_restore)
- {
- pg_log_info("excluding database \"%s\"", dbidname->str);
- dbidname->oid = InvalidOid;
- }
- else
- count_db++;
- }
-
- destroyPQExpBuffer(query);
- destroyPQExpBuffer(db_lit);
-
- return count_db;
-}
-
-/*
- * get_dbname_oid_list_from_mfile
- *
- * Open map.dat file and read line by line and then prepare a list of database
- * names and corresponding db_oid.
- *
- * Returns, total number of database names in map.dat file.
- */
-static int
-get_dbname_oid_list_from_mfile(const char *dumpdirpath,
- SimplePtrList *dbname_oid_list)
-{
- StringInfoData linebuf;
- FILE *pfile;
- char map_file_path[MAXPGPATH];
- int count = 0;
-
- /*
- * If there is no map.dat file in the dump, then return from here as there
- * is no database to restore.
- */
- if (!file_exists_in_directory(dumpdirpath, "map.dat"))
- {
- pg_log_info("database restoring is skipped because file \"%s\" does not exist in directory \"%s\"", "map.dat", dumpdirpath);
- return 0;
- }
-
- snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
-
- /* Open map.dat file. */
- pfile = fopen(map_file_path, PG_BINARY_R);
-
- if (pfile == NULL)
- pg_fatal("could not open file \"%s\": %m", map_file_path);
-
- initStringInfo(&linebuf);
-
- /* Append all the dbname/db_oid combinations to the list. */
- while (pg_get_line_buf(pfile, &linebuf))
- {
- Oid db_oid = InvalidOid;
- char *dbname;
- DbOidName *dbidname;
- int namelen;
- char *p = linebuf.data;
-
- /* look for the dboid. */
- while (isdigit((unsigned char) *p))
- p++;
-
- /* ignore lines that don't begin with a digit */
- if (p == linebuf.data)
- continue;
-
- if (*p == ' ')
- {
- sscanf(linebuf.data, "%u", &db_oid);
- p++;
- }
-
- /* dbname is the rest of the line */
- dbname = p;
- namelen = strlen(dbname);
-
- /* Strip trailing newline */
- if (namelen > 0 && dbname[namelen - 1] == '\n')
- dbname[--namelen] = '\0';
-
- /* Report error and exit if the file has any corrupted data. */
- if (!OidIsValid(db_oid) || namelen < 1)
- pg_fatal("invalid entry in file \"%s\" on line %d", map_file_path,
- count + 1);
-
- dbidname = pg_malloc(offsetof(DbOidName, str) + namelen + 1);
- dbidname->oid = db_oid;
- strlcpy(dbidname->str, dbname, namelen + 1);
-
- pg_log_info("found database \"%s\" (OID: %u) in file \"%s\"",
- dbidname->str, db_oid, map_file_path);
-
- simple_ptr_list_append(dbname_oid_list, dbidname);
- count++;
- }
-
- /* Close map.dat file. */
- fclose(pfile);
-
- pfree(linebuf.data);
-
- return count;
-}
-
-/*
- * restore_all_databases
- *
- * This will restore databases those dumps are present in
- * directory based on map.dat file mapping.
- *
- * This will skip restoring for databases that are specified with
- * exclude-database option.
- *
- * returns, number of errors while doing restore.
- */
-static int
-restore_all_databases(const char *inputFileSpec,
- SimpleStringList db_exclude_patterns, RestoreOptions *opts,
- int numWorkers)
-{
- SimplePtrList dbname_oid_list = {NULL, NULL};
- int num_db_restore = 0;
- int num_total_db;
- int n_errors_total = 0;
- char *connected_db = NULL;
- PGconn *conn = NULL;
- RestoreOptions *original_opts = pg_malloc0_object(RestoreOptions);
- RestoreOptions *tmpopts = pg_malloc0_object(RestoreOptions);
-
- memcpy(original_opts, opts, sizeof(RestoreOptions));
-
- /* Save db name to reuse it for all the database. */
- if (opts->cparams.dbname)
- connected_db = opts->cparams.dbname;
-
- num_total_db = get_dbname_oid_list_from_mfile(inputFileSpec, &dbname_oid_list);
-
- pg_log_info(ngettext("found %d database name in \"%s\"",
- "found %d database names in \"%s\"",
- num_total_db),
- num_total_db, "map.dat");
-
- /*
- * If exclude-patterns is given, connect to the database to process them.
- */
- if (db_exclude_patterns.head != NULL)
- {
- if (opts->cparams.dbname)
- {
- conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
- opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
- false, progname, NULL, NULL, NULL, NULL);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
- }
-
- if (!conn)
- {
- pg_log_info("trying to connect to database \"%s\"", "postgres");
-
- conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
- opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
- false, progname, NULL, NULL, NULL, NULL);
-
- /* Try with template1. */
- if (!conn)
- {
- pg_log_info("trying to connect to database \"%s\"", "template1");
-
- conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
- opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
- false, progname, NULL, NULL, NULL, NULL);
- if (!conn)
- {
- pg_log_error("could not connect to databases \"postgres\" or \"template1\"\n"
- "Please specify an alternative database.");
- pg_log_error_hint("Try \"%s --help\" for more information.", progname);
- exit_nicely(1);
- }
- }
- }
-
- /* Filter the db list according to the exclude patterns. */
- num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
- db_exclude_patterns);
- PQfinish(conn);
- }
- else
- num_db_restore = num_total_db;
-
- /* Exit if no db needs to be restored. */
- if (num_db_restore == 0)
- {
- pg_log_info(ngettext("no database needs restoring out of %d database",
- "no database needs restoring out of %d databases", num_total_db),
- num_total_db);
- pg_free(original_opts);
- pg_free(tmpopts);
- return 0;
- }
-
- pg_log_info("need to restore %d databases out of %d databases", num_db_restore, num_total_db);
-
- /*
- * We have a list of databases to restore after processing the
- * exclude-database switch(es). Now we can restore them one by one.
- */
- for (SimplePtrListCell *db_cell = dbname_oid_list.head;
- db_cell; db_cell = db_cell->next)
- {
- DbOidName *dbidname = (DbOidName *) db_cell->ptr;
- char subdirpath[MAXPGPATH];
- char subdirdbpath[MAXPGPATH];
- char dbfilename[MAXPGPATH];
- int n_errors;
-
- /* ignore dbs marked for skipping */
- if (dbidname->oid == InvalidOid)
- continue;
-
- /*
- * Since pg_backup_archiver.c may modify RestoreOptions during the
- * previous restore, we must provide a fresh copy of the original
- * "opts" for each call to restore_one_database.
- */
- memcpy(tmpopts, original_opts, sizeof(RestoreOptions));
-
- snprintf(subdirdbpath, MAXPGPATH, "%s/databases", inputFileSpec);
-
- /*
- * Look for the database dump file/dir. If there is an {oid}.tar or
- * {oid}.dmp file, use it. Otherwise try to use a directory called
- * {oid}
- */
- snprintf(dbfilename, MAXPGPATH, "%u.tar", dbidname->oid);
- if (file_exists_in_directory(subdirdbpath, dbfilename))
- snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", inputFileSpec, dbidname->oid);
- else
- {
- snprintf(dbfilename, MAXPGPATH, "%u.dmp", dbidname->oid);
-
- if (file_exists_in_directory(subdirdbpath, dbfilename))
- snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", inputFileSpec, dbidname->oid);
- else
- snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", inputFileSpec, dbidname->oid);
- }
-
- pg_log_info("restoring database \"%s\"", dbidname->str);
-
- /* If database is already created, then don't set createDB flag. */
- if (tmpopts->cparams.dbname)
- {
- PGconn *test_conn;
-
- test_conn = ConnectDatabase(dbidname->str, NULL, tmpopts->cparams.pghost,
- tmpopts->cparams.pgport, tmpopts->cparams.username, TRI_DEFAULT,
- false, progname, NULL, NULL, NULL, NULL);
- if (test_conn)
- {
- PQfinish(test_conn);
-
- /* Use already created database for connection. */
- tmpopts->createDB = 0;
- tmpopts->cparams.dbname = dbidname->str;
- }
- else
- {
- if (!tmpopts->createDB)
- {
- pg_log_info("skipping restore of database \"%s\": database does not exist and %s was not specified",
- dbidname->str, "-C/--create");
- continue;
- }
-
- /* We'll have to create it */
- tmpopts->createDB = 1;
- tmpopts->cparams.dbname = connected_db;
- }
- }
-
- /* Restore the single database. */
- n_errors = restore_one_database(subdirpath, tmpopts, numWorkers, true);
-
- n_errors_total += n_errors;
-
- /* Print a summary of ignored errors during single database restore. */
- if (n_errors)
- pg_log_warning("errors ignored on database \"%s\" restore: %d", dbidname->str, n_errors);
- }
-
- /* Log number of processed databases. */
- pg_log_info("number of restored databases is %d", num_db_restore);
-
- /* Free dbname and dboid list. */
- simple_ptr_list_destroy(&dbname_oid_list);
-
- pg_free(original_opts);
- pg_free(tmpopts);
-
- return n_errors_total;
-}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
index 687e842cde9..b2558046224 100644
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -87,8 +87,8 @@ command_fails_like(
command_fails_like(
[ 'pg_restore', '-s', '-a', '-f -' ],
- qr/\Qpg_restore: error: options -a\/--data-only and -s\/--schema-only cannot be used together\E/,
- 'pg_restore: options -a/--data-only and -s/--schema-only cannot be used together'
+ qr/\Qpg_restore: error: options -s\/--schema-only and -a\/--data-only cannot be used together\E/,
+ 'pg_restore: options -s/--schema-only and -a/--data-only cannot be used together'
);
command_fails_like(
@@ -205,12 +205,7 @@ command_fails_like(
command_fails_like(
[ 'pg_restore', '-f -', '-F', 'garbage' ],
qr/\Qpg_restore: error: unrecognized archive format "garbage";\E/,
- 'pg_restore: unrecognized archive format');
-
-command_fails_like(
- [ 'pg_restore', '-f -', '-F', '' ],
- qr/\Qpg_restore: error: unrecognized archive format "";\E/,
- 'pg_restore: empty archive format');
+ 'pg_dump: unrecognized archive format');
command_fails_like(
[ 'pg_dump', '--on-conflict-do-nothing' ],
@@ -251,105 +246,8 @@ command_fails_like(
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
- qr/\Qpg_dumpall: error: options --exclude-database and -g\/--globals-only cannot be used together\E/,
- 'pg_dumpall: options --exclude-database and -g/--globals-only cannot be used together'
+ qr/\Qpg_dumpall: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
-command_fails_like(
- [ 'pg_dumpall', '-a', '--no-data' ],
- qr/\Qpg_dumpall: error: options -a\/--data-only and --no-data cannot be used together\E/,
- 'pg_dumpall: options -a\/--data-only and --no-data cannot be used together'
-);
-
-command_fails_like(
- [ 'pg_dumpall', '-s', '--no-schema' ],
- qr/\Qpg_dumpall: error: options -s\/--schema-only and --no-schema cannot be used together\E/,
- 'pg_dumpall: options -s\/--schema-only and --no-schema cannot be used together'
-);
-
-command_fails_like(
- [ 'pg_dumpall', '--statistics-only', '--no-statistics' ],
- qr/\Qpg_dumpall: error: options --statistics-only and --no-statistics cannot be used together\E/,
- 'pg_dumpall: options --statistics-only and --no-statistics cannot be used together'
-);
-
-command_fails_like(
- [ 'pg_dumpall', '--statistics', '--no-statistics' ],
- qr/\Qpg_dumpall: error: options --statistics and --no-statistics cannot be used together\E/,
- 'pg_dumpall: options --statistics-only and --no-statistics cannot be used together'
-);
-
-command_fails_like(
- [ 'pg_dumpall', '--statistics', '--tablespaces-only' ],
- qr/\Qpg_dumpall: error: options --statistics and -t\/--tablespaces-only cannot be used together\E/,
- 'pg_dumpall: options --statistics and -t\/--tablespaces-only cannot be used together'
-);
-
-command_fails_like(
- [ 'pg_dumpall', '--format', 'x' ],
- qr/\Qpg_dumpall: error: unrecognized output format "x";\E/,
- 'pg_dumpall: unrecognized output format');
-
-command_fails_like(
- [ 'pg_dumpall', '--format', 'd', '--restrict-key=uu', '-f dumpfile' ],
- qr/\Qpg_dumpall: error: option --restrict-key can only be used with --format=plain\E/,
- 'pg_dumpall: --restrict-key can only be used with plain dump format');
-
-command_fails_like(
- [
- 'pg_dumpall', '--format', 'd', '--globals-only',
- '--clean', '-f', 'dumpfile'
- ],
- qr/\Qpg_dumpall: error: options --clean and -g\/--globals-only cannot be used together in non-text dump\E/,
- 'pg_dumpall: --clean and -g/--globals-only cannot be used together in non-text dump'
-);
-
-command_fails_like(
- [ 'pg_dumpall', '--format', 'd' ],
- qr/\Qpg_dumpall: error: option -F\/--format=d|c|t requires option -f\/--file\E/,
- 'pg_dumpall: non-plain format requires --file option');
-
-command_fails_like(
- [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
- qr/\Qpg_restore: error: options --exclude-database and -g\/--globals-only cannot be used together\E/,
- 'pg_restore: options --exclude-database and -g/--globals-only cannot be used together'
-);
-
-command_fails_like(
- [ 'pg_restore', '--data-only', '--globals-only', '-d', 'xxx' ],
- qr/\Qpg_restore: error: options -a\/--data-only and -g\/--globals-only cannot be used together\E/,
- 'pg_restore: error: options -a/--data-only and -g/--globals-only cannot be used together'
-);
-
-command_fails_like(
- [ 'pg_restore', '--schema-only', '--globals-only', '-d', 'xxx' ],
- qr/\Qpg_restore: error: options -g\/--globals-only and -s\/--schema-only cannot be used together\E/,
- 'pg_restore: error: options -g/--globals-only and -s/--schema-only cannot be used together'
-);
-
-command_fails_like(
- [ 'pg_restore', '--statistics-only', '--globals-only', '-d', 'xxx' ],
- qr/\Qpg_restore: error: options -g\/--globals-only and --statistics-only cannot be used together\E/,
- 'pg_restore: error: options -g/--globals-only and --statistics-only cannot be used together'
-);
-
-command_fails_like(
- [ 'pg_restore', '--exclude-database=foo', '-d', 'xxx', 'dumpdir' ],
- qr/\Qpg_restore: error: option --exclude-database can be used only when restoring an archive created by pg_dumpall\E/,
- 'When option --exclude-database is used in pg_restore with dump of pg_dump'
-);
-
-command_fails_like(
- [ 'pg_restore', '--globals-only', '-d', 'xxx', 'dumpdir' ],
- qr/\Qpg_restore: error: option -g\/--globals-only can be used only when restoring an archive created by pg_dumpall\E/,
- 'When option --globals-only is used in pg_restore with the dump of pg_dump'
-);
-
-command_fails_like(
- [
- 'pg_restore', '--globals-only', '--no-globals', '-d', 'xxx',
- 'dumpdir'
- ],
- qr/\Qpg_restore: error: options -g\/--globals-only and --no-globals cannot be used together\E/,
- 'options --no-globals and --globals-only cannot be used together');
done_testing();
diff --git a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl
index cecf0442088..72a7c90c6af 100644
--- a/src/bin/pg_dump/t/005_pg_dump_filterfile.pl
+++ b/src/bin/pg_dump/t/005_pg_dump_filterfile.pl
@@ -570,8 +570,8 @@ command_fails_like(
'--filter' => "$tempdir/inputfile.txt",
'--globals-only'
],
- qr/\Qpg_dumpall: error: options --exclude-database and -g\/--globals-only cannot be used together\E/,
- 'pg_dumpall: options --exclude-database and -g/--globals-only cannot be used together'
+ qr/\Qpg_dumpall: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
# Test invalid filter command
diff --git a/src/bin/pg_dump/t/007_pg_dumpall.pl b/src/bin/pg_dump/t/007_pg_dumpall.pl
deleted file mode 100644
index 22f11a13a9a..00000000000
--- a/src/bin/pg_dump/t/007_pg_dumpall.pl
+++ /dev/null
@@ -1,661 +0,0 @@
-# Copyright (c) 2021-2026, PostgreSQL Global Development Group
-
-use strict;
-use warnings FATAL => 'all';
-
-use PostgreSQL::Test::Cluster;
-use PostgreSQL::Test::Utils;
-use Test::More;
-
-my $tempdir = PostgreSQL::Test::Utils::tempdir;
-my $run_db = 'postgres';
-my $sep = $windows_os ? "\\" : "/";
-
-# Tablespace locations used by "restore_tablespace" test case.
-my $tablespace1 = "${tempdir}${sep}tbl1";
-my $tablespace2 = "${tempdir}${sep}tbl2";
-mkdir($tablespace1) || die "mkdir $tablespace1 $!";
-mkdir($tablespace2) || die "mkdir $tablespace2 $!";
-
-# escape tablespace locations on Windows.
-my $tablespace2_orig = $tablespace2;
-$tablespace1 = $windows_os ? ($tablespace1 =~ s/\\/\\\\/gr) : $tablespace1;
-$tablespace2 = $windows_os ? ($tablespace2 =~ s/\\/\\\\/gr) : $tablespace2;
-
-# Where pg_dumpall will be executed.
-my $node = PostgreSQL::Test::Cluster->new('node');
-$node->init;
-$node->start;
-
-
-###############################################################
-# Definition of the pg_dumpall test cases to run.
-#
-# Each of these test cases are named and those names are used for fail
-# reporting and also to save the dump and restore information needed for the
-# test to assert.
-#
-# The "setup_sql" is a psql valid script that contains SQL commands to execute
-# before of actually execute the tests. The setups are all executed before of
-# any test execution.
-#
-# The "dump_cmd" and "restore_cmd" are the commands that will be executed. The
-# "restore_cmd" must have the --file flag to save the restore output so that we
-# can assert on it.
-#
-# The "like" and "unlike" is a regexp that is used to match the pg_restore
-# output. It must have at least one of then filled per test cases but it also
-# can have both. See "excluding_databases" test case for example.
-my %pgdumpall_runs = (
- restore_roles => {
- setup_sql => '
- CREATE ROLE dumpall WITH ENCRYPTED PASSWORD \'admin\' SUPERUSER;
- CREATE ROLE dumpall2 WITH REPLICATION CONNECTION LIMIT 10;',
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_roles",
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_roles.sql",
- "$tempdir/restore_roles",
- ],
- like => qr/
- \s*\QCREATE ROLE dumpall2;\E
- \s*\QALTER ROLE dumpall2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB NOLOGIN REPLICATION NOBYPASSRLS CONNECTION LIMIT 10;\E
- /xm
- },
-
- restore_tablespace => {
- setup_sql => "
- CREATE ROLE tap;
- CREATE TABLESPACE tbl1 OWNER tap LOCATION '$tablespace1';
- CREATE TABLESPACE tbl2 OWNER tap LOCATION '$tablespace2' WITH (seq_page_cost=1.0);",
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_tablespace",
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_tablespace.sql",
- "$tempdir/restore_tablespace",
- ],
- # Match "E" as optional since it is added on LOCATION when running on
- # Windows.
- like => qr/^
- \n\QCREATE TABLESPACE tbl2 OWNER tap LOCATION \E(?:E)?\Q'$tablespace2_orig';\E
- \n\QALTER TABLESPACE tbl2 SET (seq_page_cost=1.0);\E
- /xm,
- },
-
- restore_grants => {
- setup_sql => "
- CREATE DATABASE tapgrantsdb;
- CREATE SCHEMA private;
- CREATE SEQUENCE serial START 101;
- CREATE FUNCTION fn() RETURNS void AS \$\$
- BEGIN
- END;
- \$\$ LANGUAGE plpgsql;
- CREATE ROLE super;
- CREATE ROLE grant1;
- CREATE ROLE grant2;
- CREATE ROLE grant3;
- CREATE ROLE grant4;
- CREATE ROLE grant5;
- CREATE ROLE grant6;
- CREATE ROLE grant7;
- CREATE ROLE grant8;
-
- CREATE TABLE t (id int);
- INSERT INTO t VALUES (1), (2), (3), (4);
-
- GRANT SELECT ON TABLE t TO grant1;
- GRANT INSERT ON TABLE t TO grant2;
- GRANT ALL PRIVILEGES ON TABLE t to grant3;
- GRANT CONNECT, CREATE ON DATABASE tapgrantsdb TO grant4;
- GRANT USAGE, CREATE ON SCHEMA private TO grant5;
- GRANT USAGE, SELECT, UPDATE ON SEQUENCE serial TO grant6;
- GRANT super TO grant7;
- GRANT EXECUTE ON FUNCTION fn() TO grant8;
- ",
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_grants",
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_grants.sql",
- "$tempdir/restore_grants",
- ],
- like => qr/^
- \n\QGRANT ALL ON SCHEMA private TO grant5;\E
- (.*\n)*
- \n\QGRANT ALL ON FUNCTION public.fn() TO grant8;\E
- (.*\n)*
- \n\QGRANT ALL ON SEQUENCE public.serial TO grant6;\E
- (.*\n)*
- \n\QGRANT SELECT ON TABLE public.t TO grant1;\E
- \n\QGRANT INSERT ON TABLE public.t TO grant2;\E
- \n\QGRANT ALL ON TABLE public.t TO grant3;\E
- (.*\n)*
- \n\QGRANT CREATE,CONNECT ON DATABASE tapgrantsdb TO grant4;\E
- /xm,
- },
-
- excluding_databases => {
- setup_sql => 'CREATE DATABASE db1;
- \c db1
- CREATE TABLE t1 (id int);
- INSERT INTO t1 VALUES (1), (2), (3), (4);
- CREATE TABLE t2 (id int);
- INSERT INTO t2 VALUES (1), (2), (3), (4);
-
- CREATE DATABASE db2;
- \c db2
- CREATE TABLE t3 (id int);
- INSERT INTO t3 VALUES (1), (2), (3), (4);
- CREATE TABLE t4 (id int);
- INSERT INTO t4 VALUES (1), (2), (3), (4);
-
- CREATE DATABASE dbex3;
- \c dbex3
- CREATE TABLE t5 (id int);
- INSERT INTO t5 VALUES (1), (2), (3), (4);
- CREATE TABLE t6 (id int);
- INSERT INTO t6 VALUES (1), (2), (3), (4);
-
- CREATE DATABASE dbex4;
- \c dbex4
- CREATE TABLE t7 (id int);
- INSERT INTO t7 VALUES (1), (2), (3), (4);
- CREATE TABLE t8 (id int);
- INSERT INTO t8 VALUES (1), (2), (3), (4);
-
- CREATE DATABASE db5;
- \c db5
- CREATE TABLE t9 (id int);
- INSERT INTO t9 VALUES (1), (2), (3), (4);
- CREATE TABLE t10 (id int);
- INSERT INTO t10 VALUES (1), (2), (3), (4);
- ',
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'directory',
- '--file' => "$tempdir/excluding_databases",
- '--exclude-database' => 'dbex*',
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'directory',
- '--file' => "$tempdir/excluding_databases.sql",
- '--exclude-database' => 'db5',
- "$tempdir/excluding_databases",
- ],
- like => qr/^
- \n\QCREATE DATABASE db1\E
- (.*\n)*
- \n\QCREATE TABLE public.t1 (\E
- (.*\n)*
- \n\QCREATE TABLE public.t2 (\E
- (.*\n)*
- \n\QCREATE DATABASE db2\E
- (.*\n)*
- \n\QCREATE TABLE public.t3 (\E
- (.*\n)*
- \n\QCREATE TABLE public.t4 (/xm,
- unlike => qr/^
- \n\QCREATE DATABASE db3\E
- (.*\n)*
- \n\QCREATE TABLE public.t5 (\E
- (.*\n)*
- \n\QCREATE TABLE public.t6 (\E
- (.*\n)*
- \n\QCREATE DATABASE db4\E
- (.*\n)*
- \n\QCREATE TABLE public.t7 (\E
- (.*\n)*
- \n\QCREATE TABLE public.t8 (\E
- \n\QCREATE DATABASE db5\E
- (.*\n)*
- \n\QCREATE TABLE public.t9 (\E
- (.*\n)*
- \n\QCREATE TABLE public.t10 (\E
- /xm,
- },
-
- format_directory => {
- setup_sql => "CREATE TABLE format_directory(a int, b boolean, c text);
- INSERT INTO format_directory VALUES (1, true, 'name1'), (2, false, 'name2');",
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'directory',
- '--file' => "$tempdir/format_directory",
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'directory',
- '--file' => "$tempdir/format_directory.sql",
- "$tempdir/format_directory",
- ],
- like => qr/^\n\QCOPY public.format_directory (a, b, c) FROM stdin;/xm
- },
-
- format_tar => {
- setup_sql => "CREATE TABLE format_tar(a int, b boolean, c text);
- INSERT INTO format_tar VALUES (1, false, 'name3'), (2, true, 'name4');",
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'tar',
- '--file' => "$tempdir/format_tar",
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'tar',
- '--file' => "$tempdir/format_tar.sql",
- "$tempdir/format_tar",
- ],
- like => qr/^\n\QCOPY public.format_tar (a, b, c) FROM stdin;/xm
- },
-
- format_custom => {
- setup_sql => "CREATE TABLE format_custom(a int, b boolean, c text);
- INSERT INTO format_custom VALUES (1, false, 'name5'), (2, true, 'name6');",
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'custom',
- '--file' => "$tempdir/format_custom",
- ],
- restore_cmd => [
- 'pg_restore', '-C',
- '--format' => 'custom',
- '--file' => "$tempdir/format_custom.sql",
- "$tempdir/format_custom",
- ],
- like => qr/^ \n\QCOPY public.format_custom (a, b, c) FROM stdin;/xm
- },
-
- dump_globals_only => {
- setup_sql => "CREATE TABLE format_dir(a int, b boolean, c text);
- INSERT INTO format_dir VALUES (1, false, 'name5'), (2, true, 'name6');",
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'directory',
- '--globals-only',
- '--file' => "$tempdir/dump_globals_only",
- ],
- restore_cmd => [
- 'pg_restore', '-C', '--globals-only',
- '--format' => 'directory',
- '--file' => "$tempdir/dump_globals_only.sql",
- "$tempdir/dump_globals_only",
- ],
- like => qr/
- ^\s*\QCREATE ROLE dumpall;\E\s*\n
- /xm
- },
-
- restore_no_globals => {
- setup_sql => "CREATE TABLE no_globals_test(a int, b text);
- INSERT INTO no_globals_test VALUES (1, 'hello'), (2, 'world');",
- dump_cmd => [
- 'pg_dumpall',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_no_globals",
- ],
- restore_cmd => [
- 'pg_restore', '-C', '--no-globals',
- '--format' => 'directory',
- '--file' => "$tempdir/restore_no_globals.sql",
- "$tempdir/restore_no_globals",
- ],
- like => qr/^\n\QCOPY public.no_globals_test (a, b) FROM stdin;\E/xm,
- unlike => qr/^\QCREATE ROLE dumpall;\E/xm,
- },);
-
-# First execute the setup_sql
-foreach my $run (sort keys %pgdumpall_runs)
-{
- if ($pgdumpall_runs{$run}->{setup_sql})
- {
- $node->safe_psql($run_db, $pgdumpall_runs{$run}->{setup_sql});
- }
-}
-
-# Execute the tests
-foreach my $run (sort keys %pgdumpall_runs)
-{
- # Create a new target cluster to pg_restore each test case run so that we
- # don't need to take care of the cleanup from the target cluster after each
- # run.
- my $target_node = PostgreSQL::Test::Cluster->new("target_$run");
- $target_node->init;
- $target_node->start;
-
- # Dumpall from node cluster.
- $node->command_ok(\@{ $pgdumpall_runs{$run}->{dump_cmd} },
- "$run: pg_dumpall runs");
-
- # Restore the dump on "target_node" cluster.
- my @restore_cmd = (
- @{ $pgdumpall_runs{$run}->{restore_cmd} },
- '--host', $target_node->host, '--port', $target_node->port);
-
- my ($stdout, $stderr) = run_command(\@restore_cmd);
-
- # pg_restore --file output file.
- my $output_file = slurp_file("$tempdir/${run}.sql");
-
- if ( !($pgdumpall_runs{$run}->{like})
- && !($pgdumpall_runs{$run}->{unlike}))
- {
- die "missing \"like\" or \"unlike\" in test \"$run\"";
- }
-
- if ($pgdumpall_runs{$run}->{like})
- {
- like($output_file, $pgdumpall_runs{$run}->{like}, "should dump $run");
- }
-
- if ($pgdumpall_runs{$run}->{unlike})
- {
- unlike(
- $output_file,
- $pgdumpall_runs{$run}->{unlike},
- "should not dump $run");
- }
-
- $target_node->stop;
- $target_node->clean_node;
-}
-
-# Some negative test case with dump of pg_dumpall and restore using pg_restore
-# report an error when -C is not used in pg_restore with dump of pg_dumpall
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom",
- '--format' => 'custom',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: option -C\/--create must be specified when restoring an archive created by pg_dumpall\E/,
- 'When -C is not used in pg_restore with dump of pg_dumpall');
-
-# report an error when \l/--list option is used with dump of pg_dumpall
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--list',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: option -l\/--list cannot be used when restoring an archive created by pg_dumpall\E/,
- 'When --list is used in pg_restore with dump of pg_dumpall');
-
-# report an error when -L/--use-list option is used with dump of pg_dumpall
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--use-list' => 'use',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: option -L\/--use-list cannot be used when restoring an archive created by pg_dumpall\E/,
- 'When -L/--use-list is used in pg_restore with dump of pg_dumpall');
-
-# report an error when --strict-names option is used with dump of pg_dumpall
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--strict-names',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: option --strict-names cannot be used when restoring an archive created by pg_dumpall\E/,
- 'When --strict-names is used in pg_restore with dump of pg_dumpall');
-
-# report an error when --clean and -g/--globals-only are used in pg_restore with dump of pg_dumpall
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--clean',
- '--globals-only',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: options --clean and -g\/--globals-only cannot be used together when restoring an archive created by pg_dumpall\E/,
- 'When --clean and -g/--globals-only are used in pg_restore with dump of pg_dumpall'
-);
-
-# report an error when non-exist database is given with -d option
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '-d' => 'dbpq',
- ],
- qr/\QFATAL: database "dbpq" does not exist\E/,
- 'When non-existent database is given with -d option in pg_restore with dump of pg_dumpall'
-);
-
-# report an error when --no-schema is used with dump of pg_dumpall
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--no-schema',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: option --no-schema cannot be used when restoring an archive created by pg_dumpall\E/,
- 'When --no-schema is used in pg_restore with dump of pg_dumpall');
-
-# report an error when --data-only is used with dump of pg_dumpall
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--data-only',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: option -a\/--data-only cannot be used when restoring an archive created by pg_dumpall\E/,
- 'When --data-only is used in pg_restore with dump of pg_dumpall');
-
-# report an error when --statistics-only is used with dump of pg_dumpall
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--statistics-only',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: option --statistics-only cannot be used when restoring an archive created by pg_dumpall\E/,
- 'When --statistics-only is used in pg_restore with dump of pg_dumpall');
-
-# report an error when --section excludes pre-data with dump of pg_dumpall
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--section' => 'post-data',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: option --section cannot exclude --pre-data when restoring a pg_dumpall archive\E/,
- 'When --section=post-data is used in pg_restore with dump of pg_dumpall');
-
-# report an error when --globals-only and --data-only are used together
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--globals-only',
- '--data-only',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: options -a\/--data-only and -g\/--globals-only cannot be used together\E/,
- 'When --globals-only and --data-only are used together');
-
-# report an error when --globals-only and --schema-only are used together
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--globals-only',
- '--schema-only',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: options -g\/--globals-only and -s\/--schema-only cannot be used together\E/,
- 'When --globals-only and --schema-only are used together');
-
-# report an error when --globals-only and --statistics-only are used together
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--globals-only',
- '--statistics-only',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: options -g\/--globals-only and --statistics-only cannot be used together\E/,
- 'When --globals-only and --statistics-only are used together');
-
-# report an error when --globals-only and --statistics are used together
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--globals-only',
- '--statistics',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: options --statistics and -g\/--globals-only cannot be used together\E/,
- 'When --globals-only and --statistics are used together');
-
-# report an error when --globals-only and --exit-on-error are used together
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--globals-only',
- '--exit-on-error',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: options --exit-on-error and -g\/--globals-only cannot be used together\E/,
- 'When --globals-only and --exit-on-error are used together');
-
-# report an error when --globals-only and --single-transaction are used together
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--globals-only',
- '--single-transaction',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: options -g\/--globals-only and -1\/--single-transaction cannot be used together\E/,
- 'When --globals-only and --single-transaction are used together');
-
-# report an error when --globals-only and --transaction-size are used together
-$node->command_fails_like(
- [
- 'pg_restore',
- "$tempdir/format_custom", '-C',
- '--format' => 'custom',
- '--globals-only',
- '--transaction-size' => '100',
- '--file' => "$tempdir/error_test.sql",
- ],
- qr/\Qpg_restore: error: options -g\/--globals-only and --transaction-size cannot be used together\E/,
- 'When --globals-only and --transaction-size are used together');
-
-# verify map.dat preamble exists
-my $map_dat_content = slurp_file("$tempdir/format_directory/map.dat");
-like(
- $map_dat_content,
- qr/^# map\.dat\n.*# This file maps oids to database names/ms,
- 'map.dat contains expected preamble');
-
-# verify commenting out a line in map.dat skips that database
-$node->safe_psql(
- $run_db, 'CREATE DATABASE comment_test_db;
-\c comment_test_db
-CREATE TABLE comment_test_table (id int);');
-
-$node->command_ok(
- [
- 'pg_dumpall',
- '--format' => 'directory',
- '--file' => "$tempdir/comment_test",
- ],
- 'pg_dumpall for comment test');
-
-# Modify map.dat to comment out the comment_test_db entry
-my $map_content = slurp_file("$tempdir/comment_test/map.dat");
-$map_content =~ s/^(\d+ comment_test_db)$/# $1/m;
-open(my $fh, '>', "$tempdir/comment_test/map.dat")
- or die "Cannot open map.dat: $!";
-print $fh $map_content;
-close($fh);
-
-# Create a target node and restore - commented db should be skipped
-my $target_comment = PostgreSQL::Test::Cluster->new("target_comment");
-$target_comment->init;
-$target_comment->start;
-
-$node->command_ok(
- [
- 'pg_restore', '-C',
- '--format' => 'directory',
- '--file' => "$tempdir/comment_test_restore.sql",
- '--host', $target_comment->host,
- '--port', $target_comment->port,
- "$tempdir/comment_test",
- ],
- 'pg_restore with commented out database in map.dat');
-
-my $restore_output = slurp_file("$tempdir/comment_test_restore.sql");
-unlike(
- $restore_output,
- qr/CREATE DATABASE comment_test_db/,
- 'commented out database in map.dat is not restored');
-
-# Test that --clean implies --if-exists for pg_dumpall archives
-$node->command_ok(
- [
- 'pg_restore', '-C',
- '--format' => 'custom',
- '--clean',
- '--file' => "$tempdir/clean_test.sql",
- "$tempdir/format_custom",
- ],
- 'pg_restore with --clean on pg_dumpall archive');
-
-my $clean_output = slurp_file("$tempdir/clean_test.sql");
-like(
- $clean_output,
- qr/DROP ROLE IF EXISTS/,
- '--clean implies --if-exists: DROP ROLE IF EXISTS in output');
-
-$node->stop('fast');
-
-done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f9eb23e52c9..1969d467c1d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -648,7 +648,6 @@ DatumTupleFields
DbInfo
DbInfoArr
DbLocaleInfo
-DbOidName
DdlOptType
DdlOption
DeClonePtrType
--
2.43.0
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-06-17 21:59 Noah Misch <[email protected]>
parent: Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Noah Misch @ 2026-06-17 21:59 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: jian he <[email protected]>; Mahendra Singh Thalor <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On Mon, Jun 15, 2026 at 04:11:16PM -0400, Andrew Dunstan wrote:
> On 2026-06-06 Sa 8:02 PM, Noah Misch wrote:
> > On Thu, Feb 26, 2026 at 09:02:48AM -0500, Andrew Dunstan wrote:
> > > pushed with a slight tweak.
> > Having now reviewed commit 763aaa0, I don't think it's ready to remain part of
> > v19. While some points from my v18 review are now resolved, other points
> > still seem unresolved. I didn't find discussion of the unresolved points. I
> > also see new issues.
>
> OK, here's a reversion path. It's a bit messy, and I didn't touch the
> release notes, but apart from that I think it does the right thing.
I checked "git diff 763aaa06f03^ src/bin/pg_dump/{*.c,*.h,t}" and I agree this
reverts everything. Thanks for working through the revert.
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Non-text mode for pg_dumpall
@ 2026-06-18 13:43 Andrew Dunstan <[email protected]>
parent: Noah Misch <[email protected]>
0 siblings, 0 replies; 22+ messages in thread
From: Andrew Dunstan @ 2026-06-18 13:43 UTC (permalink / raw)
To: Noah Misch <[email protected]>; +Cc: jian he <[email protected]>; Mahendra Singh Thalor <[email protected]>; tushar <[email protected]>; Vaibhav Dalvi <[email protected]>; [email protected]
On 2026-06-17 We 5:59 PM, Noah Misch wrote:
> On Mon, Jun 15, 2026 at 04:11:16PM -0400, Andrew Dunstan wrote:
>> On 2026-06-06 Sa 8:02 PM, Noah Misch wrote:
>>> On Thu, Feb 26, 2026 at 09:02:48AM -0500, Andrew Dunstan wrote:
>>>> pushed with a slight tweak.
>>> Having now reviewed commit 763aaa0, I don't think it's ready to remain part of
>>> v19. While some points from my v18 review are now resolved, other points
>>> still seem unresolved. I didn't find discussion of the unresolved points. I
>>> also see new issues.
>> OK, here's a reversion path. It's a bit messy, and I didn't touch the
>> release notes, but apart from that I think it does the right thing.
> I checked "git diff 763aaa06f03^ src/bin/pg_dump/{*.c,*.h,t}" and I agree this
> reverts everything. Thanks for working through the revert.
Thanks for checking. Pushed with the release notes update.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 22+ messages in thread
end of thread, other threads:[~2026-06-18 13:43 UTC | newest]
Thread overview: 22+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-01-28 07:34 Re: Non-text mode for pg_dumpall tushar <[email protected]>
2026-02-18 05:15 ` Mahendra Singh Thalor <[email protected]>
2026-02-19 20:00 ` Andrew Dunstan <[email protected]>
2026-02-20 14:10 ` Mahendra Singh Thalor <[email protected]>
2026-02-21 02:15 ` jian he <[email protected]>
2026-02-21 17:05 ` Andrew Dunstan <[email protected]>
2026-02-23 08:04 ` jian he <[email protected]>
2026-02-23 08:58 ` Mahendra Singh Thalor <[email protected]>
2026-02-23 20:34 ` Andrew Dunstan <[email protected]>
2026-02-24 07:21 ` jian he <[email protected]>
2026-02-24 10:40 ` Andrew Dunstan <[email protected]>
2026-02-26 14:02 ` Andrew Dunstan <[email protected]>
2026-02-26 19:11 ` Mahendra Singh Thalor <[email protected]>
2026-06-07 00:02 ` Noah Misch <[email protected]>
2026-06-15 20:11 ` Andrew Dunstan <[email protected]>
2026-06-17 21:59 ` Noah Misch <[email protected]>
2026-06-18 13:43 ` Andrew Dunstan <[email protected]>
2026-02-26 21:52 ` Tom Lane <[email protected]>
2026-02-27 12:12 ` Andrew Dunstan <[email protected]>
2026-03-03 11:17 ` Mahendra Singh Thalor <[email protected]>
2026-03-04 20:54 ` Andrew Dunstan <[email protected]>
2026-03-22 08:50 ` Andrew Dunstan <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox