public inbox for [email protected]
help / color / mirror / Atom feedRe: Non-text mode for pg_dumpall
29+ messages / 7 participants
[nested] [flat]
* Re: Non-text mode for pg_dumpall
@ 2025-03-05 01:12 Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Mahendra Singh Thalor @ 2025-03-05 01:12 UTC (permalink / raw)
To: Álvaro Herrera <[email protected]>; +Cc: jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Wed, 5 Mar 2025 at 01:02, Álvaro Herrera <[email protected]> wrote:
>
> A database name containing a newline breaks things for this patch:
>
> CREATE DATABASE "foo
> bar";
>
>
> $ pg_dumpall -Fc --file test
> shell command argument contains a newline or carriage return: "
dbname='foo
> bar'"
>
> --
> Álvaro Herrera 48°01'N 7°57'E —
https://www.EnterpriseDB.com/
Hi Alvaro,
I also reported this issue on 29-01-2025. This breaks even without this
patch also.
error with pg_dumpall when db name have new line in double quote
<https://www.postgresql.org/message-id/flat/CAFC%2Bb6qwc%2Bwpt7_b2R6YhpDkrXeFvFd5NoLbTMMoxX9tfOHjpg%4...;
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-03-05 15:12 ` Álvaro Herrera <[email protected]>
2025-03-05 16:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 2 replies; 29+ messages in thread
From: Álvaro Herrera @ 2025-03-05 15:12 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Disclaimer: I didn't review these patches fully.
On 2025-Mar-05, Mahendra Singh Thalor wrote:
> On Wed, 5 Mar 2025 at 01:02, Álvaro Herrera <[email protected]> wrote:
>
> > A database name containing a newline breaks things for this patch:
> >
> > CREATE DATABASE "foo
> > bar";
> I also reported this issue on 29-01-2025. This breaks even without this
> patch also.
Okay, we should probably fix that, but I think the new map.dat file your
patch adds is going to make the problem worse, because it doesn't look
like you handled that case in any particular way that would make it not
fail. I think it would be good to avoid digging us up even deeper in
that hole. More generally, the pg_upgrade tests contain some code to
try database names with almost all possible ascii characters (see
generate_db in pg_upgrade/t/002_pg_upgrade.pl); it would be good to
ensure that this new functionality also works correctly for that --
perhaps add an equivalent test to the pg_dumpall test suite.
Looking at 0001:
I'm not sure that the whole common_dumpall_restore.c thing is properly
structured. First, the file name shouldn't presume which programs
exactly are going to use the funcionality there. Second, it looks like
there's another PQconnectdbParams() in pg_backup_db.c and I don't
understand what the reason is for that one to be separate. In my mind,
there should be a file maybe called connection.c or connectdb.c or
whatever that's in charge of establishing connection for all the
src/bin/pg_dump programs, for cleanliness sake. (This is probably also
the place where to put an on_exit callback that cleans up any leftover
connections.)
Looking at 0002 I see it mentions looking at the EXIT_NICELY macro for
documentation. No such macro exists. But also I think the addition
(and use) of reset_exit_nicely_list() is not a good idea. It seems to
assume that the only entries in that list are ones that can be cleared
and reinstated whenever. This makes too much of an assumption about how
the program works. It may work today, but it'll get in the way of any
other patch that wants to set up some different on-exit clean up. In
other words, we shouldn't reset the on_exit list at all. Also, this is
just a weird addition:
#define exit_nicely(code) exit(code)
You added "A" as an option to the getopt_long() call in pg_restore, but
no handling for it is added.
I think the --globals-only option to pg_restore should be a separate
commit.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
@ 2025-03-05 16:48 ` Mahendra Singh Thalor <[email protected]>
2025-03-10 07:54 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
1 sibling, 1 reply; 29+ messages in thread
From: Mahendra Singh Thalor @ 2025-03-05 16:48 UTC (permalink / raw)
To: Álvaro Herrera <[email protected]>; +Cc: jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Thanks Alvaro for feedback and review.
On Wed, 5 Mar 2025 at 20:42, Álvaro Herrera <[email protected]> wrote:
>
> Disclaimer: I didn't review these patches fully.
>
> On 2025-Mar-05, Mahendra Singh Thalor wrote:
>
> > On Wed, 5 Mar 2025 at 01:02, Álvaro Herrera <[email protected]> wrote:
> >
> > > A database name containing a newline breaks things for this patch:
> > >
> > > CREATE DATABASE "foo
> > > bar";
>
> > I also reported this issue on 29-01-2025. This breaks even without this
> > patch also.
>
> Okay, we should probably fix that, but I think the new map.dat file your
> patch adds is going to make the problem worse, because it doesn't look
> like you handled that case in any particular way that would make it not
> fail. I think it would be good to avoid digging us up even deeper in
> that hole. More generally, the pg_upgrade tests contain some code to
> try database names with almost all possible ascii characters (see
> generate_db in pg_upgrade/t/002_pg_upgrade.pl); it would be good to
> ensure that this new functionality also works correctly for that --
> perhaps add an equivalent test to the pg_dumpall test suite.
In the attached patch, I tried to solve the problem of the map.dat
file. I will do more analysis based on dbnames in 002_pg_upgrade.pl
file.
>
> Looking at 0001:
>
> I'm not sure that the whole common_dumpall_restore.c thing is properly
> structured. First, the file name shouldn't presume which programs
> exactly are going to use the funcionality there. Second, it looks like
> there's another PQconnectdbParams() in pg_backup_db.c and I don't
> understand what the reason is for that one to be separate. In my mind,
> there should be a file maybe called connection.c or connectdb.c or
> whatever that's in charge of establishing connection for all the
> src/bin/pg_dump programs, for cleanliness sake. (This is probably also
> the place where to put an on_exit callback that cleans up any leftover
> connections.)
Okay. I will do these changes.
>
> Looking at 0002 I see it mentions looking at the EXIT_NICELY macro for
> documentation. No such macro exists. But also I think the addition
> (and use) of reset_exit_nicely_list() is not a good idea. It seems to
> assume that the only entries in that list are ones that can be cleared
> and reinstated whenever. This makes too much of an assumption about how
> the program works. It may work today, but it'll get in the way of any
> other patch that wants to set up some different on-exit clean up. In
> other words, we shouldn't reset the on_exit list at all. Also, this is
> just a weird addition:
I will do more study for this case and will update here.
>
> #define exit_nicely(code) exit(code)
Okay. I will fix this.
>
> You added "A" as an option to the getopt_long() call in pg_restore, but
> no handling for it is added.
Fixed.
>
> I think the --globals-only option to pg_restore should be a separate
> commit.
Okay.
>
> --
> Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
Here, I am attaching updated patches for review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v21_0002_pg_dumpall-with-non-text_format-5th_mar.patch (61.2K, 2-v21_0002_pg_dumpall-with-non-text_format-5th_mar.patch)
download | inline diff:
From 950daf288c13b889c67118770373f15fced603c7 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Wed, 5 Mar 2025 22:10:22 +0530
Subject: [PATCH 2/2] pg_dumpall with directory|tar|custom format and restore
it by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat to restore all databases. If global.dat file is exist in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list (if exist)
for --exclude-database=PATTERN for pg_restore.
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
if no db connection, then PATTERN=NAME matching only
for each database, we are cleaning on_exit_nicely_index list.
at the end of restore, we are giving warning with total number of errors (including global.dat,
and each database errors) and for each database, we are printing warning with dbname and total
errors.
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
---
---
doc/src/sgml/ref/pg_dumpall.sgml | 80 ++-
doc/src/sgml/ref/pg_restore.sgml | 41 +-
src/bin/pg_dump/Makefile | 4 +-
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 20 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 23 +-
src/bin/pg_dump/pg_backup_utils.h | 1 +
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 290 +++++++--
src/bin/pg_dump/pg_restore.c | 850 ++++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 9 +
src/tools/pgindent/typedefs.list | 2 +
14 files changed, 1249 insertions(+), 78 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index c2fa5be9519..c36802e06fd 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster based on specified dump format </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -121,7 +121,83 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- </para>
+ Note: This option can be omitted only when <option>--format</option> is plain
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 199ea3345f3..46bdbc092c3 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> database from an
+ archive file created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -37,9 +38,10 @@ PostgreSQL documentation
<title>Description</title>
<para>
- <application>pg_restore</application> is a utility for restoring a
+ <application>pg_restore</application> is a utility for restoring
<productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
database to the state it was in at the time it was saved. The
archive files also allow <application>pg_restore</application> to
@@ -140,6 +142,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from dump of <application>pg_dumpall</application>.
</para>
<para>
@@ -166,6 +170,25 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +338,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 86006d111c3..a4e557d62c7 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -47,8 +47,8 @@ all: pg_dump pg_restore pg_dumpall
pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dump.o common.o pg_dump_sort.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_restore: pg_restore.o common_dumpall_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_restore.o common_dumpall_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
pg_dumpall: pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 97dbfaeb67f..ddecac5cf09 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -69,6 +69,7 @@ bin_targets += pg_dumpall
pg_restore_sources = files(
'pg_restore.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index e783cc68d89..c7ced7add1b 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -309,7 +309,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 632077113a4..fb0711527bf 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -85,7 +85,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -336,9 +336,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -455,7 +460,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1291,7 +1296,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1670,7 +1675,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1690,7 +1696,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..0f6ad8b21d8 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -91,11 +91,7 @@ on_exit_nicely(on_exit_nicely_callback function, void *arg)
void
exit_nicely(int code)
{
- int i;
-
- for (i = on_exit_nicely_index - 1; i >= 0; i--)
- on_exit_nicely_list[i].function(code,
- on_exit_nicely_list[i].arg);
+ reset_exit_nicely_list(code);
#ifdef WIN32
if (parallel_init_done && GetCurrentThreadId() != mainThreadId)
@@ -104,3 +100,20 @@ exit_nicely(int code)
exit(code);
}
+
+/*
+ * reset_exit_nicely_list
+ *
+ * cleans index of exit_nicely list.
+ */
+void
+reset_exit_nicely_list(int code)
+{
+ int i;
+
+ for (i = on_exit_nicely_index - 1; i >= 0; i--)
+ on_exit_nicely_list[i].function(code,
+ on_exit_nicely_list[i].arg);
+
+ on_exit_nicely_index = 0;
+}
diff --git a/src/bin/pg_dump/pg_backup_utils.h b/src/bin/pg_dump/pg_backup_utils.h
index 38551944513..c2249a88185 100644
--- a/src/bin/pg_dump/pg_backup_utils.h
+++ b/src/bin/pg_dump/pg_backup_utils.h
@@ -30,6 +30,7 @@ extern const char *progname;
extern void set_dump_section(const char *arg, int *dumpSections);
extern void on_exit_nicely(on_exit_nicely_callback function, void *arg);
extern void exit_nicely(int code) pg_attribute_noreturn();
+extern void reset_exit_nicely_list(int code);
/* In pg_dump, we modify pg_fatal to call exit_nicely instead of exit */
#undef pg_fatal
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 4f4ad2ee150..b317d6d7122 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1189,7 +1189,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 1a0c1bbeae3..cd6234209d6 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -25,14 +26,16 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
-#include "pg_backup.h"
+#include "pg_backup_archiver.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
+#define exit_nicely(code) exit(code)
typedef struct
{
@@ -65,9 +68,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -76,6 +80,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
@@ -105,7 +111,7 @@ static int no_subscriptions = 0;
static int no_toast_compression = 0;
static int no_unlogged_table_data = 0;
static int no_role_passwords = 0;
-static int server_version;
+static int server_version;
static int load_via_partition_root = 0;
static int on_conflict_do_nothing = 0;
static int statistics_only = 0;
@@ -120,8 +126,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -146,6 +150,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -191,6 +196,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -240,7 +247,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -268,7 +275,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -417,6 +426,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If non-plain format is specified then we must provide the
+ * file name to create one main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -471,6 +495,33 @@ main(int argc, char *argv[])
if (statistics_only)
appendPQExpBufferStr(pgdumpopts, " --statistics-only");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char toc_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -510,19 +561,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -622,7 +660,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -635,7 +673,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -646,12 +684,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster based on specified dump format.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1554,10 +1594,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1571,7 +1614,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1579,9 +1622,33 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_fatal("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1596,6 +1663,22 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is non-plain dump format, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /*
+ * Put two line entry for dboid and dbname in map file. First line
+ * will have dboid and number of characters in dbname and second
+ * line will have actual dbname.
+ */
+ fprintf(map_file, "%s %u\n%s\n", oid, strlen(dbname), pg_strdup(dbname));
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1614,9 +1697,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1625,19 +1716,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char toc_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(toc_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ toc_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1647,7 +1749,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1656,17 +1759,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is non-plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1811,3 +1933,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if(!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 13e4dc507e0..de0f31cf693 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,30 +41,76 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/connect.h"
+#include "compress_io.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
+#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
+typedef struct SimpleDatabaseOidListCell
+{
+ struct SimpleDatabaseOidListCell *next;
+ Oid db_oid;
+ const char *db_name;
+} SimpleDatabaseOidListCell;
+
+typedef struct SimpleDatabaseOidList
+{
+ SimpleDatabaseOidListCell *head;
+ SimpleDatabaseOidListCell *tail;
+} SimpleDatabaseOidList;
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data);
+static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static void copy_or_print_global_file(const char *outfile, FILE *pfile);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleDatabaseOidList *dbname_oid_list);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname);
+static void simple_string_full_list_delete(SimpleStringList *list);
+static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
+static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list,
+ Oid db_oid, const char *dbname);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -86,6 +132,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -136,6 +183,7 @@ main(int argc, char **argv)
{"no-statistics", no_argument, &no_statistics, 1},
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -164,7 +212,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -191,11 +239,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -310,6 +361,10 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6:
+ /* list of databases patterns those needs to skip while restoring */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -337,6 +392,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -417,6 +479,106 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL && !IsFileExistsInDirectory(inputFileSpec, "toc.dat") &&
+ IsFileExistsInDirectory(inputFileSpec, "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql commands. */
+
+ /*
+ * User is suggested to use single database dump for --list option.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring multiple databases by archive of pg_dumpall");
+
+ /*
+ * To restore multiple databases, -C (create database) option should be specified.
+ * Even there is single database in dump, report error because it might be possible
+ * that database hasn't created so better we report error.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("-C/--create option should be specified when restoring multiple databases by archive of pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("If db is already created and dump has single db dump, then use particular dump file.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Connect to database to execute global sql commands from global.dat file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = connectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ /* Open global.dat file and execute/append all the global sql commands. */
+ n_errors = process_global_sql_commands(conn, inputFileSpec,
+ opts->filename);
+
+ if (conn)
+ PQfinish(conn);
+
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ n_errors = restoreAllDatabases(conn, inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_full_list_delete(&db_exclude_patterns);
+ }
+ else /* process if global.dat file does not exist. */
+ {
+ if (db_exclude_patterns.head != NULL)
+ pg_fatal("option --exclude-database can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ n_errors = restoreOneDatabase(inputFileSpec, opts, numWorkers, false);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -446,25 +608,22 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n"
+ "If archive is created by pg_dumpall, then restores multiple databases also. \n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -482,6 +641,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -494,6 +654,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -530,8 +691,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be combined\n"
+ "and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -636,3 +797,662 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * IsFileExistsInDirectory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+IsFileExistsInDirectory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.dat file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+ReadOneStatement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+ int m;
+
+ StringInfoData q;
+ initStringInfo(&q);
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != '\n' && c != ';')
+ {
+ appendStringInfoChar(inBuf, (char) c);
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != ';' && c != '\n')
+ appendStringInfoChar(inBuf, (char) c);
+ else
+ break;
+ }
+ }
+
+ if (c == '\'' || c == '"')
+ {
+ appendStringInfoChar(&q, (char) c);
+ m = c;
+
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(&q, (char) c);
+
+ if(c == m)
+ {
+ appendStringInfoString(inBuf, q.data);
+ resetStringInfo(&q);
+ break;
+ }
+ }
+ }
+
+ if (c == ';')
+ {
+ appendStringInfoChar(inBuf, (char) ';');
+ break;
+ }
+
+ if (c == '\n')
+ appendStringInfoChar(inBuf, (char) '\n');
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will remove entries from dbname_oid_list that pattern matching any
+ * in the db_exclude_patterns list. dbname_oid_list maybe inplace modified.
+ *
+ * returns, number of database will be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
+ SimpleDatabaseOidListCell *dboidprecell = NULL;
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ /* Return 0 if there is no database to restore. */
+ if (dboid_cell == NULL)
+ return 0;
+
+ query = createPQExpBuffer();
+
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ while (dboid_cell != NULL)
+ {
+ bool skip_db_restore = false;
+ SimpleDatabaseOidListCell *next = dboid_cell->next;
+
+ for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ {
+ /*
+ * the construct pattern matching query:
+ * SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE
+ * pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from the
+ * dbname_oid_list, which is initially extracted from the map.dat
+ * file located in the backup directory. that's why we need
+ * quote_literal_cstr.
+ *
+ * If no db connection, then consider PATTERN as NAME.
+ */
+ if (pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0)
+ skip_db_restore = true;
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, celldb->val, false,
+ false, NULL, quote_literal_cstr(dboid_cell->db_name),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ celldb->val);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database \"%s\" is matching with exclude pattern: \"%s\"", dboid_cell->db_name, celldb->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /* Increment count if database needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
+ simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ }
+ else
+ {
+ count_db++;
+ dboidprecell = dboid_cell;
+ }
+
+ /* Process next dbname from dbname list. */
+ dboid_cell = next;
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ /*
+ * If there is only global.dat file in dump, then return from here as there
+ * is no database to restore.
+ */
+ if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ {
+ pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid = InvalidOid;
+ size_t db_name_size = 0;
+ char dbname[MAXPGPATH + 1] = {'\0'};
+ int i = 0;
+ char ch;
+ int kk = 1;
+ while(kk);
+
+ /* Extract dboid and size of dbname. */
+ sscanf(line, "%u %u" , &db_oid, &db_name_size);
+
+ /* Now copy dbname. */
+ while (i < db_name_size)
+ dbname[i++] = fgetc(pfile);
+
+ ch = fgetc(pfile);
+
+ if (ch != '\n')
+ pg_fatal("invalid entry in map.dat file at line : %d", 2 * count + 2);
+
+ /* Add \0 in the end. */
+ dbname[i] = '\0';
+
+ pg_log_info("found database \"%s\" (OID: %u) in map.dat file while restoring.", dbname, db_oid);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", 2 * count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
+ SimpleDatabaseOidListCell *dboid_cell;
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /*
+ * If map.dat has no entry, return from here after processing
+ * global.dat file.
+ */
+ if (dbname_oid_list.head == NULL)
+ return process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = connectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL);
+ }
+ }
+
+ /*
+ * processing pg_retsore --exclude-database=PATTERN/NAME if no connection.
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ n_errors_total = process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL)
+ {
+ pg_log_info("no database needs to restore out of %d databases", num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * Till now, we made a list of databases, those needs to be restored
+ * after skipping names of exclude-database. Now we can launch parallel
+ * workers to restore these databases.
+ */
+ dboid_cell = dbname_oid_list.head;
+
+ while(dboid_cell != NULL)
+ {
+ char subdirpath[MAXPGPATH];
+ int n_errors;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored into
+ * already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+
+ pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+
+ /* Restore single database. */
+ n_errors = restoreOneDatabase(subdirpath, opts, numWorkers, true);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dboid_cell->db_name, n_errors);
+ }
+
+ dboid_cell = dboid_cell->next;
+
+ /*
+ * We need to reset on_exit_nicely_index with each database so that we can restore
+ * multiple databases by archive.
+ */
+ if (dboid_cell != NULL)
+ reset_exit_nicely_list(n_errors ? 1 : 0);
+ }
+
+ /* Log number of processed databases.*/
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+
+ return n_errors_total;
+}
+
+/*
+ * process_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ *
+ * returns the number of errors while processing global.dat
+ */
+static int
+process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+ int n_errors = 0;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ copy_or_print_global_file(outfile, pfile);
+ return 0;
+ }
+
+ /* Init sqlstatement to append commands. */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements. */
+ while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ {
+ pg_log_info("executing query: %s", sqlstatement.data);
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ n_errors++;
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ /* Print a summary of ignored errors during global.dat. */
+ if (n_errors)
+ pg_log_warning("errors ignored on global.dat file restore: %d", n_errors);
+
+ fclose(pfile);
+
+ return n_errors;
+}
+
+/*
+ * copy_or_print_global_file
+ *
+ * This will copy global.dat file into out file. If "-" is used as outfile,
+ * then print commands to the stdout.
+ */
+static void
+copy_or_print_global_file(const char *outfile, FILE *pfile)
+{
+ char out_file_path[MAXPGPATH];
+ FILE *OPF;
+ int c;
+
+ /* "-" is used for stdout. */
+ if (strcmp(outfile, "-") == 0)
+ OPF = stdout;
+ else
+ {
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ OPF = fopen(out_file_path, PG_BINARY_W);
+
+ if (OPF == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", outfile);
+ }
+ }
+
+ /* Append global.dat into out file or print to the stdout. */
+ while ((c = fgetc(pfile)) != EOF)
+ fputc(c, OPF);
+
+ fclose(pfile);
+
+ /* Close out file. */
+ if (strcmp(outfile, "-") != 0)
+ fclose(OPF);
+}
+
+/*
+ * simple_db_oid_list_append
+ *
+ * appends a node to the list in the end.
+ */
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname)
+{
+ SimpleDatabaseOidListCell *cell;
+
+ cell = pg_malloc_object(SimpleDatabaseOidListCell);
+
+ cell->next = NULL;
+ cell->db_oid = db_oid;
+ cell->db_name = pg_strdup(dbname);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * simple_db_oid_full_list_delete
+ *
+ * delete all cell from dbname and dboid list.
+ */
+static void
+simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
+{
+ SimpleDatabaseOidListCell *cell = list->head;
+ SimpleDatabaseOidListCell *nextcell = NULL;
+
+ while (cell)
+ {
+ nextcell = cell->next;
+ pfree (cell);
+ cell = nextcell;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_string_full_list_delete
+ *
+ * delete all cell from string list.
+ */
+static void
+simple_string_full_list_delete(SimpleStringList *list)
+{
+ SimpleStringListCell *cell = list->head;
+ SimpleStringListCell *cellnext = NULL;
+
+ while (cell)
+ {
+ cellnext = cell->next;
+ pfree(cell);
+ cell = cellnext;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_db_oid_list_delete
+ *
+ * delete cell from database and oid list.
+ */
+static void
+simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev)
+{
+ if (prev == NULL)
+ {
+ list->head = cell->next;
+ pfree(cell);
+ }
+ else
+ {
+ prev->next = cell->next;
+ pfree(cell);
+ }
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr
+ *
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..0bbcdbe84a7
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,11 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +249,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9840060997f..b43a3e48e3b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2695,6 +2695,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.39.3
[application/octet-stream] v21_0001_move-common-code-of-pg_dumpall-and-pg_restore-to-new_file.patch (19.5K, 3-v21_0001_move-common-code-of-pg_dumpall-and-pg_restore-to-new_file.patch)
download | inline diff:
From 77dad68de0e59ca7489da6e695f5436759ba17ef Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Thu, 20 Feb 2025 16:27:17 +0530
Subject: [PATCH 1/2] move common code of pg_dumpall and pg_restore to new file
connectDatabase is used by both pg_dumpall and pg_restore so
move common code to new file.
---
src/bin/pg_dump/Makefile | 4 +-
src/bin/pg_dump/common_dumpall_restore.c | 289 +++++++++++++++++++++++
src/bin/pg_dump/common_dumpall_restore.h | 24 ++
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/pg_dumpall.c | 268 +--------------------
5 files changed, 324 insertions(+), 262 deletions(-)
create mode 100644 src/bin/pg_dump/common_dumpall_restore.c
create mode 100644 src/bin/pg_dump/common_dumpall_restore.h
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..86006d111c3 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -50,8 +50,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o common_dumpall_restore.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/common_dumpall_restore.c b/src/bin/pg_dump/common_dumpall_restore.c
new file mode 100644
index 00000000000..92f52b7239a
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.c
@@ -0,0 +1,289 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.c
+ * This is a common file for pg_dumpall and pg_restore.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/common_dumpall_restore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "common_dumpall_restore.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+#define exit_nicely(code) exit(code)
+
+/*
+ * connectDatabase
+ *
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the 'connstr' is set to a connection string containing
+ * the options used and 'server_version' is set to version so that caller
+ * can use them.
+ */
+PGconn *
+connectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ static char *password = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 6;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. If requested, remember the options used, in the
+ * form of a connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If requested, then copy server version to out variable. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/*
+ * constructConnStr
+ *
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * executeQuery
+ *
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
diff --git a/src/bin/pg_dump/common_dumpall_restore.h b/src/bin/pg_dump/common_dumpall_restore.h
new file mode 100644
index 00000000000..7fe1c00ab71
--- /dev/null
+++ b/src/bin/pg_dump/common_dumpall_restore.h
@@ -0,0 +1,24 @@
+/*-------------------------------------------------------------------------
+ *
+ * common_dumpall_restore.h
+ * Common header file for pg_dumpall and pg_restore
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/common_dumpall_restore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef COMMON_DUMPALL_RESTORE_H
+#define COMMON_DUMPALL_RESTORE_H
+
+#include "pg_backup.h"
+
+extern PGconn *connectDatabase(const char *dbname, const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+#endif /* COMMON_DUMPALL_RESTORE_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..97dbfaeb67f 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -49,6 +49,7 @@ bin_targets += pg_dump
pg_dumpall_sources = files(
'pg_dumpall.c',
+ 'common_dumpall_restore.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index e0867242526..1a0c1bbeae3 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -20,6 +20,7 @@
#include "catalog/pg_authid_d.h"
#include "common/connect.h"
+#include "common_dumpall_restore.h"
#include "common/file_utils.h"
#include "common/hashfn_unstable.h"
#include "common/logging.h"
@@ -71,12 +72,6 @@ static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
@@ -85,7 +80,7 @@ static void read_dumpall_filters(const char *filename, SimpleStringList *pattern
static char pg_dump_bin[MAXPGPATH];
static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -484,7 +479,8 @@ main(int argc, char *argv[])
if (pgdb)
{
conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
@@ -492,10 +488,12 @@ main(int argc, char *argv[])
else
{
conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ prompt_password, false,
+ progname, &connstr, &server_version);
if (!conn)
conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ prompt_password, true,
+ progname, &connstr, &server_version);
if (!conn)
{
@@ -1718,256 +1716,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
--
2.39.3
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-05 16:48 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-03-10 07:54 ` jian he <[email protected]>
0 siblings, 0 replies; 29+ messages in thread
From: jian he @ 2025-03-10 07:54 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: Álvaro Herrera <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Thu, Mar 6, 2025 at 12:49 AM Mahendra Singh Thalor
<[email protected]> wrote:
>
> Thanks Alvaro for feedback and review.
>
> On Wed, 5 Mar 2025 at 20:42, Álvaro Herrera <[email protected]> wrote:
> >
> > Disclaimer: I didn't review these patches fully.
> >
> > On 2025-Mar-05, Mahendra Singh Thalor wrote:
> >
> > > On Wed, 5 Mar 2025 at 01:02, Álvaro Herrera <[email protected]> wrote:
> > >
> > > > A database name containing a newline breaks things for this patch:
> > > >
> > > > CREATE DATABASE "foo
> > > > bar";
> >
> > > I also reported this issue on 29-01-2025. This breaks even without this
> > > patch also.
> >
> > Okay, we should probably fix that, but I think the new map.dat file your
> > patch adds is going to make the problem worse, because it doesn't look
> > like you handled that case in any particular way that would make it not
> > fail. I think it would be good to avoid digging us up even deeper in
> > that hole. More generally, the pg_upgrade tests contain some code to
> > try database names with almost all possible ascii characters (see
> > generate_db in pg_upgrade/t/002_pg_upgrade.pl); it would be good to
> > ensure that this new functionality also works correctly for that --
> > perhaps add an equivalent test to the pg_dumpall test suite.
>
> In the attached patch, I tried to solve the problem of the map.dat
> file. I will do more analysis based on dbnames in 002_pg_upgrade.pl
> file.
>
hi.
/*
* Append the given string to the shell command being built in the buffer,
* with shell-style quoting as needed to create exactly one argument.
*
* Forbid LF or CR characters, which have scant practical use beyond designing
* security breaches. The Windows command shell is unusable as a conduit for
* arguments containing LF or CR characters. A future major release should
* reject those characters in CREATE ROLE and CREATE DATABASE, because use
* there eventually leads to errors here.
*
* appendShellString() simply prints an error and dies if LF or CR appears.
* appendShellStringNoError() omits those characters from the result, and
* returns false if there were any.
*/
void
appendShellString(PQExpBuffer buf, const char *str)
per above comments,
we need to disallow LF/CR in database name and role name when issuing
shell command.
rolename LF/CR issue already being handled in
src/bin/pg_dump/pg_dumpall.c: while(getopt_long) code:
case 3:
use_role = pg_strdup(optarg);
appendPQExpBufferStr(pgdumpopts, " --role ");
appendShellString(pgdumpopts, use_role);
we can fail earlier also for database names in dumpDatabases, right
after executeQuery.
Please check attached, which is based on *v20*.
in V21, src/bin/pg_dump/pg_dumpall.c:
+#include "common_dumpall_restore.h"
happened within v21-0001 and v21-0002, it is being included twice.
Attachments:
[application/octet-stream] v20-0001-pg_dumpall-deal-witth-newline-or-carriage-ret.no-cfbot (3.3K, 2-v20-0001-pg_dumpall-deal-witth-newline-or-carriage-ret.no-cfbot)
download
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
@ 2025-03-11 14:11 ` Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
1 sibling, 1 reply; 29+ messages in thread
From: Mahendra Singh Thalor @ 2025-03-11 14:11 UTC (permalink / raw)
To: Álvaro Herrera <[email protected]>; +Cc: jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Thanks Alvaro and Jian for the review and feedback.
On Wed, 5 Mar 2025 at 20:42, Álvaro Herrera <[email protected]> wrote:
>
> Disclaimer: I didn't review these patches fully.
>
> On 2025-Mar-05, Mahendra Singh Thalor wrote:
>
> > On Wed, 5 Mar 2025 at 01:02, Álvaro Herrera <[email protected]> wrote:
> >
> > > A database name containing a newline breaks things for this patch:
> > >
> > > CREATE DATABASE "foo
> > > bar";
>
> > I also reported this issue on 29-01-2025. This breaks even without this
> > patch also.
>
> Okay, we should probably fix that, but I think the new map.dat file your
> patch adds is going to make the problem worse, because it doesn't look
> like you handled that case in any particular way that would make it not
> fail. I think it would be good to avoid digging us up even deeper in
> that hole. More generally, the pg_upgrade tests contain some code to
> try database names with almost all possible ascii characters (see
> generate_db in pg_upgrade/t/002_pg_upgrade.pl); it would be good to
> ensure that this new functionality also works correctly for that --
> perhaps add an equivalent test to the pg_dumpall test suite.
As Jian also pointed out, we should not allow \n\r in dbnames. I am
keeping dbanames as single line names only.
I am doing testing using the pg_upgrade/t/002_pg_upgrade.pl file to
check different-2 dbnames.
>
> Looking at 0001:
>
> I'm not sure that the whole common_dumpall_restore.c thing is properly
> structured. First, the file name shouldn't presume which programs
> exactly are going to use the funcionality there. Second, it looks like
> there's another PQconnectdbParams() in pg_backup_db.c and I don't
> understand what the reason is for that one to be separate. In my mind,
> there should be a file maybe called connection.c or connectdb.c or
> whatever that's in charge of establishing connection for all the
> src/bin/pg_dump programs, for cleanliness sake. (This is probably also
> the place where to put an on_exit callback that cleans up any leftover
> connections.)
I did some more refactoring and made a connectdb.c file.
> Looking at 0002 I see it mentions looking at the EXIT_NICELY macro for
> documentation. No such macro exists. But also I think the addition
> (and use) of reset_exit_nicely_list() is not a good idea. It seems to
> assume that the only entries in that list are ones that can be cleared
> and reinstated whenever. This makes too much of an assumption about how
> the program works. It may work today, but it'll get in the way of any
> other patch that wants to set up some different on-exit clean up. In
> other words, we shouldn't reset the on_exit list at all. Also, this is
> just a weird addition:
Based on some discussions, I added handling for cleanup. for 1st
database, I am saving index of array and then I am using same index
for rest of the databases as we are closing archive file in
CloseArchive so we can use same index for next database.
>
> #define exit_nicely(code) exit(code)
Fixed.
>
> You added "A" as an option to the getopt_long() call in pg_restore, but
> no handling for it is added.
Fixed.
>
> I think the --globals-only option to pg_restore should be a separate
> commit.
I will make this in the next version.
Here, I am attaching updated patches for review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v22_0001_move-common-code-of-pg_dumpall-and-pg_restore-to-new_file.patch (17.7K, 2-v22_0001_move-common-code-of-pg_dumpall-and-pg_restore-to-new_file.patch)
download | inline diff:
From 63efb00ca40d87e853da6266d536563b0caef7f6 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Tue, 11 Mar 2025 18:40:00 +0530
Subject: [PATCH 1/2] move common code related to connection to new the file
ConnectDatabase is used by both pg_dumpall, pg_restore
and pg_dump so move common code to new file.
new file name: connectdb.c
---
src/bin/pg_dump/Makefile | 9 +-
src/bin/pg_dump/meson.build | 2 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 6 +-
src/bin/pg_dump/pg_backup_db.c | 75 +------
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 279 ++-------------------------
7 files changed, 32 insertions(+), 343 deletions(-)
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..c488ab4aecf 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -40,7 +40,8 @@ OBJS = \
pg_backup_directory.o \
pg_backup_null.o \
pg_backup_tar.o \
- pg_backup_utils.o
+ pg_backup_utils.o \
+ connectdb.o
all: pg_dump pg_restore pg_dumpall
@@ -50,8 +51,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o connectdb.o pg_backup_utils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o connectdb.o pg_backup_utils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
@@ -71,5 +72,5 @@ uninstall:
rm -f $(addprefix '$(DESTDIR)$(bindir)'/, pg_dump$(X) pg_restore$(X) pg_dumpall$(X))
clean distclean:
- rm -f pg_dump$(X) pg_restore$(X) pg_dumpall$(X) $(OBJS) pg_dump.o common.o pg_dump_sort.o pg_restore.o pg_dumpall.o
+ rm -f pg_dump$(X) pg_restore$(X) pg_dumpall$(X) $(OBJS) pg_dump.o common.o pg_dump_sort.o pg_restore.o pg_dumpall.o connectdb.o
rm -rf tmp_check
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..d5f805fb511 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -30,6 +30,7 @@ pg_dump_sources = files(
'common.c',
'pg_dump.c',
'pg_dump_sort.c',
+ 'connectdb.c',
)
if host_system == 'windows'
@@ -49,6 +50,7 @@ bin_targets += pg_dump
pg_dumpall_sources = files(
'pg_dumpall.c',
+ 'connectdb.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index e783cc68d89..731cb2d19fb 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -291,7 +291,7 @@ typedef void (*SetupWorkerPtrType) (Archive *AH);
* Main archiver interface.
*/
-extern void ConnectDatabase(Archive *AHX,
+extern void ConnectDatabaseAhx(Archive *AHX,
const ConnParams *cparams,
bool isReconnect);
extern void DisconnectDatabase(Archive *AHX);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 7480e122b61..12f3f39e39b 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -413,7 +413,7 @@ RestoreArchive(Archive *AHX)
AHX->minRemoteVersion = 0;
AHX->maxRemoteVersion = 9999999;
- ConnectDatabase(AHX, &ropt->cparams, false);
+ ConnectDatabaseAhx(AHX, &ropt->cparams, false);
/*
* If we're talking to the DB directly, don't send comments since they
@@ -4430,7 +4430,7 @@ restore_toc_entries_postfork(ArchiveHandle *AH, TocEntry *pending_list)
/*
* Now reconnect the single parent connection.
*/
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
/* re-establish fixed state */
_doSetFixedOutputState(AH);
@@ -5047,7 +5047,7 @@ CloneArchive(ArchiveHandle *AH)
* Connect our new clone object to the database, using the same connection
* parameters used for the original connection.
*/
- ConnectDatabase((Archive *) clone, &clone->public.ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) clone, &clone->public.ropt->cparams, true);
/* re-establish fixed state */
if (AH->mode == archModeRead)
diff --git a/src/bin/pg_dump/pg_backup_db.c b/src/bin/pg_dump/pg_backup_db.c
index 71c55d2466a..227dd963984 100644
--- a/src/bin/pg_dump/pg_backup_db.c
+++ b/src/bin/pg_dump/pg_backup_db.c
@@ -19,6 +19,7 @@
#include "common/connect.h"
#include "common/string.h"
+#include "connectdb.h"
#include "parallel.h"
#include "pg_backup_archiver.h"
#include "pg_backup_db.h"
@@ -86,9 +87,9 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* ArchiveHandle's connCancel, before closing old connection. Otherwise
* an ill-timed SIGINT could try to access a dead connection.
*/
- AH->connection = NULL; /* dodge error check in ConnectDatabase */
+ AH->connection = NULL; /* dodge error check in ConnectDatabaseAhx */
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
PQfinish(oldConn);
}
@@ -105,14 +106,13 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* username never does change, so one savedPassword is sufficient.
*/
void
-ConnectDatabase(Archive *AHX,
+ConnectDatabaseAhx(Archive *AHX,
const ConnParams *cparams,
bool isReconnect)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
trivalue prompt_password;
char *password;
- bool new_pass;
if (AH->connection)
pg_fatal("already connected to a database");
@@ -125,69 +125,10 @@ ConnectDatabase(Archive *AHX,
if (prompt_password == TRI_YES && password == NULL)
password = simple_prompt("Password: ", false);
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- const char *keywords[8];
- const char *values[8];
- int i = 0;
-
- /*
- * If dbname is a connstring, its entries can override the other
- * values obtained from cparams; but in turn, override_dbname can
- * override the dbname component of it.
- */
- keywords[i] = "host";
- values[i++] = cparams->pghost;
- keywords[i] = "port";
- values[i++] = cparams->pgport;
- keywords[i] = "user";
- values[i++] = cparams->username;
- keywords[i] = "password";
- values[i++] = password;
- keywords[i] = "dbname";
- values[i++] = cparams->dbname;
- if (cparams->override_dbname)
- {
- keywords[i] = "dbname";
- values[i++] = cparams->override_dbname;
- }
- keywords[i] = "fallback_application_name";
- values[i++] = progname;
- keywords[i] = NULL;
- values[i++] = NULL;
- Assert(i <= lengthof(keywords));
-
- new_pass = false;
- AH->connection = PQconnectdbParams(keywords, values, true);
-
- if (!AH->connection)
- pg_fatal("could not connect to database");
-
- if (PQstatus(AH->connection) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(AH->connection) &&
- password == NULL &&
- prompt_password != TRI_NO)
- {
- PQfinish(AH->connection);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(AH->connection) == CONNECTION_BAD)
- {
- if (isReconnect)
- pg_fatal("reconnection failed: %s",
- PQerrorMessage(AH->connection));
- else
- pg_fatal("%s",
- PQerrorMessage(AH->connection));
- }
+ AH->connection = ConnectDatabase(cparams->dbname, NULL, cparams->pghost,
+ cparams->pgport, cparams->username,
+ prompt_password, true,
+ progname, NULL, NULL, password, cparams->override_dbname);
/* Start strict; later phases may override this. */
PQclear(ExecuteSqlQueryForSingleRow((Archive *) AH,
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index c371570501a..6bb54c1a2b4 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -934,7 +934,7 @@ main(int argc, char **argv)
* Open the database using the Archiver, so it knows about it. Errors mean
* death.
*/
- ConnectDatabase(fout, &dopt.cparams, false);
+ ConnectDatabaseAhx(fout, &dopt.cparams, false);
setup_connection(fout, dumpencoding, dumpsnapshot, use_role);
/*
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index e0867242526..e7e492afa28 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -24,11 +24,11 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
-#include "pg_backup.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -71,21 +71,15 @@ static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
static char pg_dump_bin[MAXPGPATH];
-static const char *progname;
+const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -125,8 +119,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -483,19 +475,22 @@ main(int argc, char *argv[])
*/
if (pgdb)
{
- conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase(pgdb, connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
}
else
{
- conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase("postgres", connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
- conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ conn = ConnectDatabase("template1", connstr, pghost, pgport, pguser,
+ prompt_password, true,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
{
@@ -1718,256 +1713,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
--
2.39.3
[application/octet-stream] v22_0002_pg_dumpall-with-non-text_format-11th_march.patch (61.7K, 3-v22_0002_pg_dumpall-with-non-text_format-11th_march.patch)
download | inline diff:
From 7ad1ff448ebc60bcc2035bc81a217e2eb01ce20b Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Tue, 11 Mar 2025 19:29:15 +0530
Subject: [PATCH 2/2] pg_dumpall with directory|tar|custom format and restore
it by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat to restore all databases. If global.dat file is exist in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list (if exist)
for --exclude-database=PATTERN for pg_restore
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
if no db connection, then PATTERN=NAME matching only
for each database, we are cleaning on_exit_nicely_index list.
at the end of restore, we are giving warning with total number of errors (including global.dat,
and each database errors) and for each database, we are printing warning with dbname and total
errors.
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
---
---
doc/src/sgml/ref/pg_dumpall.sgml | 80 ++-
doc/src/sgml/ref/pg_restore.sgml | 41 +-
src/bin/pg_dump/meson.build | 3 +-
src/bin/pg_dump/parallel.c | 11 +-
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 20 +-
src/bin/pg_dump/pg_backup_archiver.h | 3 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 22 +-
src/bin/pg_dump/pg_backup_utils.h | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 280 +++++++--
src/bin/pg_dump/pg_restore.c | 847 ++++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 9 +
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 1248 insertions(+), 79 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index c2fa5be9519..c36802e06fd 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster based on specified dump format </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -121,7 +121,83 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- </para>
+ Note: This option can be omitted only when <option>--format</option> is plain
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 199ea3345f3..46bdbc092c3 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> database from an
+ archive file created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -37,9 +38,10 @@ PostgreSQL documentation
<title>Description</title>
<para>
- <application>pg_restore</application> is a utility for restoring a
+ <application>pg_restore</application> is a utility for restoring
<productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
database to the state it was in at the time it was saved. The
archive files also allow <application>pg_restore</application> to
@@ -140,6 +142,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from dump of <application>pg_dumpall</application>.
</para>
<para>
@@ -166,6 +170,25 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +338,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index d5f805fb511..dc1ed410838 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -16,6 +16,7 @@ pg_dump_common_sources = files(
'pg_backup_null.c',
'pg_backup_tar.c',
'pg_backup_utils.c',
+ 'connectdb.c',
)
pg_dump_common = static_library('libpgdump_common',
@@ -30,7 +31,6 @@ pg_dump_sources = files(
'common.c',
'pg_dump.c',
'pg_dump_sort.c',
- 'connectdb.c',
)
if host_system == 'windows'
@@ -70,6 +70,7 @@ bin_targets += pg_dumpall
pg_restore_sources = files(
'pg_restore.c',
+ 'connectdb.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..a36d2a5bf84 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -326,11 +326,18 @@ getThreadLocalPQExpBuffer(void)
* pg_dump and pg_restore call this to register the cleanup handler
* as soon as they've created the ArchiveHandle.
*/
-void
+int
on_exit_close_archive(Archive *AHX)
{
shutdown_info.AHX = AHX;
- on_exit_nicely(archive_close_connection, &shutdown_info);
+ return on_exit_nicely(archive_close_connection, &shutdown_info);
+}
+
+void
+replace_on_exit_close_archive(Archive *AHX, int idx)
+{
+ shutdown_info.AHX = AHX;
+ set_on_exit_nicely_entry(archive_close_connection, &shutdown_info, idx);
}
/*
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 731cb2d19fb..41ee305850a 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -309,7 +309,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 12f3f39e39b..ad2a62a2c9c 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -85,7 +85,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -336,9 +336,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -455,7 +460,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1291,7 +1296,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1670,7 +1675,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1690,7 +1696,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index a2064f471ed..ae433132435 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -385,7 +385,8 @@ struct _tocEntry
};
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
-extern void on_exit_close_archive(Archive *AHX);
+extern int on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX, int idx);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..59ece2999a8 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -61,14 +61,26 @@ set_dump_section(const char *arg, int *dumpSections)
/* Register a callback to be run when exit_nicely is invoked. */
-void
+int
on_exit_nicely(on_exit_nicely_callback function, void *arg)
{
- if (on_exit_nicely_index >= MAX_ON_EXIT_NICELY)
- pg_fatal("out of on_exit_nicely slots");
- on_exit_nicely_list[on_exit_nicely_index].function = function;
- on_exit_nicely_list[on_exit_nicely_index].arg = arg;
+ set_on_exit_nicely_entry(function, arg, on_exit_nicely_index);
on_exit_nicely_index++;
+
+ return (on_exit_nicely_index - 1);
+}
+
+void
+set_on_exit_nicely_entry(on_exit_nicely_callback function, void *arg, int i)
+{
+ if (i >= MAX_ON_EXIT_NICELY)
+ pg_fatal("out of on_exit_nicely slots");
+
+ if (i > on_exit_nicely_index)
+ pg_fatal("no entry exists on %d index into on_exit_nicely slots", i);
+
+ on_exit_nicely_list[i].function = function;
+ on_exit_nicely_list[i].arg = arg;
}
/*
diff --git a/src/bin/pg_dump/pg_backup_utils.h b/src/bin/pg_dump/pg_backup_utils.h
index 38551944513..57f3197f103 100644
--- a/src/bin/pg_dump/pg_backup_utils.h
+++ b/src/bin/pg_dump/pg_backup_utils.h
@@ -28,7 +28,8 @@ typedef void (*on_exit_nicely_callback) (int code, void *arg);
extern const char *progname;
extern void set_dump_section(const char *arg, int *dumpSections);
-extern void on_exit_nicely(on_exit_nicely_callback function, void *arg);
+extern int on_exit_nicely(on_exit_nicely_callback function, void *arg);
+extern void set_on_exit_nicely_entry(on_exit_nicely_callback function, void *arg, int idx);
extern void exit_nicely(int code) pg_attribute_noreturn();
/* In pg_dump, we modify pg_fatal to call exit_nicely instead of exit */
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 6bb54c1a2b4..5b6d4364eb3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1186,7 +1186,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index e7e492afa28..ee7c62da09d 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -64,9 +65,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -75,6 +77,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
const char *progname;
@@ -104,7 +108,7 @@ static int no_subscriptions = 0;
static int no_toast_compression = 0;
static int no_unlogged_table_data = 0;
static int no_role_passwords = 0;
-static int server_version;
+static int server_version;
static int load_via_partition_root = 0;
static int on_conflict_do_nothing = 0;
static int statistics_only = 0;
@@ -143,6 +147,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -188,6 +193,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -237,7 +244,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -265,7 +272,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -414,6 +423,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If non-plain format is specified then we must provide the
+ * file name to create one main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -468,6 +492,33 @@ main(int argc, char *argv[])
if (statistics_only)
appendPQExpBufferStr(pgdumpopts, " --statistics-only");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char toc_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -507,19 +558,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -619,7 +657,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -632,7 +670,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -643,12 +681,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster based on specified dump format.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1551,10 +1591,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1568,7 +1611,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1576,9 +1619,33 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_fatal("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1593,6 +1660,18 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is non-plain dump format, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1611,9 +1690,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1622,19 +1709,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char toc_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(toc_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ toc_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1644,7 +1742,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1653,17 +1752,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is non-plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1808,3 +1926,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if(!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 13e4dc507e0..9b802e7a6bd 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,30 +41,77 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/connect.h"
+#include "compress_io.h"
+#include "common/string.h"
+#include "connectdb.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
+#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
+typedef struct SimpleDatabaseOidListCell
+{
+ struct SimpleDatabaseOidListCell *next;
+ Oid db_oid;
+ const char *db_name;
+} SimpleDatabaseOidListCell;
+
+typedef struct SimpleDatabaseOidList
+{
+ SimpleDatabaseOidListCell *head;
+ SimpleDatabaseOidListCell *tail;
+} SimpleDatabaseOidList;
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num);
+static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static void copy_or_print_global_file(const char *outfile, FILE *pfile);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleDatabaseOidList *dbname_oid_list);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname);
+static void simple_string_full_list_delete(SimpleStringList *list);
+static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
+static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list,
+ Oid db_oid, const char *dbname);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
+static int on_exit_index = 0;
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -86,6 +133,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -136,6 +184,7 @@ main(int argc, char **argv)
{"no-statistics", no_argument, &no_statistics, 1},
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -164,7 +213,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -191,11 +240,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -310,6 +362,10 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6:
+ /* list of databases patterns those needs to skip while restoring */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -337,6 +393,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -417,6 +480,108 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL && !IsFileExistsInDirectory(inputFileSpec, "toc.dat") &&
+ IsFileExistsInDirectory(inputFileSpec, "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql commands. */
+
+ /*
+ * User is suggested to use single database dump for --list option.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring multiple databases by archive of pg_dumpall");
+
+ /*
+ * To restore multiple databases, -C (create database) option should be specified.
+ * Even there is single database in dump, report error because it might be possible
+ * that database hasn't created so better we report error.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("-C/--create option should be specified when restoring multiple databases by archive of pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("If db is already created and dump has single db dump, then use particular dump file.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Connect to database to execute global sql commands from global.dat file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ /* Open global.dat file and execute/append all the global sql commands. */
+ n_errors = process_global_sql_commands(conn, inputFileSpec,
+ opts->filename);
+
+ if (conn)
+ PQfinish(conn);
+
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ n_errors = restoreAllDatabases(conn, inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_full_list_delete(&db_exclude_patterns);
+ }
+ else /* process if global.dat file does not exist. */
+ {
+ if (db_exclude_patterns.head != NULL)
+ pg_fatal("option --exclude-database can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ n_errors = restoreOneDatabase(inputFileSpec, opts, numWorkers, false, 0);
+ }
+
+ on_exit_index = 0; /* Reset index. */
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -425,8 +590,14 @@ main(int argc, char **argv)
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
* it's still NULL, the cleanup function will just be a no-op.
+ * If we are restoring multiple databases, then save index of exit_nicely
+ * so that we can use same slot for all the databases as we already closed
+ * the previous archive by CloseArchive.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_index = on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH, on_exit_index);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -446,25 +617,22 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n"
+ "If archive is created by pg_dumpall, then restores multiple databases also. \n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -482,6 +650,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -494,6 +663,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -530,8 +700,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be combined\n"
+ "and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -636,3 +806,648 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * IsFileExistsInDirectory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+IsFileExistsInDirectory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.dat file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+ReadOneStatement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+ int m;
+
+ StringInfoData q;
+ initStringInfo(&q);
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != '\n' && c != ';')
+ {
+ appendStringInfoChar(inBuf, (char) c);
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != ';' && c != '\n')
+ appendStringInfoChar(inBuf, (char) c);
+ else
+ break;
+ }
+ }
+
+ if (c == '\'' || c == '"')
+ {
+ appendStringInfoChar(&q, (char) c);
+ m = c;
+
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(&q, (char) c);
+
+ if(c == m)
+ {
+ appendStringInfoString(inBuf, q.data);
+ resetStringInfo(&q);
+ break;
+ }
+ }
+ }
+
+ if (c == ';')
+ {
+ appendStringInfoChar(inBuf, (char) ';');
+ break;
+ }
+
+ if (c == '\n')
+ appendStringInfoChar(inBuf, (char) '\n');
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will remove entries from dbname_oid_list that pattern matching any
+ * in the db_exclude_patterns list. dbname_oid_list maybe inplace modified.
+ *
+ * returns, number of database will be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
+ SimpleDatabaseOidListCell *dboidprecell = NULL;
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ /* Return 0 if there is no database to restore. */
+ if (dboid_cell == NULL)
+ return 0;
+
+ query = createPQExpBuffer();
+
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ while (dboid_cell != NULL)
+ {
+ bool skip_db_restore = false;
+ SimpleDatabaseOidListCell *next = dboid_cell->next;
+
+ for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ {
+ /*
+ * the construct pattern matching query:
+ * SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE
+ * pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from the
+ * dbname_oid_list, which is initially extracted from the map.dat
+ * file located in the backup directory. that's why we need
+ * quote_literal_cstr.
+ *
+ * If no db connection, then consider PATTERN as NAME.
+ */
+ if (pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0)
+ skip_db_restore = true;
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, celldb->val, false,
+ false, NULL, quote_literal_cstr(dboid_cell->db_name),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ celldb->val);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database \"%s\" is matching with exclude pattern: \"%s\"", dboid_cell->db_name, celldb->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /* Increment count if database needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
+ simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ }
+ else
+ {
+ count_db++;
+ dboidprecell = dboid_cell;
+ }
+
+ /* Process next dbname from dbname list. */
+ dboid_cell = next;
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ /*
+ * If there is only global.dat file in dump, then return from here as there
+ * is no database to restore.
+ */
+ if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ {
+ pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid = InvalidOid;
+ char db_oid_str[MAXPGPATH + 1] = {'\0'};
+ char dbname[MAXPGPATH + 1] = {'\0'};
+
+ /* Extract dboid. */
+ sscanf(line, "%u" , &db_oid);
+ sscanf(line, "%s" , db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found database \"%s\" (OID: %u) in map.dat file while restoring.", dbname, db_oid);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
+ SimpleDatabaseOidListCell *dboid_cell;
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /*
+ * If map.dat has no entry, return from here after processing
+ * global.dat file.
+ */
+ if (dbname_oid_list.head == NULL)
+ return process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+
+ /*
+ * processing pg_retsore --exclude-database=PATTERN/NAME if no connection.
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ n_errors_total = process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL)
+ {
+ pg_log_info("no database needs to restore out of %d databases", num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * Till now, we made a list of databases, those needs to be restored
+ * after skipping names of exclude-database. Now we can launch parallel
+ * workers to restore these databases.
+ */
+ dboid_cell = dbname_oid_list.head;
+
+ while(dboid_cell != NULL)
+ {
+ char subdirpath[MAXPGPATH];
+ int n_errors;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored into
+ * already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+
+ pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+
+ /* Restore single database. */
+ n_errors = restoreOneDatabase(subdirpath, opts, numWorkers, true, count);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dboid_cell->db_name, n_errors);
+ }
+
+ dboid_cell = dboid_cell->next;
+ count++;
+ }
+
+ /* Log number of processed databases.*/
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+
+ return n_errors_total;
+}
+
+/*
+ * process_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ *
+ * returns the number of errors while processing global.dat
+ */
+static int
+process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+ int n_errors = 0;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ copy_or_print_global_file(outfile, pfile);
+ return 0;
+ }
+
+ /* Init sqlstatement to append commands. */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements. */
+ while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ {
+ pg_log_info("executing query: %s", sqlstatement.data);
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ n_errors++;
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ /* Print a summary of ignored errors during global.dat. */
+ if (n_errors)
+ pg_log_warning("errors ignored on global.dat file restore: %d", n_errors);
+
+ fclose(pfile);
+
+ return n_errors;
+}
+
+/*
+ * copy_or_print_global_file
+ *
+ * This will copy global.dat file into out file. If "-" is used as outfile,
+ * then print commands to the stdout.
+ */
+static void
+copy_or_print_global_file(const char *outfile, FILE *pfile)
+{
+ char out_file_path[MAXPGPATH];
+ FILE *OPF;
+ int c;
+
+ /* "-" is used for stdout. */
+ if (strcmp(outfile, "-") == 0)
+ OPF = stdout;
+ else
+ {
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ OPF = fopen(out_file_path, PG_BINARY_W);
+
+ if (OPF == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", outfile);
+ }
+ }
+
+ /* Append global.dat into out file or print to the stdout. */
+ while ((c = fgetc(pfile)) != EOF)
+ fputc(c, OPF);
+
+ fclose(pfile);
+
+ /* Close out file. */
+ if (strcmp(outfile, "-") != 0)
+ fclose(OPF);
+}
+
+/*
+ * simple_db_oid_list_append
+ *
+ * appends a node to the list in the end.
+ */
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname)
+{
+ SimpleDatabaseOidListCell *cell;
+
+ cell = pg_malloc_object(SimpleDatabaseOidListCell);
+
+ cell->next = NULL;
+ cell->db_oid = db_oid;
+ cell->db_name = pg_strdup(dbname);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * simple_db_oid_full_list_delete
+ *
+ * delete all cell from dbname and dboid list.
+ */
+static void
+simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
+{
+ SimpleDatabaseOidListCell *cell = list->head;
+ SimpleDatabaseOidListCell *nextcell = NULL;
+
+ while (cell)
+ {
+ nextcell = cell->next;
+ pfree (cell);
+ cell = nextcell;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_string_full_list_delete
+ *
+ * delete all cell from string list.
+ */
+static void
+simple_string_full_list_delete(SimpleStringList *list)
+{
+ SimpleStringListCell *cell = list->head;
+ SimpleStringListCell *cellnext = NULL;
+
+ while (cell)
+ {
+ cellnext = cell->next;
+ pfree(cell);
+ cell = cellnext;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_db_oid_list_delete
+ *
+ * delete cell from database and oid list.
+ */
+static void
+simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev)
+{
+ if (prev == NULL)
+ {
+ list->head = cell->next;
+ pfree(cell);
+ }
+ else
+ {
+ prev->next = cell->next;
+ pfree(cell);
+ }
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr
+ *
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..0bbcdbe84a7
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,11 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +249,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9840060997f..b43a3e48e3b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2695,6 +2695,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.39.3
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-03-11 14:42 ` Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Álvaro Herrera @ 2025-03-11 14:42 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-Mar-11, Mahendra Singh Thalor wrote:
> On Wed, 5 Mar 2025 at 20:42, Álvaro Herrera <[email protected]> wrote:
> > Okay, we should probably fix that, but I think the new map.dat file your
> > patch adds is going to make the problem worse, because it doesn't look
> > like you handled that case in any particular way that would make it not
> > fail.
>
> As Jian also pointed out, we should not allow \n\r in dbnames. I am
> keeping dbanames as single line names only.
Ehm, did you get consensus on adding such a restriction?
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
@ 2025-03-11 15:41 ` Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Mahendra Singh Thalor @ 2025-03-11 15:41 UTC (permalink / raw)
To: Álvaro Herrera <[email protected]>; +Cc: jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Tue, 11 Mar 2025 at 20:12, Álvaro Herrera <[email protected]>
wrote:
>
> On 2025-Mar-11, Mahendra Singh Thalor wrote:
>
> > On Wed, 5 Mar 2025 at 20:42, Álvaro Herrera <[email protected]>
wrote:
>
> > > Okay, we should probably fix that, but I think the new map.dat file
your
> > > patch adds is going to make the problem worse, because it doesn't look
> > > like you handled that case in any particular way that would make it
not
> > > fail.
> >
> > As Jian also pointed out, we should not allow \n\r in dbnames. I am
> > keeping dbanames as single line names only.
>
> Ehm, did you get consensus on adding such a restriction?
>
Hi Alvaro,
In map.dat file, I tried to fix this issue by adding number of characters
in dbname but as per code comments, as of now, we are not supporting \n\r
in dbnames so i removed handling.
I will do some more study to fix this issue.
/*
> * Append the given string to the shell command being built in the buffer,
> * with shell-style quoting as needed to create exactly one argument.
> *
> * Forbid LF or CR characters, which have scant practical use beyond
> designing
> * security breaches. The Windows command shell is unusable as a conduit
> for
> * arguments containing LF or CR characters. A future major release should
> * reject those characters in CREATE ROLE and CREATE DATABASE, because use
> * there eventually leads to errors here.
> *
> * appendShellString() simply prints an error and dies if LF or CR appears.
> * appendShellStringNoError() omits those characters from the result, and
> * returns false if there were any.
> */
> void
> appendShellString(PQExpBuffer buf, const char *str)
Sorry, in the v22 patches, I missed to use the "git add connectdb.c" file.
(Thanks Andrew for reporting this offline)
Here, I am attaching updated patches for review and testing.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v23_0001_move-common-code-of-pg_dumpall-and-pg_restore-to-new_file.patch (26.6K, 3-v23_0001_move-common-code-of-pg_dumpall-and-pg_restore-to-new_file.patch)
download | inline diff:
From f61af4fcf612e0b811824c72d35e1bcdb6eb8ae6 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Tue, 11 Mar 2025 20:54:33 +0530
Subject: [PATCH 1/2] move common code related to connection to new the file
ConnectDatabase is used by both pg_dumpall, pg_restore
and pg_dump so move common code to new file.
new file name: connectdb.c
---
src/bin/pg_dump/Makefile | 9 +-
src/bin/pg_dump/connectdb.c | 294 +++++++++++++++++++++++++++
src/bin/pg_dump/connectdb.h | 26 +++
src/bin/pg_dump/meson.build | 2 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 6 +-
src/bin/pg_dump/pg_backup_db.c | 75 +------
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 279 ++-----------------------
9 files changed, 352 insertions(+), 343 deletions(-)
create mode 100644 src/bin/pg_dump/connectdb.c
create mode 100644 src/bin/pg_dump/connectdb.h
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..c488ab4aecf 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -40,7 +40,8 @@ OBJS = \
pg_backup_directory.o \
pg_backup_null.o \
pg_backup_tar.o \
- pg_backup_utils.o
+ pg_backup_utils.o \
+ connectdb.o
all: pg_dump pg_restore pg_dumpall
@@ -50,8 +51,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o dumputils.o filter.o connectdb.o pg_backup_utils.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o connectdb.o pg_backup_utils.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
@@ -71,5 +72,5 @@ uninstall:
rm -f $(addprefix '$(DESTDIR)$(bindir)'/, pg_dump$(X) pg_restore$(X) pg_dumpall$(X))
clean distclean:
- rm -f pg_dump$(X) pg_restore$(X) pg_dumpall$(X) $(OBJS) pg_dump.o common.o pg_dump_sort.o pg_restore.o pg_dumpall.o
+ rm -f pg_dump$(X) pg_restore$(X) pg_dumpall$(X) $(OBJS) pg_dump.o common.o pg_dump_sort.o pg_restore.o pg_dumpall.o connectdb.o
rm -rf tmp_check
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
new file mode 100644
index 00000000000..3e1fbe98c25
--- /dev/null
+++ b/src/bin/pg_dump/connectdb.c
@@ -0,0 +1,294 @@
+/*-------------------------------------------------------------------------
+ *
+ * connectdb.c
+ * This is a common file connection to the database.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/connectdb.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "connectdb.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+
+/*
+ * ConnectDatabase
+ *
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the 'connstr' is set to a connection string containing
+ * the options used and 'server_version' is set to version so that caller
+ * can use them.
+ */
+PGconn *
+ConnectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version, char *password,
+ char *override_dbname)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 8;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ if (override_dbname)
+ {
+ keywords[i] = "dbname";
+ values[i++] = override_dbname;
+ }
+
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. If requested, remember the options used, in the
+ * form of a connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If requested, then copy server version to out variable. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/*
+ * constructConnStr
+ *
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * executeQuery
+ *
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
diff --git a/src/bin/pg_dump/connectdb.h b/src/bin/pg_dump/connectdb.h
new file mode 100644
index 00000000000..9e1e7ef33d0
--- /dev/null
+++ b/src/bin/pg_dump/connectdb.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * connectdb.h
+ * Common header file for connection to the database.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/connectdb.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONNECTDB_H
+#define CONNECTDB_H
+
+#include "pg_backup.h"
+#include "pg_backup_utils.h"
+
+extern PGconn *ConnectDatabase(const char *dbname, const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version,
+ char *password, char *override_dbname);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+#endif /* CONNECTDB_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..d5f805fb511 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -30,6 +30,7 @@ pg_dump_sources = files(
'common.c',
'pg_dump.c',
'pg_dump_sort.c',
+ 'connectdb.c',
)
if host_system == 'windows'
@@ -49,6 +50,7 @@ bin_targets += pg_dump
pg_dumpall_sources = files(
'pg_dumpall.c',
+ 'connectdb.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index e783cc68d89..731cb2d19fb 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -291,7 +291,7 @@ typedef void (*SetupWorkerPtrType) (Archive *AH);
* Main archiver interface.
*/
-extern void ConnectDatabase(Archive *AHX,
+extern void ConnectDatabaseAhx(Archive *AHX,
const ConnParams *cparams,
bool isReconnect);
extern void DisconnectDatabase(Archive *AHX);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 7480e122b61..12f3f39e39b 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -413,7 +413,7 @@ RestoreArchive(Archive *AHX)
AHX->minRemoteVersion = 0;
AHX->maxRemoteVersion = 9999999;
- ConnectDatabase(AHX, &ropt->cparams, false);
+ ConnectDatabaseAhx(AHX, &ropt->cparams, false);
/*
* If we're talking to the DB directly, don't send comments since they
@@ -4430,7 +4430,7 @@ restore_toc_entries_postfork(ArchiveHandle *AH, TocEntry *pending_list)
/*
* Now reconnect the single parent connection.
*/
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
/* re-establish fixed state */
_doSetFixedOutputState(AH);
@@ -5047,7 +5047,7 @@ CloneArchive(ArchiveHandle *AH)
* Connect our new clone object to the database, using the same connection
* parameters used for the original connection.
*/
- ConnectDatabase((Archive *) clone, &clone->public.ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) clone, &clone->public.ropt->cparams, true);
/* re-establish fixed state */
if (AH->mode == archModeRead)
diff --git a/src/bin/pg_dump/pg_backup_db.c b/src/bin/pg_dump/pg_backup_db.c
index 71c55d2466a..227dd963984 100644
--- a/src/bin/pg_dump/pg_backup_db.c
+++ b/src/bin/pg_dump/pg_backup_db.c
@@ -19,6 +19,7 @@
#include "common/connect.h"
#include "common/string.h"
+#include "connectdb.h"
#include "parallel.h"
#include "pg_backup_archiver.h"
#include "pg_backup_db.h"
@@ -86,9 +87,9 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* ArchiveHandle's connCancel, before closing old connection. Otherwise
* an ill-timed SIGINT could try to access a dead connection.
*/
- AH->connection = NULL; /* dodge error check in ConnectDatabase */
+ AH->connection = NULL; /* dodge error check in ConnectDatabaseAhx */
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
PQfinish(oldConn);
}
@@ -105,14 +106,13 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* username never does change, so one savedPassword is sufficient.
*/
void
-ConnectDatabase(Archive *AHX,
+ConnectDatabaseAhx(Archive *AHX,
const ConnParams *cparams,
bool isReconnect)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
trivalue prompt_password;
char *password;
- bool new_pass;
if (AH->connection)
pg_fatal("already connected to a database");
@@ -125,69 +125,10 @@ ConnectDatabase(Archive *AHX,
if (prompt_password == TRI_YES && password == NULL)
password = simple_prompt("Password: ", false);
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- const char *keywords[8];
- const char *values[8];
- int i = 0;
-
- /*
- * If dbname is a connstring, its entries can override the other
- * values obtained from cparams; but in turn, override_dbname can
- * override the dbname component of it.
- */
- keywords[i] = "host";
- values[i++] = cparams->pghost;
- keywords[i] = "port";
- values[i++] = cparams->pgport;
- keywords[i] = "user";
- values[i++] = cparams->username;
- keywords[i] = "password";
- values[i++] = password;
- keywords[i] = "dbname";
- values[i++] = cparams->dbname;
- if (cparams->override_dbname)
- {
- keywords[i] = "dbname";
- values[i++] = cparams->override_dbname;
- }
- keywords[i] = "fallback_application_name";
- values[i++] = progname;
- keywords[i] = NULL;
- values[i++] = NULL;
- Assert(i <= lengthof(keywords));
-
- new_pass = false;
- AH->connection = PQconnectdbParams(keywords, values, true);
-
- if (!AH->connection)
- pg_fatal("could not connect to database");
-
- if (PQstatus(AH->connection) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(AH->connection) &&
- password == NULL &&
- prompt_password != TRI_NO)
- {
- PQfinish(AH->connection);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(AH->connection) == CONNECTION_BAD)
- {
- if (isReconnect)
- pg_fatal("reconnection failed: %s",
- PQerrorMessage(AH->connection));
- else
- pg_fatal("%s",
- PQerrorMessage(AH->connection));
- }
+ AH->connection = ConnectDatabase(cparams->dbname, NULL, cparams->pghost,
+ cparams->pgport, cparams->username,
+ prompt_password, true,
+ progname, NULL, NULL, password, cparams->override_dbname);
/* Start strict; later phases may override this. */
PQclear(ExecuteSqlQueryForSingleRow((Archive *) AH,
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index c371570501a..6bb54c1a2b4 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -934,7 +934,7 @@ main(int argc, char **argv)
* Open the database using the Archiver, so it knows about it. Errors mean
* death.
*/
- ConnectDatabase(fout, &dopt.cparams, false);
+ ConnectDatabaseAhx(fout, &dopt.cparams, false);
setup_connection(fout, dumpencoding, dumpsnapshot, use_role);
/*
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index e0867242526..e7e492afa28 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -24,11 +24,11 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
-#include "pg_backup.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -71,21 +71,15 @@ static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
static char pg_dump_bin[MAXPGPATH];
-static const char *progname;
+const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -125,8 +119,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -483,19 +475,22 @@ main(int argc, char *argv[])
*/
if (pgdb)
{
- conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase(pgdb, connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
}
else
{
- conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase("postgres", connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
- conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ conn = ConnectDatabase("template1", connstr, pghost, pgport, pguser,
+ prompt_password, true,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
{
@@ -1718,256 +1713,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
--
2.39.3
[application/octet-stream] v23_0002_pg_dumpall-with-non-text_format-11th_march.patch (61.7K, 4-v23_0002_pg_dumpall-with-non-text_format-11th_march.patch)
download | inline diff:
From bc18451be43867723959f20f7192007342964393 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Tue, 11 Mar 2025 19:29:15 +0530
Subject: [PATCH 2/2] pg_dumpall with directory|tar|custom format and restore
it by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat to restore all databases. If global.dat file is exist in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list (if exist)
for --exclude-database=PATTERN for pg_restore
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
if no db connection, then PATTERN=NAME matching only
for each database, we are cleaning on_exit_nicely_index list.
at the end of restore, we are giving warning with total number of errors (including global.dat,
and each database errors) and for each database, we are printing warning with dbname and total
errors.
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
---
doc/src/sgml/ref/pg_dumpall.sgml | 80 ++-
doc/src/sgml/ref/pg_restore.sgml | 41 +-
src/bin/pg_dump/meson.build | 3 +-
src/bin/pg_dump/parallel.c | 11 +-
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 20 +-
src/bin/pg_dump/pg_backup_archiver.h | 3 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 22 +-
src/bin/pg_dump/pg_backup_utils.h | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 280 +++++++--
src/bin/pg_dump/pg_restore.c | 847 ++++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 9 +
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 1248 insertions(+), 79 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index c2fa5be9519..c36802e06fd 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster based on specified dump format </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -121,7 +121,83 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- </para>
+ Note: This option can be omitted only when <option>--format</option> is plain
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 199ea3345f3..46bdbc092c3 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> database from an
+ archive file created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -37,9 +38,10 @@ PostgreSQL documentation
<title>Description</title>
<para>
- <application>pg_restore</application> is a utility for restoring a
+ <application>pg_restore</application> is a utility for restoring
<productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
database to the state it was in at the time it was saved. The
archive files also allow <application>pg_restore</application> to
@@ -140,6 +142,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from dump of <application>pg_dumpall</application>.
</para>
<para>
@@ -166,6 +170,25 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +338,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index d5f805fb511..dc1ed410838 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -16,6 +16,7 @@ pg_dump_common_sources = files(
'pg_backup_null.c',
'pg_backup_tar.c',
'pg_backup_utils.c',
+ 'connectdb.c',
)
pg_dump_common = static_library('libpgdump_common',
@@ -30,7 +31,6 @@ pg_dump_sources = files(
'common.c',
'pg_dump.c',
'pg_dump_sort.c',
- 'connectdb.c',
)
if host_system == 'windows'
@@ -70,6 +70,7 @@ bin_targets += pg_dumpall
pg_restore_sources = files(
'pg_restore.c',
+ 'connectdb.c',
)
if host_system == 'windows'
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..a36d2a5bf84 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -326,11 +326,18 @@ getThreadLocalPQExpBuffer(void)
* pg_dump and pg_restore call this to register the cleanup handler
* as soon as they've created the ArchiveHandle.
*/
-void
+int
on_exit_close_archive(Archive *AHX)
{
shutdown_info.AHX = AHX;
- on_exit_nicely(archive_close_connection, &shutdown_info);
+ return on_exit_nicely(archive_close_connection, &shutdown_info);
+}
+
+void
+replace_on_exit_close_archive(Archive *AHX, int idx)
+{
+ shutdown_info.AHX = AHX;
+ set_on_exit_nicely_entry(archive_close_connection, &shutdown_info, idx);
}
/*
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 731cb2d19fb..41ee305850a 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -309,7 +309,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 12f3f39e39b..ad2a62a2c9c 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -85,7 +85,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -336,9 +336,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -455,7 +460,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1291,7 +1296,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1670,7 +1675,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1690,7 +1696,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index a2064f471ed..ae433132435 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -385,7 +385,8 @@ struct _tocEntry
};
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
-extern void on_exit_close_archive(Archive *AHX);
+extern int on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX, int idx);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..59ece2999a8 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -61,14 +61,26 @@ set_dump_section(const char *arg, int *dumpSections)
/* Register a callback to be run when exit_nicely is invoked. */
-void
+int
on_exit_nicely(on_exit_nicely_callback function, void *arg)
{
- if (on_exit_nicely_index >= MAX_ON_EXIT_NICELY)
- pg_fatal("out of on_exit_nicely slots");
- on_exit_nicely_list[on_exit_nicely_index].function = function;
- on_exit_nicely_list[on_exit_nicely_index].arg = arg;
+ set_on_exit_nicely_entry(function, arg, on_exit_nicely_index);
on_exit_nicely_index++;
+
+ return (on_exit_nicely_index - 1);
+}
+
+void
+set_on_exit_nicely_entry(on_exit_nicely_callback function, void *arg, int i)
+{
+ if (i >= MAX_ON_EXIT_NICELY)
+ pg_fatal("out of on_exit_nicely slots");
+
+ if (i > on_exit_nicely_index)
+ pg_fatal("no entry exists on %d index into on_exit_nicely slots", i);
+
+ on_exit_nicely_list[i].function = function;
+ on_exit_nicely_list[i].arg = arg;
}
/*
diff --git a/src/bin/pg_dump/pg_backup_utils.h b/src/bin/pg_dump/pg_backup_utils.h
index 38551944513..57f3197f103 100644
--- a/src/bin/pg_dump/pg_backup_utils.h
+++ b/src/bin/pg_dump/pg_backup_utils.h
@@ -28,7 +28,8 @@ typedef void (*on_exit_nicely_callback) (int code, void *arg);
extern const char *progname;
extern void set_dump_section(const char *arg, int *dumpSections);
-extern void on_exit_nicely(on_exit_nicely_callback function, void *arg);
+extern int on_exit_nicely(on_exit_nicely_callback function, void *arg);
+extern void set_on_exit_nicely_entry(on_exit_nicely_callback function, void *arg, int idx);
extern void exit_nicely(int code) pg_attribute_noreturn();
/* In pg_dump, we modify pg_fatal to call exit_nicely instead of exit */
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 6bb54c1a2b4..5b6d4364eb3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1186,7 +1186,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index e7e492afa28..ee7c62da09d 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -64,9 +65,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -75,6 +77,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
const char *progname;
@@ -104,7 +108,7 @@ static int no_subscriptions = 0;
static int no_toast_compression = 0;
static int no_unlogged_table_data = 0;
static int no_role_passwords = 0;
-static int server_version;
+static int server_version;
static int load_via_partition_root = 0;
static int on_conflict_do_nothing = 0;
static int statistics_only = 0;
@@ -143,6 +147,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -188,6 +193,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -237,7 +244,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -265,7 +272,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -414,6 +423,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If non-plain format is specified then we must provide the
+ * file name to create one main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -468,6 +492,33 @@ main(int argc, char *argv[])
if (statistics_only)
appendPQExpBufferStr(pgdumpopts, " --statistics-only");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char toc_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -507,19 +558,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -619,7 +657,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -632,7 +670,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -643,12 +681,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster based on specified dump format.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1551,10 +1591,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1568,7 +1611,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1576,9 +1619,33 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_fatal("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1593,6 +1660,18 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is non-plain dump format, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1611,9 +1690,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1622,19 +1709,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char toc_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(toc_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ toc_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1644,7 +1742,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1653,17 +1752,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is non-plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1808,3 +1926,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if(!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 13e4dc507e0..9b802e7a6bd 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,30 +41,77 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/connect.h"
+#include "compress_io.h"
+#include "common/string.h"
+#include "connectdb.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
+#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
+typedef struct SimpleDatabaseOidListCell
+{
+ struct SimpleDatabaseOidListCell *next;
+ Oid db_oid;
+ const char *db_name;
+} SimpleDatabaseOidListCell;
+
+typedef struct SimpleDatabaseOidList
+{
+ SimpleDatabaseOidListCell *head;
+ SimpleDatabaseOidListCell *tail;
+} SimpleDatabaseOidList;
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num);
+static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static void copy_or_print_global_file(const char *outfile, FILE *pfile);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleDatabaseOidList *dbname_oid_list);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname);
+static void simple_string_full_list_delete(SimpleStringList *list);
+static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
+static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list,
+ Oid db_oid, const char *dbname);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
+static int on_exit_index = 0;
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -86,6 +133,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -136,6 +184,7 @@ main(int argc, char **argv)
{"no-statistics", no_argument, &no_statistics, 1},
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -164,7 +213,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -191,11 +240,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -310,6 +362,10 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6:
+ /* list of databases patterns those needs to skip while restoring */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -337,6 +393,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -417,6 +480,108 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL && !IsFileExistsInDirectory(inputFileSpec, "toc.dat") &&
+ IsFileExistsInDirectory(inputFileSpec, "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql commands. */
+
+ /*
+ * User is suggested to use single database dump for --list option.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring multiple databases by archive of pg_dumpall");
+
+ /*
+ * To restore multiple databases, -C (create database) option should be specified.
+ * Even there is single database in dump, report error because it might be possible
+ * that database hasn't created so better we report error.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("-C/--create option should be specified when restoring multiple databases by archive of pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("If db is already created and dump has single db dump, then use particular dump file.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Connect to database to execute global sql commands from global.dat file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ /* Open global.dat file and execute/append all the global sql commands. */
+ n_errors = process_global_sql_commands(conn, inputFileSpec,
+ opts->filename);
+
+ if (conn)
+ PQfinish(conn);
+
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ n_errors = restoreAllDatabases(conn, inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_full_list_delete(&db_exclude_patterns);
+ }
+ else /* process if global.dat file does not exist. */
+ {
+ if (db_exclude_patterns.head != NULL)
+ pg_fatal("option --exclude-database can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ n_errors = restoreOneDatabase(inputFileSpec, opts, numWorkers, false, 0);
+ }
+
+ on_exit_index = 0; /* Reset index. */
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -425,8 +590,14 @@ main(int argc, char **argv)
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
* it's still NULL, the cleanup function will just be a no-op.
+ * If we are restoring multiple databases, then save index of exit_nicely
+ * so that we can use same slot for all the databases as we already closed
+ * the previous archive by CloseArchive.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_index = on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH, on_exit_index);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -446,25 +617,22 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n"
+ "If archive is created by pg_dumpall, then restores multiple databases also. \n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -482,6 +650,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -494,6 +663,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -530,8 +700,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be combined\n"
+ "and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -636,3 +806,648 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * IsFileExistsInDirectory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+IsFileExistsInDirectory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.dat file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+ReadOneStatement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+ int m;
+
+ StringInfoData q;
+ initStringInfo(&q);
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != '\n' && c != ';')
+ {
+ appendStringInfoChar(inBuf, (char) c);
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != ';' && c != '\n')
+ appendStringInfoChar(inBuf, (char) c);
+ else
+ break;
+ }
+ }
+
+ if (c == '\'' || c == '"')
+ {
+ appendStringInfoChar(&q, (char) c);
+ m = c;
+
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(&q, (char) c);
+
+ if(c == m)
+ {
+ appendStringInfoString(inBuf, q.data);
+ resetStringInfo(&q);
+ break;
+ }
+ }
+ }
+
+ if (c == ';')
+ {
+ appendStringInfoChar(inBuf, (char) ';');
+ break;
+ }
+
+ if (c == '\n')
+ appendStringInfoChar(inBuf, (char) '\n');
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will remove entries from dbname_oid_list that pattern matching any
+ * in the db_exclude_patterns list. dbname_oid_list maybe inplace modified.
+ *
+ * returns, number of database will be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
+ SimpleDatabaseOidListCell *dboidprecell = NULL;
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ /* Return 0 if there is no database to restore. */
+ if (dboid_cell == NULL)
+ return 0;
+
+ query = createPQExpBuffer();
+
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ while (dboid_cell != NULL)
+ {
+ bool skip_db_restore = false;
+ SimpleDatabaseOidListCell *next = dboid_cell->next;
+
+ for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ {
+ /*
+ * the construct pattern matching query:
+ * SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE
+ * pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from the
+ * dbname_oid_list, which is initially extracted from the map.dat
+ * file located in the backup directory. that's why we need
+ * quote_literal_cstr.
+ *
+ * If no db connection, then consider PATTERN as NAME.
+ */
+ if (pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0)
+ skip_db_restore = true;
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, celldb->val, false,
+ false, NULL, quote_literal_cstr(dboid_cell->db_name),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ celldb->val);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database \"%s\" is matching with exclude pattern: \"%s\"", dboid_cell->db_name, celldb->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /* Increment count if database needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
+ simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ }
+ else
+ {
+ count_db++;
+ dboidprecell = dboid_cell;
+ }
+
+ /* Process next dbname from dbname list. */
+ dboid_cell = next;
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ /*
+ * If there is only global.dat file in dump, then return from here as there
+ * is no database to restore.
+ */
+ if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ {
+ pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid = InvalidOid;
+ char db_oid_str[MAXPGPATH + 1] = {'\0'};
+ char dbname[MAXPGPATH + 1] = {'\0'};
+
+ /* Extract dboid. */
+ sscanf(line, "%u" , &db_oid);
+ sscanf(line, "%s" , db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found database \"%s\" (OID: %u) in map.dat file while restoring.", dbname, db_oid);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
+ SimpleDatabaseOidListCell *dboid_cell;
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /*
+ * If map.dat has no entry, return from here after processing
+ * global.dat file.
+ */
+ if (dbname_oid_list.head == NULL)
+ return process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+
+ /*
+ * processing pg_retsore --exclude-database=PATTERN/NAME if no connection.
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ n_errors_total = process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL)
+ {
+ pg_log_info("no database needs to restore out of %d databases", num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * Till now, we made a list of databases, those needs to be restored
+ * after skipping names of exclude-database. Now we can launch parallel
+ * workers to restore these databases.
+ */
+ dboid_cell = dbname_oid_list.head;
+
+ while(dboid_cell != NULL)
+ {
+ char subdirpath[MAXPGPATH];
+ int n_errors;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored into
+ * already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+
+ pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+
+ /* Restore single database. */
+ n_errors = restoreOneDatabase(subdirpath, opts, numWorkers, true, count);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dboid_cell->db_name, n_errors);
+ }
+
+ dboid_cell = dboid_cell->next;
+ count++;
+ }
+
+ /* Log number of processed databases.*/
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+
+ return n_errors_total;
+}
+
+/*
+ * process_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ *
+ * returns the number of errors while processing global.dat
+ */
+static int
+process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+ int n_errors = 0;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ copy_or_print_global_file(outfile, pfile);
+ return 0;
+ }
+
+ /* Init sqlstatement to append commands. */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements. */
+ while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ {
+ pg_log_info("executing query: %s", sqlstatement.data);
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ n_errors++;
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ /* Print a summary of ignored errors during global.dat. */
+ if (n_errors)
+ pg_log_warning("errors ignored on global.dat file restore: %d", n_errors);
+
+ fclose(pfile);
+
+ return n_errors;
+}
+
+/*
+ * copy_or_print_global_file
+ *
+ * This will copy global.dat file into out file. If "-" is used as outfile,
+ * then print commands to the stdout.
+ */
+static void
+copy_or_print_global_file(const char *outfile, FILE *pfile)
+{
+ char out_file_path[MAXPGPATH];
+ FILE *OPF;
+ int c;
+
+ /* "-" is used for stdout. */
+ if (strcmp(outfile, "-") == 0)
+ OPF = stdout;
+ else
+ {
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ OPF = fopen(out_file_path, PG_BINARY_W);
+
+ if (OPF == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", outfile);
+ }
+ }
+
+ /* Append global.dat into out file or print to the stdout. */
+ while ((c = fgetc(pfile)) != EOF)
+ fputc(c, OPF);
+
+ fclose(pfile);
+
+ /* Close out file. */
+ if (strcmp(outfile, "-") != 0)
+ fclose(OPF);
+}
+
+/*
+ * simple_db_oid_list_append
+ *
+ * appends a node to the list in the end.
+ */
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname)
+{
+ SimpleDatabaseOidListCell *cell;
+
+ cell = pg_malloc_object(SimpleDatabaseOidListCell);
+
+ cell->next = NULL;
+ cell->db_oid = db_oid;
+ cell->db_name = pg_strdup(dbname);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * simple_db_oid_full_list_delete
+ *
+ * delete all cell from dbname and dboid list.
+ */
+static void
+simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
+{
+ SimpleDatabaseOidListCell *cell = list->head;
+ SimpleDatabaseOidListCell *nextcell = NULL;
+
+ while (cell)
+ {
+ nextcell = cell->next;
+ pfree (cell);
+ cell = nextcell;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_string_full_list_delete
+ *
+ * delete all cell from string list.
+ */
+static void
+simple_string_full_list_delete(SimpleStringList *list)
+{
+ SimpleStringListCell *cell = list->head;
+ SimpleStringListCell *cellnext = NULL;
+
+ while (cell)
+ {
+ cellnext = cell->next;
+ pfree(cell);
+ cell = cellnext;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_db_oid_list_delete
+ *
+ * delete cell from database and oid list.
+ */
+static void
+simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev)
+{
+ if (prev == NULL)
+ {
+ list->head = cell->next;
+ pfree(cell);
+ }
+ else
+ {
+ prev->next = cell->next;
+ pfree(cell);
+ }
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr
+ *
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..0bbcdbe84a7
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,11 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +249,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index dfe2690bdd3..a922d983514 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2697,6 +2697,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.39.3
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-03-11 17:05 ` Álvaro Herrera <[email protected]>
2025-03-11 17:52 ` Re: Non-text mode for pg_dumpall Dagfinn Ilmari Mannsåker <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
0 siblings, 2 replies; 29+ messages in thread
From: Álvaro Herrera @ 2025-03-11 17:05 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Hello,
On 2025-Mar-11, Mahendra Singh Thalor wrote:
> In map.dat file, I tried to fix this issue by adding number of characters
> in dbname but as per code comments, as of now, we are not supporting \n\r
> in dbnames so i removed handling.
> I will do some more study to fix this issue.
Yeah, I think this is saying that you should not consider the contents
of map.dat as a shell string. After all, you're not going to _execute_
that file via the shell.
Maybe for map.dat you need to escape such characters somehow, so that
they don't appear as literal newlines/carriage returns.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
@ 2025-03-11 17:52 ` Dagfinn Ilmari Mannsåker <[email protected]>
2025-03-11 20:16 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
1 sibling, 1 reply; 29+ messages in thread
From: Dagfinn Ilmari Mannsåker @ 2025-03-11 17:52 UTC (permalink / raw)
To: Álvaro Herrera <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Álvaro Herrera <[email protected]> writes:
> Hello,
>
> On 2025-Mar-11, Mahendra Singh Thalor wrote:
>
>> In map.dat file, I tried to fix this issue by adding number of characters
>> in dbname but as per code comments, as of now, we are not supporting \n\r
>> in dbnames so i removed handling.
>> I will do some more study to fix this issue.
>
> Yeah, I think this is saying that you should not consider the contents
> of map.dat as a shell string. After all, you're not going to _execute_
> that file via the shell.
>
> Maybe for map.dat you need to escape such characters somehow, so that
> they don't appear as literal newlines/carriage returns.
I haven't looked at the code for this, but why are we inventing an
ad-hoc file format? Why not use JSON, like we do for backup manifests?
Then storing arbitrary database names won't be a problem.
- ilmari
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 17:52 ` Re: Non-text mode for pg_dumpall Dagfinn Ilmari Mannsåker <[email protected]>
@ 2025-03-11 20:16 ` Andrew Dunstan <[email protected]>
2025-03-11 21:03 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Dunstan @ 2025-03-11 20:16 UTC (permalink / raw)
To: Dagfinn Ilmari Mannsåker <[email protected]>; Álvaro Herrera <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-03-11 Tu 1:52 PM, Dagfinn Ilmari Mannsåker wrote:
> Álvaro Herrera <[email protected]> writes:
>
>> Hello,
>>
>> On 2025-Mar-11, Mahendra Singh Thalor wrote:
>>
>>> In map.dat file, I tried to fix this issue by adding number of characters
>>> in dbname but as per code comments, as of now, we are not supporting \n\r
>>> in dbnames so i removed handling.
>>> I will do some more study to fix this issue.
>> Yeah, I think this is saying that you should not consider the contents
>> of map.dat as a shell string. After all, you're not going to _execute_
>> that file via the shell.
>>
>> Maybe for map.dat you need to escape such characters somehow, so that
>> they don't appear as literal newlines/carriage returns.
> I haven't looked at the code for this, but why are we inventing an
> ad-hoc file format? Why not use JSON, like we do for backup manifests?
> Then storing arbitrary database names won't be a problem.
>
I'm not sure everyone thinks that was a good idea for backup manifests
(in fact I know some don't), and it seems somewhat like overkill for a
simple map of oids to database names.
cheers
andrew
>
>
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 17:52 ` Re: Non-text mode for pg_dumpall Dagfinn Ilmari Mannsåker <[email protected]>
2025-03-11 20:16 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
@ 2025-03-11 21:03 ` Álvaro Herrera <[email protected]>
2025-03-11 22:37 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Álvaro Herrera @ 2025-03-11 21:03 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Dagfinn Ilmari Mannsåker <[email protected]>; Mahendra Singh Thalor <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-Mar-11, Andrew Dunstan wrote:
> I'm not sure everyone thinks that was a good idea for backup manifests (in
> fact I know some don't), and it seems somewhat like overkill for a simple
> map of oids to database names.
If such a simple system can be made to work for all possible valid
database names, then I agree with you. But if it forces us to restrict
database names to not contain newlines or other funny chars that are so
far unrestricted, then I would take the other position.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"La victoria es para quien se atreve a estar solo"
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 17:52 ` Re: Non-text mode for pg_dumpall Dagfinn Ilmari Mannsåker <[email protected]>
2025-03-11 20:16 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-11 21:03 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
@ 2025-03-11 22:37 ` Andrew Dunstan <[email protected]>
2025-03-11 23:14 ` Re: Non-text mode for pg_dumpall Isaac Morland <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Dunstan @ 2025-03-11 22:37 UTC (permalink / raw)
To: Álvaro Herrera <[email protected]>; +Cc: Dagfinn Ilmari Mannsåker <[email protected]>; Mahendra Singh Thalor <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-03-11 Tu 5:03 PM, Álvaro Herrera wrote:
> On 2025-Mar-11, Andrew Dunstan wrote:
>
>> I'm not sure everyone thinks that was a good idea for backup manifests (in
>> fact I know some don't), and it seems somewhat like overkill for a simple
>> map of oids to database names.
> If such a simple system can be made to work for all possible valid
> database names, then I agree with you. But if it forces us to restrict
> database names to not contain newlines or other funny chars that are so
> far unrestricted, then I would take the other position.
>
Well, JSON is supposed to be UTF8. What should we do about database
names that are not UTF8?
It's kinda tempting to say we should have the file consist of lines like:
oid base64_encoded_name escaped_human_readable name
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 17:52 ` Re: Non-text mode for pg_dumpall Dagfinn Ilmari Mannsåker <[email protected]>
2025-03-11 20:16 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-11 21:03 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 22:37 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
@ 2025-03-11 23:14 ` Isaac Morland <[email protected]>
2025-03-12 07:24 ` Re: Non-text mode for pg_dumpall Laurenz Albe <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Isaac Morland @ 2025-03-11 23:14 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Álvaro Herrera <[email protected]>; Dagfinn Ilmari Mannsåker <[email protected]>; Mahendra Singh Thalor <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Tue, 11 Mar 2025 at 18:37, Andrew Dunstan <[email protected]> wrote:
Well, JSON is supposed to be UTF8. What should we do about database
> names that are not UTF8?
>
How can you have a database name that isn't encodeable in UTF-8? At this
point I'm pretty sure Unicode has subsumed essentially every character ever
mentioned in a standards document.
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 17:52 ` Re: Non-text mode for pg_dumpall Dagfinn Ilmari Mannsåker <[email protected]>
2025-03-11 20:16 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-11 21:03 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 22:37 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-11 23:14 ` Re: Non-text mode for pg_dumpall Isaac Morland <[email protected]>
@ 2025-03-12 07:24 ` Laurenz Albe <[email protected]>
0 siblings, 0 replies; 29+ messages in thread
From: Laurenz Albe @ 2025-03-12 07:24 UTC (permalink / raw)
To: Isaac Morland <[email protected]>; Andrew Dunstan <[email protected]>; +Cc: Álvaro Herrera <[email protected]>; Dagfinn Ilmari Mannsåker <[email protected]>; Mahendra Singh Thalor <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Tue, 2025-03-11 at 19:14 -0400, Isaac Morland wrote:
> On Tue, 11 Mar 2025 at 18:37, Andrew Dunstan <[email protected]> wrote:
>
> > Well, JSON is supposed to be UTF8. What should we do about database
> > names that are not UTF8?
>
> How can you have a database name that isn't encodeable in UTF-8? At this point
> I'm pretty sure Unicode has subsumed essentially every character ever mentioned
> in a standards document.
There is a difference between "encodable" and "encoded". You'd have to figure
out the actual encoding of the database name and convert that to UTF-8.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
@ 2025-03-12 07:03 ` jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
1 sibling, 1 reply; 29+ messages in thread
From: jian he @ 2025-03-12 07:03 UTC (permalink / raw)
To: Álvaro Herrera <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Wed, Mar 12, 2025 at 1:06 AM Álvaro Herrera <[email protected]> wrote:
>
> Hello,
>
> On 2025-Mar-11, Mahendra Singh Thalor wrote:
>
> > In map.dat file, I tried to fix this issue by adding number of characters
> > in dbname but as per code comments, as of now, we are not supporting \n\r
> > in dbnames so i removed handling.
> > I will do some more study to fix this issue.
>
> Yeah, I think this is saying that you should not consider the contents
> of map.dat as a shell string. After all, you're not going to _execute_
> that file via the shell.
>
> Maybe for map.dat you need to escape such characters somehow, so that
> they don't appear as literal newlines/carriage returns.
>
I am confused.
currently pg_dumpall plain format will abort when encountering dbname
containing newline.
the left dumped plain file does not contain all the cluster databases data.
if pg_dumpall non-text format aborts earlier,
it's aligned with pg_dumpall plain format?
it's also an improvement since aborts earlier, nothing will be dumped?
am i missing something?
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
@ 2025-03-12 15:48 ` Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Dunstan @ 2025-03-12 15:48 UTC (permalink / raw)
To: jian he <[email protected]>; Álvaro Herrera <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-03-12 We 3:03 AM, jian he wrote:
> On Wed, Mar 12, 2025 at 1:06 AM Álvaro Herrera <[email protected]> wrote:
>> Hello,
>>
>> On 2025-Mar-11, Mahendra Singh Thalor wrote:
>>
>>> In map.dat file, I tried to fix this issue by adding number of characters
>>> in dbname but as per code comments, as of now, we are not supporting \n\r
>>> in dbnames so i removed handling.
>>> I will do some more study to fix this issue.
>> Yeah, I think this is saying that you should not consider the contents
>> of map.dat as a shell string. After all, you're not going to _execute_
>> that file via the shell.
>>
>> Maybe for map.dat you need to escape such characters somehow, so that
>> they don't appear as literal newlines/carriage returns.
>>
> I am confused.
> currently pg_dumpall plain format will abort when encountering dbname
> containing newline.
> the left dumped plain file does not contain all the cluster databases data.
>
>
> if pg_dumpall non-text format aborts earlier,
> it's aligned with pg_dumpall plain format?
> it's also an improvement since aborts earlier, nothing will be dumped?
>
>
> am i missing something?
>
>
I think we should fix that.
But for the current proposal, Álvaro and I were talking this morning,
and we thought the simplest thing here would be to have the one line
format and escape NL/CRs in the database name.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
@ 2025-03-19 06:41 ` Mahendra Singh Thalor <[email protected]>
2025-03-27 21:15 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Mahendra Singh Thalor @ 2025-03-19 06:41 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: jian he <[email protected]>; Álvaro Herrera <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Wed, 12 Mar 2025 at 21:18, Andrew Dunstan <[email protected]> wrote:
>
>
> On 2025-03-12 We 3:03 AM, jian he wrote:
> > On Wed, Mar 12, 2025 at 1:06 AM Álvaro Herrera <[email protected]> wrote:
> >> Hello,
> >>
> >> On 2025-Mar-11, Mahendra Singh Thalor wrote:
> >>
> >>> In map.dat file, I tried to fix this issue by adding number of characters
> >>> in dbname but as per code comments, as of now, we are not supporting \n\r
> >>> in dbnames so i removed handling.
> >>> I will do some more study to fix this issue.
> >> Yeah, I think this is saying that you should not consider the contents
> >> of map.dat as a shell string. After all, you're not going to _execute_
> >> that file via the shell.
> >>
> >> Maybe for map.dat you need to escape such characters somehow, so that
> >> they don't appear as literal newlines/carriage returns.
> >>
> > I am confused.
> > currently pg_dumpall plain format will abort when encountering dbname
> > containing newline.
> > the left dumped plain file does not contain all the cluster databases data.
> >
> >
> > if pg_dumpall non-text format aborts earlier,
> > it's aligned with pg_dumpall plain format?
> > it's also an improvement since aborts earlier, nothing will be dumped?
> >
> >
> > am i missing something?
> >
> >
>
> I think we should fix that.
>
> But for the current proposal, Álvaro and I were talking this morning,
> and we thought the simplest thing here would be to have the one line
> format and escape NL/CRs in the database name.
>
>
> cheers
>
Okay. As per discussions, we will keep one line entry for each
database into map.file.
Thanks all for feedback and review.
Here, I am attaching updated patches for review and testing. These
patches can be applied on commit a6524105d20b.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v24_0001_move-common-code-of-pg_dumpall-and-pg_restore-to-new_file.patch (26.2K, 2-v24_0001_move-common-code-of-pg_dumpall-and-pg_restore-to-new_file.patch)
download | inline diff:
From 4877f3617511d245edf0012dc8ae828dd8a595e3 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Wed, 19 Mar 2025 01:18:46 +0530
Subject: [PATCH 1/2] move common code related to connection to new the file
ConnectDatabase is used by both pg_dumpall, pg_restore
and pg_dump so move common code to new file.
new file name: connectdb.c
---
src/bin/pg_dump/Makefile | 5 +-
src/bin/pg_dump/connectdb.c | 294 +++++++++++++++++++++++++++
src/bin/pg_dump/connectdb.h | 26 +++
src/bin/pg_dump/meson.build | 3 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 6 +-
src/bin/pg_dump/pg_backup_db.c | 75 +------
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 278 +------------------------
9 files changed, 350 insertions(+), 341 deletions(-)
create mode 100644 src/bin/pg_dump/connectdb.c
create mode 100644 src/bin/pg_dump/connectdb.h
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..fa795883e9f 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -31,6 +31,7 @@ OBJS = \
compress_lz4.o \
compress_none.o \
compress_zstd.o \
+ connectdb.o \
dumputils.o \
filter.o \
parallel.o \
@@ -50,8 +51,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
new file mode 100644
index 00000000000..3e1fbe98c25
--- /dev/null
+++ b/src/bin/pg_dump/connectdb.c
@@ -0,0 +1,294 @@
+/*-------------------------------------------------------------------------
+ *
+ * connectdb.c
+ * This is a common file connection to the database.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/connectdb.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "connectdb.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+
+/*
+ * ConnectDatabase
+ *
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the 'connstr' is set to a connection string containing
+ * the options used and 'server_version' is set to version so that caller
+ * can use them.
+ */
+PGconn *
+ConnectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version, char *password,
+ char *override_dbname)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 8;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ if (override_dbname)
+ {
+ keywords[i] = "dbname";
+ values[i++] = override_dbname;
+ }
+
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. If requested, remember the options used, in the
+ * form of a connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If requested, then copy server version to out variable. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/*
+ * constructConnStr
+ *
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * executeQuery
+ *
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
diff --git a/src/bin/pg_dump/connectdb.h b/src/bin/pg_dump/connectdb.h
new file mode 100644
index 00000000000..9e1e7ef33d0
--- /dev/null
+++ b/src/bin/pg_dump/connectdb.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * connectdb.h
+ * Common header file for connection to the database.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/connectdb.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONNECTDB_H
+#define CONNECTDB_H
+
+#include "pg_backup.h"
+#include "pg_backup_utils.h"
+
+extern PGconn *ConnectDatabase(const char *dbname, const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version,
+ char *password, char *override_dbname);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+#endif /* CONNECTDB_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..9031737d013 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -6,6 +6,7 @@ pg_dump_common_sources = files(
'compress_lz4.c',
'compress_none.c',
'compress_zstd.c',
+ 'connectdb.c',
'dumputils.c',
'filter.c',
'parallel.c',
@@ -48,6 +49,7 @@ bin_targets += pg_dump
pg_dumpall_sources = files(
+ 'connectdb.c',
'pg_dumpall.c',
)
@@ -67,6 +69,7 @@ bin_targets += pg_dumpall
pg_restore_sources = files(
+ 'connectdb.c',
'pg_restore.c',
)
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 658986de6f8..c68a21027fa 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -293,7 +293,7 @@ typedef void (*SetupWorkerPtrType) (Archive *AH);
* Main archiver interface.
*/
-extern void ConnectDatabase(Archive *AHX,
+extern void ConnectDatabaseAhx(Archive *AHX,
const ConnParams *cparams,
bool isReconnect);
extern void DisconnectDatabase(Archive *AHX);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 82d51c89ac6..3fd2818223c 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -414,7 +414,7 @@ RestoreArchive(Archive *AHX)
AHX->minRemoteVersion = 0;
AHX->maxRemoteVersion = 9999999;
- ConnectDatabase(AHX, &ropt->cparams, false);
+ ConnectDatabaseAhx(AHX, &ropt->cparams, false);
/*
* If we're talking to the DB directly, don't send comments since they
@@ -4437,7 +4437,7 @@ restore_toc_entries_postfork(ArchiveHandle *AH, TocEntry *pending_list)
/*
* Now reconnect the single parent connection.
*/
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
/* re-establish fixed state */
_doSetFixedOutputState(AH);
@@ -5054,7 +5054,7 @@ CloneArchive(ArchiveHandle *AH)
* Connect our new clone object to the database, using the same connection
* parameters used for the original connection.
*/
- ConnectDatabase((Archive *) clone, &clone->public.ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) clone, &clone->public.ropt->cparams, true);
/* re-establish fixed state */
if (AH->mode == archModeRead)
diff --git a/src/bin/pg_dump/pg_backup_db.c b/src/bin/pg_dump/pg_backup_db.c
index 71c55d2466a..227dd963984 100644
--- a/src/bin/pg_dump/pg_backup_db.c
+++ b/src/bin/pg_dump/pg_backup_db.c
@@ -19,6 +19,7 @@
#include "common/connect.h"
#include "common/string.h"
+#include "connectdb.h"
#include "parallel.h"
#include "pg_backup_archiver.h"
#include "pg_backup_db.h"
@@ -86,9 +87,9 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* ArchiveHandle's connCancel, before closing old connection. Otherwise
* an ill-timed SIGINT could try to access a dead connection.
*/
- AH->connection = NULL; /* dodge error check in ConnectDatabase */
+ AH->connection = NULL; /* dodge error check in ConnectDatabaseAhx */
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
PQfinish(oldConn);
}
@@ -105,14 +106,13 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* username never does change, so one savedPassword is sufficient.
*/
void
-ConnectDatabase(Archive *AHX,
+ConnectDatabaseAhx(Archive *AHX,
const ConnParams *cparams,
bool isReconnect)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
trivalue prompt_password;
char *password;
- bool new_pass;
if (AH->connection)
pg_fatal("already connected to a database");
@@ -125,69 +125,10 @@ ConnectDatabase(Archive *AHX,
if (prompt_password == TRI_YES && password == NULL)
password = simple_prompt("Password: ", false);
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- const char *keywords[8];
- const char *values[8];
- int i = 0;
-
- /*
- * If dbname is a connstring, its entries can override the other
- * values obtained from cparams; but in turn, override_dbname can
- * override the dbname component of it.
- */
- keywords[i] = "host";
- values[i++] = cparams->pghost;
- keywords[i] = "port";
- values[i++] = cparams->pgport;
- keywords[i] = "user";
- values[i++] = cparams->username;
- keywords[i] = "password";
- values[i++] = password;
- keywords[i] = "dbname";
- values[i++] = cparams->dbname;
- if (cparams->override_dbname)
- {
- keywords[i] = "dbname";
- values[i++] = cparams->override_dbname;
- }
- keywords[i] = "fallback_application_name";
- values[i++] = progname;
- keywords[i] = NULL;
- values[i++] = NULL;
- Assert(i <= lengthof(keywords));
-
- new_pass = false;
- AH->connection = PQconnectdbParams(keywords, values, true);
-
- if (!AH->connection)
- pg_fatal("could not connect to database");
-
- if (PQstatus(AH->connection) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(AH->connection) &&
- password == NULL &&
- prompt_password != TRI_NO)
- {
- PQfinish(AH->connection);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(AH->connection) == CONNECTION_BAD)
- {
- if (isReconnect)
- pg_fatal("reconnection failed: %s",
- PQerrorMessage(AH->connection));
- else
- pg_fatal("%s",
- PQerrorMessage(AH->connection));
- }
+ AH->connection = ConnectDatabase(cparams->dbname, NULL, cparams->pghost,
+ cparams->pgport, cparams->username,
+ prompt_password, true,
+ progname, NULL, NULL, password, cparams->override_dbname);
/* Start strict; later phases may override this. */
PQclear(ExecuteSqlQueryForSingleRow((Archive *) AH,
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 428ed2d60fc..f81667403dc 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -935,7 +935,7 @@ main(int argc, char **argv)
* Open the database using the Archiver, so it knows about it. Errors mean
* death.
*/
- ConnectDatabase(fout, &dopt.cparams, false);
+ ConnectDatabaseAhx(fout, &dopt.cparams, false);
setup_connection(fout, dumpencoding, dumpsnapshot, use_role);
/*
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 2935cac2c46..455103e38bc 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -24,11 +24,11 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
-#include "pg_backup.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -71,21 +71,14 @@ static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
static char pg_dump_bin[MAXPGPATH];
-static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -126,8 +119,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -487,19 +478,22 @@ main(int argc, char *argv[])
*/
if (pgdb)
{
- conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase(pgdb, connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
}
else
{
- conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase("postgres", connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
- conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ conn = ConnectDatabase("template1", connstr, pghost, pgport, pguser,
+ prompt_password, true,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
{
@@ -1723,256 +1717,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
--
2.39.3
[application/octet-stream] v24_0002_pg_dumpall-with-non-text_format-18th_march.patch (61.0K, 3-v24_0002_pg_dumpall-with-non-text_format-18th_march.patch)
download | inline diff:
From 2138efb24a1c84d182b3ef4ce78ae620d6cef56f Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Wed, 19 Mar 2025 01:30:12 +0530
Subject: [PATCH 2/2] pg_dumpall with directory|tar|custom format and restore
it by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat to restore all databases. If global.dat file is exist in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list (if exist)
for --exclude-database=PATTERN for pg_restore
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
if no db connection, then PATTERN=NAME matching only
for each database, we are cleaning on_exit_nicely_index list.
at the end of restore, we are giving warning with total number of errors (including global.dat,
and each database errors) and for each database, we are printing warning with dbname and total
errors.
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
---
doc/src/sgml/ref/pg_dumpall.sgml | 80 ++-
doc/src/sgml/ref/pg_restore.sgml | 41 +-
src/bin/pg_dump/parallel.c | 11 +-
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 20 +-
src/bin/pg_dump/pg_backup_archiver.h | 3 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 22 +-
src/bin/pg_dump/pg_backup_utils.h | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 280 +++++++--
src/bin/pg_dump/pg_restore.c | 847 ++++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 9 +
src/tools/pgindent/typedefs.list | 2 +
14 files changed, 1246 insertions(+), 78 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index ae5afb3c7d5..c38906a1aac 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster based on specified dump format </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -121,7 +121,83 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
- </para>
+ Note: This option can be omitted only when <option>--format</option> is plain
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index 35140187807..4cf46ea9333 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> database from an
+ archive file created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -37,9 +38,10 @@ PostgreSQL documentation
<title>Description</title>
<para>
- <application>pg_restore</application> is a utility for restoring a
+ <application>pg_restore</application> is a utility for restoring
<productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
database to the state it was in at the time it was saved. The
archive files also allow <application>pg_restore</application> to
@@ -140,6 +142,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from dump of <application>pg_dumpall</application>.
</para>
<para>
@@ -166,6 +170,25 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +338,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..a36d2a5bf84 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -326,11 +326,18 @@ getThreadLocalPQExpBuffer(void)
* pg_dump and pg_restore call this to register the cleanup handler
* as soon as they've created the ArchiveHandle.
*/
-void
+int
on_exit_close_archive(Archive *AHX)
{
shutdown_info.AHX = AHX;
- on_exit_nicely(archive_close_connection, &shutdown_info);
+ return on_exit_nicely(archive_close_connection, &shutdown_info);
+}
+
+void
+replace_on_exit_close_archive(Archive *AHX, int idx)
+{
+ shutdown_info.AHX = AHX;
+ set_on_exit_nicely_entry(archive_close_connection, &shutdown_info, idx);
}
/*
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index c68a21027fa..89459dedc4b 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -311,7 +311,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 3fd2818223c..e22a8810b45 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -85,7 +85,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -337,9 +337,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -456,7 +461,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1292,7 +1297,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1671,7 +1676,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1691,7 +1697,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index a2064f471ed..ae433132435 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -385,7 +385,8 @@ struct _tocEntry
};
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
-extern void on_exit_close_archive(Archive *AHX);
+extern int on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX, int idx);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..59ece2999a8 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -61,14 +61,26 @@ set_dump_section(const char *arg, int *dumpSections)
/* Register a callback to be run when exit_nicely is invoked. */
-void
+int
on_exit_nicely(on_exit_nicely_callback function, void *arg)
{
- if (on_exit_nicely_index >= MAX_ON_EXIT_NICELY)
- pg_fatal("out of on_exit_nicely slots");
- on_exit_nicely_list[on_exit_nicely_index].function = function;
- on_exit_nicely_list[on_exit_nicely_index].arg = arg;
+ set_on_exit_nicely_entry(function, arg, on_exit_nicely_index);
on_exit_nicely_index++;
+
+ return (on_exit_nicely_index - 1);
+}
+
+void
+set_on_exit_nicely_entry(on_exit_nicely_callback function, void *arg, int i)
+{
+ if (i >= MAX_ON_EXIT_NICELY)
+ pg_fatal("out of on_exit_nicely slots");
+
+ if (i > on_exit_nicely_index)
+ pg_fatal("no entry exists on %d index into on_exit_nicely slots", i);
+
+ on_exit_nicely_list[i].function = function;
+ on_exit_nicely_list[i].arg = arg;
}
/*
diff --git a/src/bin/pg_dump/pg_backup_utils.h b/src/bin/pg_dump/pg_backup_utils.h
index ba042016879..bbefdc112f5 100644
--- a/src/bin/pg_dump/pg_backup_utils.h
+++ b/src/bin/pg_dump/pg_backup_utils.h
@@ -28,7 +28,8 @@ typedef void (*on_exit_nicely_callback) (int code, void *arg);
extern const char *progname;
extern void set_dump_section(const char *arg, int *dumpSections);
-extern void on_exit_nicely(on_exit_nicely_callback function, void *arg);
+extern int on_exit_nicely(on_exit_nicely_callback function, void *arg);
+extern void set_on_exit_nicely_entry(on_exit_nicely_callback function, void *arg, int idx);
pg_noreturn extern void exit_nicely(int code);
/* In pg_dump, we modify pg_fatal to call exit_nicely instead of exit */
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index f81667403dc..76f74ba1666 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1188,7 +1188,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 455103e38bc..1a57077986b 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -64,9 +65,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -75,6 +77,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -104,7 +108,7 @@ static int no_subscriptions = 0;
static int no_toast_compression = 0;
static int no_unlogged_table_data = 0;
static int no_role_passwords = 0;
-static int server_version;
+static int server_version;
static int load_via_partition_root = 0;
static int on_conflict_do_nothing = 0;
static int statistics_only = 0;
@@ -143,6 +147,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -189,6 +194,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -238,7 +245,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -266,7 +273,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -415,6 +424,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If non-plain format is specified then we must provide the
+ * file name to create one main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -471,6 +495,33 @@ main(int argc, char *argv[])
if (statistics_only)
appendPQExpBufferStr(pgdumpopts, " --statistics-only");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char toc_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -510,19 +561,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -622,7 +660,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -635,7 +673,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -646,12 +684,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster based on specified dump format.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1555,10 +1595,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1572,7 +1615,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1580,9 +1623,33 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_fatal("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1597,6 +1664,18 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is non-plain dump format, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1615,9 +1694,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1626,19 +1713,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char toc_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(toc_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ toc_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1648,7 +1746,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1657,17 +1756,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is non-plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1812,3 +1930,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if(!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 337e64a8a29..ef164fbe4bb 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,30 +41,77 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/connect.h"
+#include "compress_io.h"
+#include "common/string.h"
+#include "connectdb.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
+#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
+typedef struct SimpleDatabaseOidListCell
+{
+ struct SimpleDatabaseOidListCell *next;
+ Oid db_oid;
+ const char *db_name;
+} SimpleDatabaseOidListCell;
+
+typedef struct SimpleDatabaseOidList
+{
+ SimpleDatabaseOidListCell *head;
+ SimpleDatabaseOidListCell *tail;
+} SimpleDatabaseOidList;
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num);
+static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static void copy_or_print_global_file(const char *outfile, FILE *pfile);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleDatabaseOidList *dbname_oid_list);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname);
+static void simple_string_full_list_delete(SimpleStringList *list);
+static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
+static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list,
+ Oid db_oid, const char *dbname);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
+static int on_exit_index = 0;
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -87,6 +134,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -138,6 +186,7 @@ main(int argc, char **argv)
{"no-statistics", no_argument, &no_statistics, 1},
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -166,7 +215,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -193,11 +242,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -312,6 +364,10 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6:
+ /* list of databases patterns those needs to skip while restoring */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -339,6 +395,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -420,6 +483,108 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL && !IsFileExistsInDirectory(inputFileSpec, "toc.dat") &&
+ IsFileExistsInDirectory(inputFileSpec, "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql commands. */
+
+ /*
+ * User is suggested to use single database dump for --list option.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring multiple databases by archive of pg_dumpall");
+
+ /*
+ * To restore multiple databases, -C (create database) option should be specified.
+ * Even there is single database in dump, report error because it might be possible
+ * that database hasn't created so better we report error.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("-C/--create option should be specified when restoring multiple databases by archive of pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("If db is already created and dump has single db dump, then use particular dump file.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Connect to database to execute global sql commands from global.dat file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ /* Open global.dat file and execute/append all the global sql commands. */
+ n_errors = process_global_sql_commands(conn, inputFileSpec,
+ opts->filename);
+
+ if (conn)
+ PQfinish(conn);
+
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ n_errors = restoreAllDatabases(conn, inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_full_list_delete(&db_exclude_patterns);
+ }
+ else /* process if global.dat file does not exist. */
+ {
+ if (db_exclude_patterns.head != NULL)
+ pg_fatal("option --exclude-database can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ n_errors = restoreOneDatabase(inputFileSpec, opts, numWorkers, false, 0);
+ }
+
+ on_exit_index = 0; /* Reset index. */
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -428,8 +593,14 @@ main(int argc, char **argv)
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
* it's still NULL, the cleanup function will just be a no-op.
+ * If we are restoring multiple databases, then save index of exit_nicely
+ * so that we can use same slot for all the databases as we already closed
+ * the previous archive by CloseArchive.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_index = on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH, on_exit_index);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -449,25 +620,22 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n"
+ "If archive is created by pg_dumpall, then restores multiple databases also. \n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -485,6 +653,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -497,6 +666,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -534,8 +704,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be combined\n"
+ "and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -640,3 +810,648 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * IsFileExistsInDirectory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+IsFileExistsInDirectory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.dat file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+ReadOneStatement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+ int m;
+
+ StringInfoData q;
+ initStringInfo(&q);
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != '\n' && c != ';')
+ {
+ appendStringInfoChar(inBuf, (char) c);
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != ';' && c != '\n')
+ appendStringInfoChar(inBuf, (char) c);
+ else
+ break;
+ }
+ }
+
+ if (c == '\'' || c == '"')
+ {
+ appendStringInfoChar(&q, (char) c);
+ m = c;
+
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(&q, (char) c);
+
+ if(c == m)
+ {
+ appendStringInfoString(inBuf, q.data);
+ resetStringInfo(&q);
+ break;
+ }
+ }
+ }
+
+ if (c == ';')
+ {
+ appendStringInfoChar(inBuf, (char) ';');
+ break;
+ }
+
+ if (c == '\n')
+ appendStringInfoChar(inBuf, (char) '\n');
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will remove entries from dbname_oid_list that pattern matching any
+ * in the db_exclude_patterns list. dbname_oid_list maybe inplace modified.
+ *
+ * returns, number of database will be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
+ SimpleDatabaseOidListCell *dboidprecell = NULL;
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ /* Return 0 if there is no database to restore. */
+ if (dboid_cell == NULL)
+ return 0;
+
+ query = createPQExpBuffer();
+
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ while (dboid_cell != NULL)
+ {
+ bool skip_db_restore = false;
+ SimpleDatabaseOidListCell *next = dboid_cell->next;
+
+ for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ {
+ /*
+ * the construct pattern matching query:
+ * SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE
+ * pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from the
+ * dbname_oid_list, which is initially extracted from the map.dat
+ * file located in the backup directory. that's why we need
+ * quote_literal_cstr.
+ *
+ * If no db connection, then consider PATTERN as NAME.
+ */
+ if (pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0)
+ skip_db_restore = true;
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, celldb->val, false,
+ false, NULL, quote_literal_cstr(dboid_cell->db_name),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ celldb->val);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database \"%s\" is matching with exclude pattern: \"%s\"", dboid_cell->db_name, celldb->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /* Increment count if database needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
+ simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ }
+ else
+ {
+ count_db++;
+ dboidprecell = dboid_cell;
+ }
+
+ /* Process next dbname from dbname list. */
+ dboid_cell = next;
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ /*
+ * If there is only global.dat file in dump, then return from here as there
+ * is no database to restore.
+ */
+ if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ {
+ pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid = InvalidOid;
+ char db_oid_str[MAXPGPATH + 1] = {'\0'};
+ char dbname[MAXPGPATH + 1] = {'\0'};
+
+ /* Extract dboid. */
+ sscanf(line, "%u" , &db_oid);
+ sscanf(line, "%s" , db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found database \"%s\" (OID: %u) in map.dat file while restoring.", dbname, db_oid);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
+ SimpleDatabaseOidListCell *dboid_cell;
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /*
+ * If map.dat has no entry, return from here after processing
+ * global.dat file.
+ */
+ if (dbname_oid_list.head == NULL)
+ return process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+
+ /*
+ * processing pg_retsore --exclude-database=PATTERN/NAME if no connection.
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ n_errors_total = process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL)
+ {
+ pg_log_info("no database needs to restore out of %d databases", num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * Till now, we made a list of databases, those needs to be restored
+ * after skipping names of exclude-database. Now we can launch parallel
+ * workers to restore these databases.
+ */
+ dboid_cell = dbname_oid_list.head;
+
+ while(dboid_cell != NULL)
+ {
+ char subdirpath[MAXPGPATH];
+ int n_errors;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored into
+ * already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+
+ pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+
+ /* Restore single database. */
+ n_errors = restoreOneDatabase(subdirpath, opts, numWorkers, true, count);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dboid_cell->db_name, n_errors);
+ }
+
+ dboid_cell = dboid_cell->next;
+ count++;
+ }
+
+ /* Log number of processed databases.*/
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+
+ return n_errors_total;
+}
+
+/*
+ * process_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ *
+ * returns the number of errors while processing global.dat
+ */
+static int
+process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+ int n_errors = 0;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ copy_or_print_global_file(outfile, pfile);
+ return 0;
+ }
+
+ /* Init sqlstatement to append commands. */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements. */
+ while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ {
+ pg_log_info("executing query: %s", sqlstatement.data);
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ n_errors++;
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ /* Print a summary of ignored errors during global.dat. */
+ if (n_errors)
+ pg_log_warning("errors ignored on global.dat file restore: %d", n_errors);
+
+ fclose(pfile);
+
+ return n_errors;
+}
+
+/*
+ * copy_or_print_global_file
+ *
+ * This will copy global.dat file into out file. If "-" is used as outfile,
+ * then print commands to the stdout.
+ */
+static void
+copy_or_print_global_file(const char *outfile, FILE *pfile)
+{
+ char out_file_path[MAXPGPATH];
+ FILE *OPF;
+ int c;
+
+ /* "-" is used for stdout. */
+ if (strcmp(outfile, "-") == 0)
+ OPF = stdout;
+ else
+ {
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ OPF = fopen(out_file_path, PG_BINARY_W);
+
+ if (OPF == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", outfile);
+ }
+ }
+
+ /* Append global.dat into out file or print to the stdout. */
+ while ((c = fgetc(pfile)) != EOF)
+ fputc(c, OPF);
+
+ fclose(pfile);
+
+ /* Close out file. */
+ if (strcmp(outfile, "-") != 0)
+ fclose(OPF);
+}
+
+/*
+ * simple_db_oid_list_append
+ *
+ * appends a node to the list in the end.
+ */
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname)
+{
+ SimpleDatabaseOidListCell *cell;
+
+ cell = pg_malloc_object(SimpleDatabaseOidListCell);
+
+ cell->next = NULL;
+ cell->db_oid = db_oid;
+ cell->db_name = pg_strdup(dbname);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * simple_db_oid_full_list_delete
+ *
+ * delete all cell from dbname and dboid list.
+ */
+static void
+simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
+{
+ SimpleDatabaseOidListCell *cell = list->head;
+ SimpleDatabaseOidListCell *nextcell = NULL;
+
+ while (cell)
+ {
+ nextcell = cell->next;
+ pfree (cell);
+ cell = nextcell;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_string_full_list_delete
+ *
+ * delete all cell from string list.
+ */
+static void
+simple_string_full_list_delete(SimpleStringList *list)
+{
+ SimpleStringListCell *cell = list->head;
+ SimpleStringListCell *cellnext = NULL;
+
+ while (cell)
+ {
+ cellnext = cell->next;
+ pfree(cell);
+ cell = cellnext;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_db_oid_list_delete
+ *
+ * delete cell from database and oid list.
+ */
+static void
+simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev)
+{
+ if (prev == NULL)
+ {
+ list->head = cell->next;
+ pfree(cell);
+ }
+ else
+ {
+ prev->next = cell->next;
+ pfree(cell);
+ }
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr
+ *
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..0bbcdbe84a7
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,11 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +249,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index c04a47cf222..be75739e995 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2722,6 +2722,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.39.3
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-03-27 21:15 ` Andrew Dunstan <[email protected]>
2025-03-28 22:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Dunstan @ 2025-03-27 21:15 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: jian he <[email protected]>; Álvaro Herrera <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-03-19 We 2:41 AM, Mahendra Singh Thalor wrote:
> On Wed, 12 Mar 2025 at 21:18, Andrew Dunstan <[email protected]> wrote:
>>
>> On 2025-03-12 We 3:03 AM, jian he wrote:
>>> On Wed, Mar 12, 2025 at 1:06 AM Álvaro Herrera <[email protected]> wrote:
>>>> Hello,
>>>>
>>>> On 2025-Mar-11, Mahendra Singh Thalor wrote:
>>>>
>>>>> In map.dat file, I tried to fix this issue by adding number of characters
>>>>> in dbname but as per code comments, as of now, we are not supporting \n\r
>>>>> in dbnames so i removed handling.
>>>>> I will do some more study to fix this issue.
>>>> Yeah, I think this is saying that you should not consider the contents
>>>> of map.dat as a shell string. After all, you're not going to _execute_
>>>> that file via the shell.
>>>>
>>>> Maybe for map.dat you need to escape such characters somehow, so that
>>>> they don't appear as literal newlines/carriage returns.
>>>>
>>> I am confused.
>>> currently pg_dumpall plain format will abort when encountering dbname
>>> containing newline.
>>> the left dumped plain file does not contain all the cluster databases data.
>>>
>>>
>>> if pg_dumpall non-text format aborts earlier,
>>> it's aligned with pg_dumpall plain format?
>>> it's also an improvement since aborts earlier, nothing will be dumped?
>>>
>>>
>>> am i missing something?
>>>
>>>
>> I think we should fix that.
>>
>> But for the current proposal, Álvaro and I were talking this morning,
>> and we thought the simplest thing here would be to have the one line
>> format and escape NL/CRs in the database name.
>>
>>
>> cheers
>>
> Okay. As per discussions, we will keep one line entry for each
> database into map.file.
>
> Thanks all for feedback and review.
>
> Here, I am attaching updated patches for review and testing. These
> patches can be applied on commit a6524105d20b.
I'm working through this patch set with a view to committing it.
Attached is some cleanup which is where I got to today, although there
is more to do. One thing I am wondering is why not put the
SimpleDatabaseOidList stuff in fe_utils/simle_list.{c,h} ? That's where
all the similar stuff belongs, and it feels strange to have this inline
in pg_restore.c. (I also don't like the name much - SimpleOidStringList
or maybe SimpleOidPlusStringList might be better).
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index a3dcc585ace..6aab1bfe831 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -434,13 +434,13 @@ main(int argc, char *argv[])
archDumpFormat = parseDumpFormat(formatName);
/*
- * If non-plain format is specified then we must provide the
- * file name to create one main directory.
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
*/
if (archDumpFormat != archNull &&
(!filename || strcmp(filename, "") == 0))
{
- pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit_nicely(1);
}
@@ -513,14 +513,14 @@ main(int argc, char *argv[])
*/
if (archDumpFormat != archNull)
{
- char toc_path[MAXPGPATH];
+ char global_path[MAXPGPATH];
/* Create new directory or accept the empty existing directory. */
create_or_open_dir(filename);
- snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
- OPF = fopen(toc_path, PG_BINARY_W);
+ OPF = fopen(global_path, PG_BINARY_W);
if (!OPF)
pg_fatal("could not open global.dat file: %s", strerror(errno));
}
@@ -1680,7 +1680,7 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
}
/*
- * If this is non-plain dump format, then append dboid and dbname to
+ * If this is not a plain format dump, then append dboid and dbname to
* the map.dat file.
*/
if (archDumpFormat != archNull)
@@ -1688,7 +1688,7 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
/* Put one line entry for dboid and dbname in map file. */
- fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ fprintf(map_file, "%s %s\n", oid, dbname);
}
pg_log_info("dumping database \"%s\"", dbname);
@@ -1734,17 +1734,17 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
if (filename)
{
- char toc_path[MAXPGPATH];
+ char global_path[MAXPGPATH];
if (archDumpFormat != archNull)
- snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
else
- snprintf(toc_path, MAXPGPATH, "%s", filename);
+ snprintf(global_path, MAXPGPATH, "%s", filename);
- OPF = fopen(toc_path, PG_BINARY_A);
+ OPF = fopen(global_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- toc_path);
+ global_path);
}
}
@@ -1772,7 +1772,7 @@ runPgDump(const char *dbname, const char *create_opts, char *dbfile,
initPQExpBuffer(&cmd);
/*
- * If this is non-plain format dump, then append file name and dump
+ * If this is not a plain format dump, then append file name and dump
* format to the pg_dump command to get archive dump.
*/
if (archDumpFormat != archNull)
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index e4093427e2f..91602a2e37b 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -46,8 +46,6 @@
#include <termios.h>
#endif
-#include "common/connect.h"
-#include "compress_io.h"
#include "common/string.h"
#include "connectdb.h"
#include "fe_utils/option_utils.h"
@@ -55,7 +53,6 @@
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
-#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
typedef struct SimpleDatabaseOidListCell
@@ -73,10 +70,10 @@ typedef struct SimpleDatabaseOidList
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
-static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static bool file_exists_in_directory(const char *dir, const char *filename);
static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
int numWorkers, bool append_data, int num);
-static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int read_one_statement(StringInfo inBuf, FILE *pfile);
static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
static int process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
@@ -89,7 +86,6 @@ static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
SimpleDatabaseOidList *dbname_oid_list);
static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
const char *dbname);
-static void simple_string_full_list_delete(SimpleStringList *list);
static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
SimpleDatabaseOidListCell *cell,
@@ -521,8 +517,8 @@ main(int argc, char **argv)
* databases from map.dat(if exist) file list and skip restoring for
* --exclude-database patterns.
*/
- if (inputFileSpec != NULL && !IsFileExistsInDirectory(inputFileSpec, "toc.dat") &&
- IsFileExistsInDirectory(inputFileSpec, "global.dat"))
+ if (inputFileSpec != NULL && !file_exists_in_directory(inputFileSpec, "toc.dat") &&
+ file_exists_in_directory(inputFileSpec, "global.dat"))
{
PGconn *conn = NULL; /* Connection to restore global sql commands. */
@@ -578,7 +574,7 @@ main(int argc, char **argv)
}
/* Free db pattern list. */
- simple_string_full_list_delete(&db_exclude_patterns);
+ simple_string_list_destroy(&db_exclude_patterns);
}
else /* process if global.dat file does not exist. */
{
@@ -847,12 +843,12 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
}
/*
- * IsFileExistsInDirectory
+ * file_exists_in_directory
*
* Returns true if file exist in current directory.
*/
static bool
-IsFileExistsInDirectory(const char *dir, const char *filename)
+file_exists_in_directory(const char *dir, const char *filename)
{
struct stat st;
char buf[MAXPGPATH];
@@ -864,7 +860,7 @@ IsFileExistsInDirectory(const char *dir, const char *filename)
}
/*
- * ReadOneStatement
+ * read_one_statement
*
* This will start reading from passed file pointer using fgetc and read till
* semicolon(sql statement terminator for global.dat file)
@@ -873,7 +869,7 @@ IsFileExistsInDirectory(const char *dir, const char *filename)
*/
static int
-ReadOneStatement(StringInfo inBuf, FILE *pfile)
+read_one_statement(StringInfo inBuf, FILE *pfile)
{
int c; /* character read from getc() */
int m;
@@ -1064,7 +1060,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *d
* If there is only global.dat file in dump, then return from here as there
* is no database to restore.
*/
- if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
{
pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
return 0;
@@ -1281,7 +1277,7 @@ process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *o
initStringInfo(&sqlstatement);
/* Process file till EOF and execute sql statements. */
- while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ while (read_one_statement(&sqlstatement, pfile) != EOF)
{
pg_log_info("executing query: %s", sqlstatement.data);
result = PQexec(conn, sqlstatement.data);
@@ -1393,28 +1389,6 @@ simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
list->tail = NULL;
}
-/*
- * simple_string_full_list_delete
- *
- * delete all cell from string list.
- */
-static void
-simple_string_full_list_delete(SimpleStringList *list)
-{
- SimpleStringListCell *cell = list->head;
- SimpleStringListCell *cellnext = NULL;
-
- while (cell)
- {
- cellnext = cell->next;
- pfree(cell);
- cell = cellnext;
- }
-
- list->head = NULL;
- list->tail = NULL;
-}
-
/*
* simple_db_oid_list_delete
*
Attachments:
[text/plain] dumpall_cleanup.patch-noci (8.0K, 2-dumpall_cleanup.patch-noci)
download | inline diff:
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index a3dcc585ace..6aab1bfe831 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -434,13 +434,13 @@ main(int argc, char *argv[])
archDumpFormat = parseDumpFormat(formatName);
/*
- * If non-plain format is specified then we must provide the
- * file name to create one main directory.
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
*/
if (archDumpFormat != archNull &&
(!filename || strcmp(filename, "") == 0))
{
- pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit_nicely(1);
}
@@ -513,14 +513,14 @@ main(int argc, char *argv[])
*/
if (archDumpFormat != archNull)
{
- char toc_path[MAXPGPATH];
+ char global_path[MAXPGPATH];
/* Create new directory or accept the empty existing directory. */
create_or_open_dir(filename);
- snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
- OPF = fopen(toc_path, PG_BINARY_W);
+ OPF = fopen(global_path, PG_BINARY_W);
if (!OPF)
pg_fatal("could not open global.dat file: %s", strerror(errno));
}
@@ -1680,7 +1680,7 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
}
/*
- * If this is non-plain dump format, then append dboid and dbname to
+ * If this is not a plain format dump, then append dboid and dbname to
* the map.dat file.
*/
if (archDumpFormat != archNull)
@@ -1688,7 +1688,7 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
/* Put one line entry for dboid and dbname in map file. */
- fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ fprintf(map_file, "%s %s\n", oid, dbname);
}
pg_log_info("dumping database \"%s\"", dbname);
@@ -1734,17 +1734,17 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
if (filename)
{
- char toc_path[MAXPGPATH];
+ char global_path[MAXPGPATH];
if (archDumpFormat != archNull)
- snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
else
- snprintf(toc_path, MAXPGPATH, "%s", filename);
+ snprintf(global_path, MAXPGPATH, "%s", filename);
- OPF = fopen(toc_path, PG_BINARY_A);
+ OPF = fopen(global_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- toc_path);
+ global_path);
}
}
@@ -1772,7 +1772,7 @@ runPgDump(const char *dbname, const char *create_opts, char *dbfile,
initPQExpBuffer(&cmd);
/*
- * If this is non-plain format dump, then append file name and dump
+ * If this is not a plain format dump, then append file name and dump
* format to the pg_dump command to get archive dump.
*/
if (archDumpFormat != archNull)
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index e4093427e2f..91602a2e37b 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -46,8 +46,6 @@
#include <termios.h>
#endif
-#include "common/connect.h"
-#include "compress_io.h"
#include "common/string.h"
#include "connectdb.h"
#include "fe_utils/option_utils.h"
@@ -55,7 +53,6 @@
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
-#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
typedef struct SimpleDatabaseOidListCell
@@ -73,10 +70,10 @@ typedef struct SimpleDatabaseOidList
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
-static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static bool file_exists_in_directory(const char *dir, const char *filename);
static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
int numWorkers, bool append_data, int num);
-static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int read_one_statement(StringInfo inBuf, FILE *pfile);
static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
static int process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
@@ -89,7 +86,6 @@ static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
SimpleDatabaseOidList *dbname_oid_list);
static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
const char *dbname);
-static void simple_string_full_list_delete(SimpleStringList *list);
static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
SimpleDatabaseOidListCell *cell,
@@ -521,8 +517,8 @@ main(int argc, char **argv)
* databases from map.dat(if exist) file list and skip restoring for
* --exclude-database patterns.
*/
- if (inputFileSpec != NULL && !IsFileExistsInDirectory(inputFileSpec, "toc.dat") &&
- IsFileExistsInDirectory(inputFileSpec, "global.dat"))
+ if (inputFileSpec != NULL && !file_exists_in_directory(inputFileSpec, "toc.dat") &&
+ file_exists_in_directory(inputFileSpec, "global.dat"))
{
PGconn *conn = NULL; /* Connection to restore global sql commands. */
@@ -578,7 +574,7 @@ main(int argc, char **argv)
}
/* Free db pattern list. */
- simple_string_full_list_delete(&db_exclude_patterns);
+ simple_string_list_destroy(&db_exclude_patterns);
}
else /* process if global.dat file does not exist. */
{
@@ -847,12 +843,12 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
}
/*
- * IsFileExistsInDirectory
+ * file_exists_in_directory
*
* Returns true if file exist in current directory.
*/
static bool
-IsFileExistsInDirectory(const char *dir, const char *filename)
+file_exists_in_directory(const char *dir, const char *filename)
{
struct stat st;
char buf[MAXPGPATH];
@@ -864,7 +860,7 @@ IsFileExistsInDirectory(const char *dir, const char *filename)
}
/*
- * ReadOneStatement
+ * read_one_statement
*
* This will start reading from passed file pointer using fgetc and read till
* semicolon(sql statement terminator for global.dat file)
@@ -873,7 +869,7 @@ IsFileExistsInDirectory(const char *dir, const char *filename)
*/
static int
-ReadOneStatement(StringInfo inBuf, FILE *pfile)
+read_one_statement(StringInfo inBuf, FILE *pfile)
{
int c; /* character read from getc() */
int m;
@@ -1064,7 +1060,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *d
* If there is only global.dat file in dump, then return from here as there
* is no database to restore.
*/
- if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
{
pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
return 0;
@@ -1281,7 +1277,7 @@ process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *o
initStringInfo(&sqlstatement);
/* Process file till EOF and execute sql statements. */
- while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ while (read_one_statement(&sqlstatement, pfile) != EOF)
{
pg_log_info("executing query: %s", sqlstatement.data);
result = PQexec(conn, sqlstatement.data);
@@ -1393,28 +1389,6 @@ simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
list->tail = NULL;
}
-/*
- * simple_string_full_list_delete
- *
- * delete all cell from string list.
- */
-static void
-simple_string_full_list_delete(SimpleStringList *list)
-{
- SimpleStringListCell *cell = list->head;
- SimpleStringListCell *cellnext = NULL;
-
- while (cell)
- {
- cellnext = cell->next;
- pfree(cell);
- cell = cellnext;
- }
-
- list->head = NULL;
- list->tail = NULL;
-}
-
/*
* simple_db_oid_list_delete
*
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-27 21:15 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
@ 2025-03-28 22:20 ` Andrew Dunstan <[email protected]>
2025-03-29 05:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Dunstan @ 2025-03-28 22:20 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: jian he <[email protected]>; Álvaro Herrera <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-03-27 Th 5:15 PM, Andrew Dunstan wrote:
>
> On 2025-03-19 We 2:41 AM, Mahendra Singh Thalor wrote:
>> On Wed, 12 Mar 2025 at 21:18, Andrew Dunstan <[email protected]>
>> wrote:
>>>
>>> On 2025-03-12 We 3:03 AM, jian he wrote:
>>>> On Wed, Mar 12, 2025 at 1:06 AM Álvaro Herrera
>>>> <[email protected]> wrote:
>>>>> Hello,
>>>>>
>>>>> On 2025-Mar-11, Mahendra Singh Thalor wrote:
>>>>>
>>>>>> In map.dat file, I tried to fix this issue by adding number of
>>>>>> characters
>>>>>> in dbname but as per code comments, as of now, we are not
>>>>>> supporting \n\r
>>>>>> in dbnames so i removed handling.
>>>>>> I will do some more study to fix this issue.
>>>>> Yeah, I think this is saying that you should not consider the
>>>>> contents
>>>>> of map.dat as a shell string. After all, you're not going to
>>>>> _execute_
>>>>> that file via the shell.
>>>>>
>>>>> Maybe for map.dat you need to escape such characters somehow, so that
>>>>> they don't appear as literal newlines/carriage returns.
>>>>>
>>>> I am confused.
>>>> currently pg_dumpall plain format will abort when encountering dbname
>>>> containing newline.
>>>> the left dumped plain file does not contain all the cluster
>>>> databases data.
>>>>
>>>>
>>>> if pg_dumpall non-text format aborts earlier,
>>>> it's aligned with pg_dumpall plain format?
>>>> it's also an improvement since aborts earlier, nothing will be dumped?
>>>>
>>>>
>>>> am i missing something?
>>>>
>>>>
>>> I think we should fix that.
>>>
>>> But for the current proposal, Álvaro and I were talking this morning,
>>> and we thought the simplest thing here would be to have the one line
>>> format and escape NL/CRs in the database name.
>>>
>>>
>>> cheers
>>>
>> Okay. As per discussions, we will keep one line entry for each
>> database into map.file.
>>
>> Thanks all for feedback and review.
>>
>> Here, I am attaching updated patches for review and testing. These
>> patches can be applied on commit a6524105d20b.
>
>
>
> I'm working through this patch set with a view to committing it.
> Attached is some cleanup which is where I got to today, although there
> is more to do. One thing I am wondering is why not put the
> SimpleDatabaseOidList stuff in fe_utils/simle_list.{c,h} ? That's
> where all the similar stuff belongs, and it feels strange to have this
> inline in pg_restore.c. (I also don't like the name much -
> SimpleOidStringList or maybe SimpleOidPlusStringList might be better).
>
>
>
OK, I have done that, so here is the result. The first two are you
original patches. patch 3 adds the new list type to fe-utils, and patch
4 contains my cleanups and use of the new list type. Apart from some
relatively minor cleanup, the one thing I would like to change is how
dumps are named. If we are producing tar or custom format dumps, I think
the file names should reflect that (oid.dmp and oid.tar rather than a
bare oid as the filename), and pg_restore should look for those. I'm
going to work on that tomorrow - I don't think it will be terribly
difficult.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
Attachments:
[text/x-patch] 0001-move-common-code-related-to-connection-to-new-the-fi.patch (26.2K, 2-0001-move-common-code-related-to-connection-to-new-the-fi.patch)
download | inline diff:
From 7105ed4a08b0c2b4d30e7b3eedb6c94882eb7421 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Wed, 19 Mar 2025 01:18:46 +0530
Subject: [PATCH 1/4] move common code related to connection to new the file
ConnectDatabase is used by both pg_dumpall, pg_restore
and pg_dump so move common code to new file.
new file name: connectdb.c
---
src/bin/pg_dump/Makefile | 5 +-
src/bin/pg_dump/connectdb.c | 294 +++++++++++++++++++++++++++
src/bin/pg_dump/connectdb.h | 26 +++
src/bin/pg_dump/meson.build | 3 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 6 +-
src/bin/pg_dump/pg_backup_db.c | 75 +------
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 278 +------------------------
9 files changed, 350 insertions(+), 341 deletions(-)
create mode 100644 src/bin/pg_dump/connectdb.c
create mode 100644 src/bin/pg_dump/connectdb.h
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..fa795883e9f 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -31,6 +31,7 @@ OBJS = \
compress_lz4.o \
compress_none.o \
compress_zstd.o \
+ connectdb.o \
dumputils.o \
filter.o \
parallel.o \
@@ -50,8 +51,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
new file mode 100644
index 00000000000..3e1fbe98c25
--- /dev/null
+++ b/src/bin/pg_dump/connectdb.c
@@ -0,0 +1,294 @@
+/*-------------------------------------------------------------------------
+ *
+ * connectdb.c
+ * This is a common file connection to the database.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/connectdb.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "connectdb.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+
+/*
+ * ConnectDatabase
+ *
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the 'connstr' is set to a connection string containing
+ * the options used and 'server_version' is set to version so that caller
+ * can use them.
+ */
+PGconn *
+ConnectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version, char *password,
+ char *override_dbname)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 8;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ if (override_dbname)
+ {
+ keywords[i] = "dbname";
+ values[i++] = override_dbname;
+ }
+
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. If requested, remember the options used, in the
+ * form of a connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If requested, then copy server version to out variable. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/*
+ * constructConnStr
+ *
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * executeQuery
+ *
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
diff --git a/src/bin/pg_dump/connectdb.h b/src/bin/pg_dump/connectdb.h
new file mode 100644
index 00000000000..9e1e7ef33d0
--- /dev/null
+++ b/src/bin/pg_dump/connectdb.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * connectdb.h
+ * Common header file for connection to the database.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/connectdb.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONNECTDB_H
+#define CONNECTDB_H
+
+#include "pg_backup.h"
+#include "pg_backup_utils.h"
+
+extern PGconn *ConnectDatabase(const char *dbname, const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version,
+ char *password, char *override_dbname);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+#endif /* CONNECTDB_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..9031737d013 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -6,6 +6,7 @@ pg_dump_common_sources = files(
'compress_lz4.c',
'compress_none.c',
'compress_zstd.c',
+ 'connectdb.c',
'dumputils.c',
'filter.c',
'parallel.c',
@@ -48,6 +49,7 @@ bin_targets += pg_dump
pg_dumpall_sources = files(
+ 'connectdb.c',
'pg_dumpall.c',
)
@@ -67,6 +69,7 @@ bin_targets += pg_dumpall
pg_restore_sources = files(
+ 'connectdb.c',
'pg_restore.c',
)
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 658986de6f8..c68a21027fa 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -293,7 +293,7 @@ typedef void (*SetupWorkerPtrType) (Archive *AH);
* Main archiver interface.
*/
-extern void ConnectDatabase(Archive *AHX,
+extern void ConnectDatabaseAhx(Archive *AHX,
const ConnParams *cparams,
bool isReconnect);
extern void DisconnectDatabase(Archive *AHX);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 82d51c89ac6..3fd2818223c 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -414,7 +414,7 @@ RestoreArchive(Archive *AHX)
AHX->minRemoteVersion = 0;
AHX->maxRemoteVersion = 9999999;
- ConnectDatabase(AHX, &ropt->cparams, false);
+ ConnectDatabaseAhx(AHX, &ropt->cparams, false);
/*
* If we're talking to the DB directly, don't send comments since they
@@ -4437,7 +4437,7 @@ restore_toc_entries_postfork(ArchiveHandle *AH, TocEntry *pending_list)
/*
* Now reconnect the single parent connection.
*/
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
/* re-establish fixed state */
_doSetFixedOutputState(AH);
@@ -5054,7 +5054,7 @@ CloneArchive(ArchiveHandle *AH)
* Connect our new clone object to the database, using the same connection
* parameters used for the original connection.
*/
- ConnectDatabase((Archive *) clone, &clone->public.ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) clone, &clone->public.ropt->cparams, true);
/* re-establish fixed state */
if (AH->mode == archModeRead)
diff --git a/src/bin/pg_dump/pg_backup_db.c b/src/bin/pg_dump/pg_backup_db.c
index 71c55d2466a..227dd963984 100644
--- a/src/bin/pg_dump/pg_backup_db.c
+++ b/src/bin/pg_dump/pg_backup_db.c
@@ -19,6 +19,7 @@
#include "common/connect.h"
#include "common/string.h"
+#include "connectdb.h"
#include "parallel.h"
#include "pg_backup_archiver.h"
#include "pg_backup_db.h"
@@ -86,9 +87,9 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* ArchiveHandle's connCancel, before closing old connection. Otherwise
* an ill-timed SIGINT could try to access a dead connection.
*/
- AH->connection = NULL; /* dodge error check in ConnectDatabase */
+ AH->connection = NULL; /* dodge error check in ConnectDatabaseAhx */
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
PQfinish(oldConn);
}
@@ -105,14 +106,13 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* username never does change, so one savedPassword is sufficient.
*/
void
-ConnectDatabase(Archive *AHX,
+ConnectDatabaseAhx(Archive *AHX,
const ConnParams *cparams,
bool isReconnect)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
trivalue prompt_password;
char *password;
- bool new_pass;
if (AH->connection)
pg_fatal("already connected to a database");
@@ -125,69 +125,10 @@ ConnectDatabase(Archive *AHX,
if (prompt_password == TRI_YES && password == NULL)
password = simple_prompt("Password: ", false);
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- const char *keywords[8];
- const char *values[8];
- int i = 0;
-
- /*
- * If dbname is a connstring, its entries can override the other
- * values obtained from cparams; but in turn, override_dbname can
- * override the dbname component of it.
- */
- keywords[i] = "host";
- values[i++] = cparams->pghost;
- keywords[i] = "port";
- values[i++] = cparams->pgport;
- keywords[i] = "user";
- values[i++] = cparams->username;
- keywords[i] = "password";
- values[i++] = password;
- keywords[i] = "dbname";
- values[i++] = cparams->dbname;
- if (cparams->override_dbname)
- {
- keywords[i] = "dbname";
- values[i++] = cparams->override_dbname;
- }
- keywords[i] = "fallback_application_name";
- values[i++] = progname;
- keywords[i] = NULL;
- values[i++] = NULL;
- Assert(i <= lengthof(keywords));
-
- new_pass = false;
- AH->connection = PQconnectdbParams(keywords, values, true);
-
- if (!AH->connection)
- pg_fatal("could not connect to database");
-
- if (PQstatus(AH->connection) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(AH->connection) &&
- password == NULL &&
- prompt_password != TRI_NO)
- {
- PQfinish(AH->connection);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(AH->connection) == CONNECTION_BAD)
- {
- if (isReconnect)
- pg_fatal("reconnection failed: %s",
- PQerrorMessage(AH->connection));
- else
- pg_fatal("%s",
- PQerrorMessage(AH->connection));
- }
+ AH->connection = ConnectDatabase(cparams->dbname, NULL, cparams->pghost,
+ cparams->pgport, cparams->username,
+ prompt_password, true,
+ progname, NULL, NULL, password, cparams->override_dbname);
/* Start strict; later phases may override this. */
PQclear(ExecuteSqlQueryForSingleRow((Archive *) AH,
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e41e645f649..015a434fc0b 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -966,7 +966,7 @@ main(int argc, char **argv)
* Open the database using the Archiver, so it knows about it. Errors mean
* death.
*/
- ConnectDatabase(fout, &dopt.cparams, false);
+ ConnectDatabaseAhx(fout, &dopt.cparams, false);
setup_connection(fout, dumpencoding, dumpsnapshot, use_role);
/*
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 2ea574b0f06..573a8b61a45 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -24,11 +24,11 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
-#include "pg_backup.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -71,21 +71,14 @@ static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
static char pg_dump_bin[MAXPGPATH];
-static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -129,8 +122,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -499,19 +490,22 @@ main(int argc, char *argv[])
*/
if (pgdb)
{
- conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase(pgdb, connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
}
else
{
- conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase("postgres", connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
- conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ conn = ConnectDatabase("template1", connstr, pghost, pgport, pguser,
+ prompt_password, true,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
{
@@ -1738,256 +1732,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
--
2.34.1
[text/x-patch] 0002-pg_dumpall-with-directory-tar-custom-format-and-rest.patch (60.8K, 3-0002-pg_dumpall-with-directory-tar-custom-format-and-rest.patch)
download | inline diff:
From e3e3d4fc316b22202b7369947e00312e3acd92e2 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Wed, 19 Mar 2025 01:30:12 +0530
Subject: [PATCH 2/4] pg_dumpall with directory|tar|custom format and restore
it by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat to restore all databases. If global.dat file is exist in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list (if exist)
for --exclude-database=PATTERN for pg_restore
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
if no db connection, then PATTERN=NAME matching only
for each database, we are cleaning on_exit_nicely_index list.
at the end of restore, we are giving warning with total number of errors (including global.dat,
and each database errors) and for each database, we are printing warning with dbname and total
errors.
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
---
doc/src/sgml/ref/pg_dumpall.sgml | 78 ++-
doc/src/sgml/ref/pg_restore.sgml | 41 +-
src/bin/pg_dump/parallel.c | 11 +-
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 20 +-
src/bin/pg_dump/pg_backup_archiver.h | 3 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 22 +-
src/bin/pg_dump/pg_backup_utils.h | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 278 +++++++--
src/bin/pg_dump/pg_restore.c | 847 ++++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 9 +
src/tools/pgindent/typedefs.list | 2 +
14 files changed, 1244 insertions(+), 76 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 765b30a3a66..82ea2028469 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster based on specified dump format </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -121,10 +121,86 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can be omitted only when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index c840a807ae9..f0a24134595 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> database from an
+ archive file created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -37,9 +38,10 @@ PostgreSQL documentation
<title>Description</title>
<para>
- <application>pg_restore</application> is a utility for restoring a
+ <application>pg_restore</application> is a utility for restoring
<productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
database to the state it was in at the time it was saved. The
archive files also allow <application>pg_restore</application> to
@@ -140,6 +142,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from dump of <application>pg_dumpall</application>.
</para>
<para>
@@ -166,6 +170,25 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +338,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..a36d2a5bf84 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -326,11 +326,18 @@ getThreadLocalPQExpBuffer(void)
* pg_dump and pg_restore call this to register the cleanup handler
* as soon as they've created the ArchiveHandle.
*/
-void
+int
on_exit_close_archive(Archive *AHX)
{
shutdown_info.AHX = AHX;
- on_exit_nicely(archive_close_connection, &shutdown_info);
+ return on_exit_nicely(archive_close_connection, &shutdown_info);
+}
+
+void
+replace_on_exit_close_archive(Archive *AHX, int idx)
+{
+ shutdown_info.AHX = AHX;
+ set_on_exit_nicely_entry(archive_close_connection, &shutdown_info, idx);
}
/*
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index c68a21027fa..89459dedc4b 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -311,7 +311,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 3fd2818223c..e22a8810b45 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -85,7 +85,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -337,9 +337,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -456,7 +461,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1292,7 +1297,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1671,7 +1676,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1691,7 +1697,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index a2064f471ed..ae433132435 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -385,7 +385,8 @@ struct _tocEntry
};
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
-extern void on_exit_close_archive(Archive *AHX);
+extern int on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX, int idx);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..59ece2999a8 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -61,14 +61,26 @@ set_dump_section(const char *arg, int *dumpSections)
/* Register a callback to be run when exit_nicely is invoked. */
-void
+int
on_exit_nicely(on_exit_nicely_callback function, void *arg)
{
- if (on_exit_nicely_index >= MAX_ON_EXIT_NICELY)
- pg_fatal("out of on_exit_nicely slots");
- on_exit_nicely_list[on_exit_nicely_index].function = function;
- on_exit_nicely_list[on_exit_nicely_index].arg = arg;
+ set_on_exit_nicely_entry(function, arg, on_exit_nicely_index);
on_exit_nicely_index++;
+
+ return (on_exit_nicely_index - 1);
+}
+
+void
+set_on_exit_nicely_entry(on_exit_nicely_callback function, void *arg, int i)
+{
+ if (i >= MAX_ON_EXIT_NICELY)
+ pg_fatal("out of on_exit_nicely slots");
+
+ if (i > on_exit_nicely_index)
+ pg_fatal("no entry exists on %d index into on_exit_nicely slots", i);
+
+ on_exit_nicely_list[i].function = function;
+ on_exit_nicely_list[i].arg = arg;
}
/*
diff --git a/src/bin/pg_dump/pg_backup_utils.h b/src/bin/pg_dump/pg_backup_utils.h
index ba042016879..bbefdc112f5 100644
--- a/src/bin/pg_dump/pg_backup_utils.h
+++ b/src/bin/pg_dump/pg_backup_utils.h
@@ -28,7 +28,8 @@ typedef void (*on_exit_nicely_callback) (int code, void *arg);
extern const char *progname;
extern void set_dump_section(const char *arg, int *dumpSections);
-extern void on_exit_nicely(on_exit_nicely_callback function, void *arg);
+extern int on_exit_nicely(on_exit_nicely_callback function, void *arg);
+extern void set_on_exit_nicely_entry(on_exit_nicely_callback function, void *arg, int idx);
pg_noreturn extern void exit_nicely(int code);
/* In pg_dump, we modify pg_fatal to call exit_nicely instead of exit */
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 015a434fc0b..608b0696c03 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1219,7 +1219,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 573a8b61a45..a3dcc585ace 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -64,9 +65,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -75,6 +77,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -146,6 +150,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -195,6 +200,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -244,7 +251,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -272,7 +279,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -421,6 +430,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If non-plain format is specified then we must provide the
+ * file name to create one main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -483,6 +507,33 @@ main(int argc, char *argv[])
if (statistics_only)
appendPQExpBufferStr(pgdumpopts, " --statistics-only");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char toc_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -522,19 +573,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -634,7 +672,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -647,7 +685,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -658,12 +696,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster based on specified dump format.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1570,10 +1610,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1587,7 +1630,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1595,9 +1638,33 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_fatal("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1612,6 +1679,18 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is non-plain dump format, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1630,9 +1709,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1641,19 +1728,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char toc_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(toc_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(toc_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ toc_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1663,7 +1761,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1672,17 +1771,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is non-plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1827,3 +1945,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if(!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 47f7b0dd3a1..e4093427e2f 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,30 +41,77 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/connect.h"
+#include "compress_io.h"
+#include "common/string.h"
+#include "connectdb.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
+#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
+typedef struct SimpleDatabaseOidListCell
+{
+ struct SimpleDatabaseOidListCell *next;
+ Oid db_oid;
+ const char *db_name;
+} SimpleDatabaseOidListCell;
+
+typedef struct SimpleDatabaseOidList
+{
+ SimpleDatabaseOidListCell *head;
+ SimpleDatabaseOidListCell *tail;
+} SimpleDatabaseOidList;
+
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num);
+static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static void copy_or_print_global_file(const char *outfile, FILE *pfile);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleDatabaseOidList *dbname_oid_list);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname);
+static void simple_string_full_list_delete(SimpleStringList *list);
+static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
+static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev);
+static void simple_db_oid_list_append(SimpleDatabaseOidList *list,
+ Oid db_oid, const char *dbname);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
+static int on_exit_index = 0;
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -90,6 +137,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -144,6 +192,7 @@ main(int argc, char **argv)
{"with-statistics", no_argument, &with_statistics, 1},
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -172,7 +221,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -199,11 +248,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -318,6 +370,10 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6:
+ /* list of databases patterns those needs to skip while restoring */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -345,6 +401,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -452,6 +515,108 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL && !IsFileExistsInDirectory(inputFileSpec, "toc.dat") &&
+ IsFileExistsInDirectory(inputFileSpec, "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql commands. */
+
+ /*
+ * User is suggested to use single database dump for --list option.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring multiple databases by archive of pg_dumpall");
+
+ /*
+ * To restore multiple databases, -C (create database) option should be specified.
+ * Even there is single database in dump, report error because it might be possible
+ * that database hasn't created so better we report error.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("-C/--create option should be specified when restoring multiple databases by archive of pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("If db is already created and dump has single db dump, then use particular dump file.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Connect to database to execute global sql commands from global.dat file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ /* Open global.dat file and execute/append all the global sql commands. */
+ n_errors = process_global_sql_commands(conn, inputFileSpec,
+ opts->filename);
+
+ if (conn)
+ PQfinish(conn);
+
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ n_errors = restoreAllDatabases(conn, inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_full_list_delete(&db_exclude_patterns);
+ }
+ else /* process if global.dat file does not exist. */
+ {
+ if (db_exclude_patterns.head != NULL)
+ pg_fatal("option --exclude-database can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ n_errors = restoreOneDatabase(inputFileSpec, opts, numWorkers, false, 0);
+ }
+
+ on_exit_index = 0; /* Reset index. */
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -460,8 +625,14 @@ main(int argc, char **argv)
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
* it's still NULL, the cleanup function will just be a no-op.
+ * If we are restoring multiple databases, then save index of exit_nicely
+ * so that we can use same slot for all the databases as we already closed
+ * the previous archive by CloseArchive.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_index = on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH, on_exit_index);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -481,25 +652,22 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n"
+ "If archive is created by pg_dumpall, then restores multiple databases also. \n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -517,6 +685,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -529,6 +698,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -569,8 +739,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be combined\n"
+ "and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -675,3 +845,648 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * IsFileExistsInDirectory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+IsFileExistsInDirectory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * ReadOneStatement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.dat file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+ReadOneStatement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+ int m;
+
+ StringInfoData q;
+ initStringInfo(&q);
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != '\n' && c != ';')
+ {
+ appendStringInfoChar(inBuf, (char) c);
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != ';' && c != '\n')
+ appendStringInfoChar(inBuf, (char) c);
+ else
+ break;
+ }
+ }
+
+ if (c == '\'' || c == '"')
+ {
+ appendStringInfoChar(&q, (char) c);
+ m = c;
+
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(&q, (char) c);
+
+ if(c == m)
+ {
+ appendStringInfoString(inBuf, q.data);
+ resetStringInfo(&q);
+ break;
+ }
+ }
+ }
+
+ if (c == ';')
+ {
+ appendStringInfoChar(inBuf, (char) ';');
+ break;
+ }
+
+ if (c == '\n')
+ appendStringInfoChar(inBuf, (char) '\n');
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will remove entries from dbname_oid_list that pattern matching any
+ * in the db_exclude_patterns list. dbname_oid_list maybe inplace modified.
+ *
+ * returns, number of database will be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimpleDatabaseOidList *dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
+ SimpleDatabaseOidListCell *dboidprecell = NULL;
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ /* Return 0 if there is no database to restore. */
+ if (dboid_cell == NULL)
+ return 0;
+
+ query = createPQExpBuffer();
+
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ while (dboid_cell != NULL)
+ {
+ bool skip_db_restore = false;
+ SimpleDatabaseOidListCell *next = dboid_cell->next;
+
+ for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ {
+ /*
+ * the construct pattern matching query:
+ * SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE
+ * pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from the
+ * dbname_oid_list, which is initially extracted from the map.dat
+ * file located in the backup directory. that's why we need
+ * quote_literal_cstr.
+ *
+ * If no db connection, then consider PATTERN as NAME.
+ */
+ if (pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0)
+ skip_db_restore = true;
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, celldb->val, false,
+ false, NULL, quote_literal_cstr(dboid_cell->db_name),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ celldb->val);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database \"%s\" is matching with exclude pattern: \"%s\"", dboid_cell->db_name, celldb->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /* Increment count if database needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
+ simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ }
+ else
+ {
+ count_db++;
+ dboidprecell = dboid_cell;
+ }
+
+ /* Process next dbname from dbname list. */
+ dboid_cell = next;
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ /*
+ * If there is only global.dat file in dump, then return from here as there
+ * is no database to restore.
+ */
+ if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ {
+ pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid = InvalidOid;
+ char db_oid_str[MAXPGPATH + 1] = {'\0'};
+ char dbname[MAXPGPATH + 1] = {'\0'};
+
+ /* Extract dboid. */
+ sscanf(line, "%u" , &db_oid);
+ sscanf(line, "%s" , db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found database \"%s\" (OID: %u) in map.dat file while restoring.", dbname, db_oid);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making
+ * a list of all the databases.
+ */
+ simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
+ SimpleDatabaseOidListCell *dboid_cell;
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /*
+ * If map.dat has no entry, return from here after processing
+ * global.dat file.
+ */
+ if (dbname_oid_list.head == NULL)
+ return process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+
+ /*
+ * processing pg_retsore --exclude-database=PATTERN/NAME if no connection.
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ n_errors_total = process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL)
+ {
+ pg_log_info("no database needs to restore out of %d databases", num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * Till now, we made a list of databases, those needs to be restored
+ * after skipping names of exclude-database. Now we can launch parallel
+ * workers to restore these databases.
+ */
+ dboid_cell = dbname_oid_list.head;
+
+ while(dboid_cell != NULL)
+ {
+ char subdirpath[MAXPGPATH];
+ int n_errors;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored into
+ * already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+
+ pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+
+ /* Restore single database. */
+ n_errors = restoreOneDatabase(subdirpath, opts, numWorkers, true, count);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", dboid_cell->db_name, n_errors);
+ }
+
+ dboid_cell = dboid_cell->next;
+ count++;
+ }
+
+ /* Log number of processed databases.*/
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_db_oid_full_list_delete(&dbname_oid_list);
+
+ return n_errors_total;
+}
+
+/*
+ * process_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ *
+ * returns the number of errors while processing global.dat
+ */
+static int
+process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+ int n_errors = 0;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ copy_or_print_global_file(outfile, pfile);
+ return 0;
+ }
+
+ /* Init sqlstatement to append commands. */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements. */
+ while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ {
+ pg_log_info("executing query: %s", sqlstatement.data);
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ n_errors++;
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ /* Print a summary of ignored errors during global.dat. */
+ if (n_errors)
+ pg_log_warning("errors ignored on global.dat file restore: %d", n_errors);
+
+ fclose(pfile);
+
+ return n_errors;
+}
+
+/*
+ * copy_or_print_global_file
+ *
+ * This will copy global.dat file into out file. If "-" is used as outfile,
+ * then print commands to the stdout.
+ */
+static void
+copy_or_print_global_file(const char *outfile, FILE *pfile)
+{
+ char out_file_path[MAXPGPATH];
+ FILE *OPF;
+ int c;
+
+ /* "-" is used for stdout. */
+ if (strcmp(outfile, "-") == 0)
+ OPF = stdout;
+ else
+ {
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ OPF = fopen(out_file_path, PG_BINARY_W);
+
+ if (OPF == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", outfile);
+ }
+ }
+
+ /* Append global.dat into out file or print to the stdout. */
+ while ((c = fgetc(pfile)) != EOF)
+ fputc(c, OPF);
+
+ fclose(pfile);
+
+ /* Close out file. */
+ if (strcmp(outfile, "-") != 0)
+ fclose(OPF);
+}
+
+/*
+ * simple_db_oid_list_append
+ *
+ * appends a node to the list in the end.
+ */
+static void
+simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
+ const char *dbname)
+{
+ SimpleDatabaseOidListCell *cell;
+
+ cell = pg_malloc_object(SimpleDatabaseOidListCell);
+
+ cell->next = NULL;
+ cell->db_oid = db_oid;
+ cell->db_name = pg_strdup(dbname);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * simple_db_oid_full_list_delete
+ *
+ * delete all cell from dbname and dboid list.
+ */
+static void
+simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
+{
+ SimpleDatabaseOidListCell *cell = list->head;
+ SimpleDatabaseOidListCell *nextcell = NULL;
+
+ while (cell)
+ {
+ nextcell = cell->next;
+ pfree (cell);
+ cell = nextcell;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_string_full_list_delete
+ *
+ * delete all cell from string list.
+ */
+static void
+simple_string_full_list_delete(SimpleStringList *list)
+{
+ SimpleStringListCell *cell = list->head;
+ SimpleStringListCell *cellnext = NULL;
+
+ while (cell)
+ {
+ cellnext = cell->next;
+ pfree(cell);
+ cell = cellnext;
+ }
+
+ list->head = NULL;
+ list->tail = NULL;
+}
+
+/*
+ * simple_db_oid_list_delete
+ *
+ * delete cell from database and oid list.
+ */
+static void
+simple_db_oid_list_delete(SimpleDatabaseOidList *list,
+ SimpleDatabaseOidListCell *cell,
+ SimpleDatabaseOidListCell *prev)
+{
+ if (prev == NULL)
+ {
+ list->head = cell->next;
+ pfree(cell);
+ }
+ else
+ {
+ prev->next = cell->next;
+ pfree(cell);
+ }
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr
+ *
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..0bbcdbe84a7
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,11 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +249,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3fbf5a4c212..f3999ee3f9d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2726,6 +2726,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.34.1
[text/x-patch] 0003-add-new-list-type-simple_oid_string_list-to-fe-utils.patch (2.6K, 4-0003-add-new-list-type-simple_oid_string_list-to-fe-utils.patch)
download | inline diff:
From bfc7c37284cf38ecad18b79c6e08a9fda06512eb Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email protected]>
Date: Fri, 28 Mar 2025 18:10:24 -0400
Subject: [PATCH 3/4] add new list type simple_oid_string_list to
fe-utils/simple_list
---
src/fe_utils/simple_list.c | 41 ++++++++++++++++++++++++++++++
src/include/fe_utils/simple_list.h | 16 ++++++++++++
2 files changed, 57 insertions(+)
diff --git a/src/fe_utils/simple_list.c b/src/fe_utils/simple_list.c
index 483d5455594..b0686e57c4a 100644
--- a/src/fe_utils/simple_list.c
+++ b/src/fe_utils/simple_list.c
@@ -192,3 +192,44 @@ simple_ptr_list_destroy(SimplePtrList *list)
cell = next;
}
}
+
+/*
+ * Add to an oid_string list
+ */
+void
+simple_oid_string_list_append(SimpleOidStringList *list, Oid oid, const char *str)
+{
+ SimpleOidStringListCell *cell;
+
+ cell = (SimpleOidStringListCell *)
+ pg_malloc(offsetof(SimpleOidStringListCell, str) + strlen(str) + 1);
+
+ cell->next = NULL;
+ cell->oid = oid;
+ strcpy(cell->str, str);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * Destroy an oid_string list
+ */
+void
+simple_oid_string_list_destroy(SimpleOidStringList *list)
+{
+ SimpleOidStringListCell *cell;
+
+ cell = list->head;
+ while (cell != NULL)
+ {
+ SimpleOidStringListCell *next;
+
+ next = cell->next;
+ pg_free(cell);
+ cell = next;
+ }
+}
diff --git a/src/include/fe_utils/simple_list.h b/src/include/fe_utils/simple_list.h
index 3b8e38414ec..d5492408d6c 100644
--- a/src/include/fe_utils/simple_list.h
+++ b/src/include/fe_utils/simple_list.h
@@ -55,6 +55,19 @@ typedef struct SimplePtrList
SimplePtrListCell *tail;
} SimplePtrList;
+typedef struct SimpleOidStringListCell
+{
+ struct SimpleOidStringListCell *next;
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} SimpleOidStringListCell;
+
+typedef struct SimpleOidStringList
+{
+ SimpleOidStringListCell *head;
+ SimpleOidStringListCell *tail;
+} SimpleOidStringList;
+
extern void simple_oid_list_append(SimpleOidList *list, Oid val);
extern bool simple_oid_list_member(SimpleOidList *list, Oid val);
extern void simple_oid_list_destroy(SimpleOidList *list);
@@ -68,4 +81,7 @@ extern const char *simple_string_list_not_touched(SimpleStringList *list);
extern void simple_ptr_list_append(SimplePtrList *list, void *ptr);
extern void simple_ptr_list_destroy(SimplePtrList *list);
+extern void simple_oid_string_list_append(SimpleOidStringList *list, Oid oid, const char *str);
+extern void simple_oid_string_list_destroy(SimpleOidStringList *list);
+
#endif /* SIMPLE_LIST_H */
--
2.34.1
[text/x-patch] 0004-cleanups-and-use-new-simple-list-type.patch (17.5K, 5-0004-cleanups-and-use-new-simple-list-type.patch)
download | inline diff:
From b51df1f2584f99d67381a3702cec87cb3264e4a3 Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email protected]>
Date: Fri, 28 Mar 2025 18:10:57 -0400
Subject: [PATCH 4/4] cleanups and use new simple list type
---
src/bin/pg_dump/pg_dumpall.c | 28 ++---
src/bin/pg_dump/pg_restore.c | 208 +++++++----------------------------
2 files changed, 56 insertions(+), 180 deletions(-)
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index a3dcc585ace..6aab1bfe831 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -434,13 +434,13 @@ main(int argc, char *argv[])
archDumpFormat = parseDumpFormat(formatName);
/*
- * If non-plain format is specified then we must provide the
- * file name to create one main directory.
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
*/
if (archDumpFormat != archNull &&
(!filename || strcmp(filename, "") == 0))
{
- pg_log_error("options -F/--format=d|c|t requires option -f/--file with non-empty string");
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
exit_nicely(1);
}
@@ -513,14 +513,14 @@ main(int argc, char *argv[])
*/
if (archDumpFormat != archNull)
{
- char toc_path[MAXPGPATH];
+ char global_path[MAXPGPATH];
/* Create new directory or accept the empty existing directory. */
create_or_open_dir(filename);
- snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
- OPF = fopen(toc_path, PG_BINARY_W);
+ OPF = fopen(global_path, PG_BINARY_W);
if (!OPF)
pg_fatal("could not open global.dat file: %s", strerror(errno));
}
@@ -1680,7 +1680,7 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
}
/*
- * If this is non-plain dump format, then append dboid and dbname to
+ * If this is not a plain format dump, then append dboid and dbname to
* the map.dat file.
*/
if (archDumpFormat != archNull)
@@ -1688,7 +1688,7 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
/* Put one line entry for dboid and dbname in map file. */
- fprintf(map_file, "%s %s\n", oid, pg_strdup(dbname));
+ fprintf(map_file, "%s %s\n", oid, dbname);
}
pg_log_info("dumping database \"%s\"", dbname);
@@ -1734,17 +1734,17 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
if (filename)
{
- char toc_path[MAXPGPATH];
+ char global_path[MAXPGPATH];
if (archDumpFormat != archNull)
- snprintf(toc_path, MAXPGPATH, "%s/global.dat", filename);
+ snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
else
- snprintf(toc_path, MAXPGPATH, "%s", filename);
+ snprintf(global_path, MAXPGPATH, "%s", filename);
- OPF = fopen(toc_path, PG_BINARY_A);
+ OPF = fopen(global_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- toc_path);
+ global_path);
}
}
@@ -1772,7 +1772,7 @@ runPgDump(const char *dbname, const char *create_opts, char *dbfile,
initPQExpBuffer(&cmd);
/*
- * If this is non-plain format dump, then append file name and dump
+ * If this is not a plain format dump, then append file name and dump
* format to the pg_dump command to get archive dump.
*/
if (archDumpFormat != archNull)
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index e4093427e2f..44a24791a6e 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -46,8 +46,6 @@
#include <termios.h>
#endif
-#include "common/connect.h"
-#include "compress_io.h"
#include "common/string.h"
#include "connectdb.h"
#include "fe_utils/option_utils.h"
@@ -55,47 +53,24 @@
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
-#include "pg_backup_archiver.h"
#include "pg_backup_utils.h"
-typedef struct SimpleDatabaseOidListCell
-{
- struct SimpleDatabaseOidListCell *next;
- Oid db_oid;
- const char *db_name;
-} SimpleDatabaseOidListCell;
-
-typedef struct SimpleDatabaseOidList
-{
- SimpleDatabaseOidListCell *head;
- SimpleDatabaseOidListCell *tail;
-} SimpleDatabaseOidList;
-
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
-static bool IsFileExistsInDirectory(const char *dir, const char *filename);
+static bool file_exists_in_directory(const char *dir, const char *filename);
static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
int numWorkers, bool append_data, int num);
-static int ReadOneStatement(StringInfo inBuf, FILE *pfile);
+static int read_one_statement(StringInfo inBuf, FILE *pfile);
static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
static int process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
const char *outfile);
static void copy_or_print_global_file(const char *outfile, FILE *pfile);
static int get_dbnames_list_to_restore(PGconn *conn,
- SimpleDatabaseOidList *dbname_oid_list,
+ SimpleOidStringList *dbname_oid_list,
SimpleStringList db_exclude_patterns);
static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
- SimpleDatabaseOidList *dbname_oid_list);
-static void simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
- const char *dbname);
-static void simple_string_full_list_delete(SimpleStringList *list);
-static void simple_db_oid_full_list_delete(SimpleDatabaseOidList *list);
-static void simple_db_oid_list_delete(SimpleDatabaseOidList *list,
- SimpleDatabaseOidListCell *cell,
- SimpleDatabaseOidListCell *prev);
-static void simple_db_oid_list_append(SimpleDatabaseOidList *list,
- Oid db_oid, const char *dbname);
+ SimpleOidStringList *dbname_oid_list);
static size_t quote_literal_internal(char *dst, const char *src, size_t len);
static char *quote_literal_cstr(const char *rawstr);
static int on_exit_index = 0;
@@ -521,8 +496,8 @@ main(int argc, char **argv)
* databases from map.dat(if exist) file list and skip restoring for
* --exclude-database patterns.
*/
- if (inputFileSpec != NULL && !IsFileExistsInDirectory(inputFileSpec, "toc.dat") &&
- IsFileExistsInDirectory(inputFileSpec, "global.dat"))
+ if (inputFileSpec != NULL && !file_exists_in_directory(inputFileSpec, "toc.dat") &&
+ file_exists_in_directory(inputFileSpec, "global.dat"))
{
PGconn *conn = NULL; /* Connection to restore global sql commands. */
@@ -578,7 +553,7 @@ main(int argc, char **argv)
}
/* Free db pattern list. */
- simple_string_full_list_delete(&db_exclude_patterns);
+ simple_string_list_destroy(&db_exclude_patterns);
}
else /* process if global.dat file does not exist. */
{
@@ -847,12 +822,12 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
}
/*
- * IsFileExistsInDirectory
+ * file_exists_in_directory
*
* Returns true if file exist in current directory.
*/
static bool
-IsFileExistsInDirectory(const char *dir, const char *filename)
+file_exists_in_directory(const char *dir, const char *filename)
{
struct stat st;
char buf[MAXPGPATH];
@@ -864,7 +839,7 @@ IsFileExistsInDirectory(const char *dir, const char *filename)
}
/*
- * ReadOneStatement
+ * read_one_statement
*
* This will start reading from passed file pointer using fgetc and read till
* semicolon(sql statement terminator for global.dat file)
@@ -873,7 +848,7 @@ IsFileExistsInDirectory(const char *dir, const char *filename)
*/
static int
-ReadOneStatement(StringInfo inBuf, FILE *pfile)
+read_one_statement(StringInfo inBuf, FILE *pfile)
{
int c; /* character read from getc() */
int m;
@@ -941,27 +916,21 @@ ReadOneStatement(StringInfo inBuf, FILE *pfile)
/*
* get_dbnames_list_to_restore
*
- * This will remove entries from dbname_oid_list that pattern matching any
- * in the db_exclude_patterns list. dbname_oid_list maybe inplace modified.
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
*
- * returns, number of database will be restored.
+ * Returns the number of database to be restored.
*
*/
static int
get_dbnames_list_to_restore(PGconn *conn,
- SimpleDatabaseOidList *dbname_oid_list,
+ SimpleOidStringList *dbname_oid_list,
SimpleStringList db_exclude_patterns)
{
- SimpleDatabaseOidListCell *dboid_cell = dbname_oid_list->head;
- SimpleDatabaseOidListCell *dboidprecell = NULL;
int count_db = 0;
PQExpBuffer query;
PGresult *res;
- /* Return 0 if there is no database to restore. */
- if (dboid_cell == NULL)
- return 0;
-
query = createPQExpBuffer();
if (!conn)
@@ -971,12 +940,12 @@ get_dbnames_list_to_restore(PGconn *conn,
* Process one by one all dbnames and if specified to skip restoring, then
* remove dbname from list.
*/
- while (dboid_cell != NULL)
+ for (SimpleOidStringListCell *db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
{
bool skip_db_restore = false;
- SimpleDatabaseOidListCell *next = dboid_cell->next;
- for (SimpleStringListCell *celldb = db_exclude_patterns.head; celldb; celldb = celldb->next)
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
{
/*
* the construct pattern matching query:
@@ -990,21 +959,21 @@ get_dbnames_list_to_restore(PGconn *conn,
*
* If no db connection, then consider PATTERN as NAME.
*/
- if (pg_strcasecmp(dboid_cell->db_name, celldb->val) == 0)
+ if (pg_strcasecmp(db_cell->str, pat_cell->val) == 0)
skip_db_restore = true;
else if (conn)
{
int dotcnt;
appendPQExpBufferStr(query, "SELECT 1 ");
- processSQLNamePattern(conn, query, celldb->val, false,
- false, NULL, quote_literal_cstr(dboid_cell->db_name),
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, quote_literal_cstr(db_cell->str),
NULL, NULL, NULL, &dotcnt);
if (dotcnt > 0)
{
pg_log_error("improper qualified name (too many dotted names): %s",
- celldb->val);
+ db_cell->str);
PQfinish(conn);
exit_nicely(1);
}
@@ -1014,7 +983,7 @@ get_dbnames_list_to_restore(PGconn *conn,
if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
{
skip_db_restore = true;
- pg_log_info("database \"%s\" is matching with exclude pattern: \"%s\"", dboid_cell->db_name, celldb->val);
+ pg_log_info("database \"%s\" matches exclude pattern: \"%s\"", db_cell->str, pat_cell->val);
}
PQclear(res);
@@ -1028,17 +997,13 @@ get_dbnames_list_to_restore(PGconn *conn,
/* Increment count if database needs to be restored. */
if (skip_db_restore)
{
- pg_log_info("excluding database \"%s\"", dboid_cell->db_name);
- simple_db_oid_list_delete(dbname_oid_list, dboid_cell, dboidprecell);
+ pg_log_info("excluding database \"%s\"", db_cell->str);
+ db_cell->oid = InvalidOid;
}
else
{
count_db++;
- dboidprecell = dboid_cell;
}
-
- /* Process next dbname from dbname list. */
- dboid_cell = next;
}
return count_db;
@@ -1053,7 +1018,7 @@ get_dbnames_list_to_restore(PGconn *conn,
* Returns, total number of database names in map.dat file.
*/
static int
-get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *dbname_oid_list)
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleOidStringList *dbname_oid_list)
{
FILE *pfile;
char map_file_path[MAXPGPATH];
@@ -1064,7 +1029,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *d
* If there is only global.dat file in dump, then return from here as there
* is no database to restore.
*/
- if (!IsFileExistsInDirectory(pg_strdup(dumpdirpath), "map.dat"))
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
{
pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
return 0;
@@ -1087,7 +1052,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *d
/* Extract dboid. */
sscanf(line, "%u" , &db_oid);
- sscanf(line, "%s" , db_oid_str);
+ sscanf(line, "%20s" , db_oid_str);
/* Now copy dbname. */
strcpy(dbname, line + strlen(db_oid_str) + 1);
@@ -1106,7 +1071,7 @@ get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleDatabaseOidList *d
* needs to skipped for restore or not but as of now, we are making
* a list of all the databases.
*/
- simple_db_oid_list_append(dbname_oid_list, db_oid, dbname);
+ simple_oid_string_list_append(dbname_oid_list, db_oid, dbname);
count++;
}
@@ -1132,8 +1097,7 @@ restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
SimpleStringList db_exclude_patterns, RestoreOptions *opts,
int numWorkers)
{
- SimpleDatabaseOidList dbname_oid_list = {NULL, NULL};
- SimpleDatabaseOidListCell *dboid_cell;
+ SimpleOidStringList dbname_oid_list = {NULL, NULL};
int num_db_restore = 0;
int num_total_db;
int n_errors_total;
@@ -1183,7 +1147,7 @@ restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
PQfinish(conn);
/* Exit if no db needs to be restored. */
- if (dbname_oid_list.head == NULL)
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
{
pg_log_info("no database needs to restore out of %d databases", num_total_db);
return n_errors_total;
@@ -1196,13 +1160,16 @@ restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
* after skipping names of exclude-database. Now we can launch parallel
* workers to restore these databases.
*/
- dboid_cell = dbname_oid_list.head;
-
- while(dboid_cell != NULL)
+ for (SimpleOidStringListCell *db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
{
char subdirpath[MAXPGPATH];
int n_errors;
+ /* ignore dbs marked for skipping */
+ if (db_cell->oid == InvalidOid)
+ continue;
+
/*
* We need to reset override_dbname so that objects can be restored into
* already created database. (used with -d/--dbname option)
@@ -1213,9 +1180,9 @@ restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
opts->cparams.override_dbname = NULL;
}
- snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, dboid_cell->db_oid);
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, db_cell->oid);
- pg_log_info("restoring database \"%s\"", dboid_cell->db_name);
+ pg_log_info("restoring database \"%s\"", db_cell->str);
/* Restore single database. */
n_errors = restoreOneDatabase(subdirpath, opts, numWorkers, true, count);
@@ -1224,10 +1191,9 @@ restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
if (n_errors)
{
n_errors_total += n_errors;
- pg_log_warning("errors ignored on database \"%s\" restore: %d", dboid_cell->db_name, n_errors);
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", db_cell->str, n_errors);
}
- dboid_cell = dboid_cell->next;
count++;
}
@@ -1235,7 +1201,7 @@ restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
pg_log_info("number of restored databases are %d", num_db_restore);
/* Free dbname and dboid list. */
- simple_db_oid_full_list_delete(&dbname_oid_list);
+ simple_oid_string_list_destroy(&dbname_oid_list);
return n_errors_total;
}
@@ -1281,7 +1247,7 @@ process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *o
initStringInfo(&sqlstatement);
/* Process file till EOF and execute sql statements. */
- while (ReadOneStatement(&sqlstatement, pfile) != EOF)
+ while (read_one_statement(&sqlstatement, pfile) != EOF)
{
pg_log_info("executing query: %s", sqlstatement.data);
result = PQexec(conn, sqlstatement.data);
@@ -1347,96 +1313,6 @@ copy_or_print_global_file(const char *outfile, FILE *pfile)
fclose(OPF);
}
-/*
- * simple_db_oid_list_append
- *
- * appends a node to the list in the end.
- */
-static void
-simple_db_oid_list_append(SimpleDatabaseOidList *list, Oid db_oid,
- const char *dbname)
-{
- SimpleDatabaseOidListCell *cell;
-
- cell = pg_malloc_object(SimpleDatabaseOidListCell);
-
- cell->next = NULL;
- cell->db_oid = db_oid;
- cell->db_name = pg_strdup(dbname);
-
- if (list->tail)
- list->tail->next = cell;
- else
- list->head = cell;
- list->tail = cell;
-}
-
-/*
- * simple_db_oid_full_list_delete
- *
- * delete all cell from dbname and dboid list.
- */
-static void
-simple_db_oid_full_list_delete(SimpleDatabaseOidList *list)
-{
- SimpleDatabaseOidListCell *cell = list->head;
- SimpleDatabaseOidListCell *nextcell = NULL;
-
- while (cell)
- {
- nextcell = cell->next;
- pfree (cell);
- cell = nextcell;
- }
-
- list->head = NULL;
- list->tail = NULL;
-}
-
-/*
- * simple_string_full_list_delete
- *
- * delete all cell from string list.
- */
-static void
-simple_string_full_list_delete(SimpleStringList *list)
-{
- SimpleStringListCell *cell = list->head;
- SimpleStringListCell *cellnext = NULL;
-
- while (cell)
- {
- cellnext = cell->next;
- pfree(cell);
- cell = cellnext;
- }
-
- list->head = NULL;
- list->tail = NULL;
-}
-
-/*
- * simple_db_oid_list_delete
- *
- * delete cell from database and oid list.
- */
-static void
-simple_db_oid_list_delete(SimpleDatabaseOidList *list,
- SimpleDatabaseOidListCell *cell,
- SimpleDatabaseOidListCell *prev)
-{
- if (prev == NULL)
- {
- list->head = cell->next;
- pfree(cell);
- }
- else
- {
- prev->next = cell->next;
- pfree(cell);
- }
-}
-
/*
* quote_literal_internal
*/
--
2.34.1
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-27 21:15 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-28 22:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
@ 2025-03-29 05:17 ` Mahendra Singh Thalor <[email protected]>
2025-03-30 16:50 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Mahendra Singh Thalor @ 2025-03-29 05:17 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: jian he <[email protected]>; Álvaro Herrera <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Sat, 29 Mar 2025 at 03:50, Andrew Dunstan <[email protected]> wrote:
>
>
> On 2025-03-27 Th 5:15 PM, Andrew Dunstan wrote:
> >
> > On 2025-03-19 We 2:41 AM, Mahendra Singh Thalor wrote:
> >> On Wed, 12 Mar 2025 at 21:18, Andrew Dunstan <[email protected]>
> >> wrote:
> >>>
> >>> On 2025-03-12 We 3:03 AM, jian he wrote:
> >>>> On Wed, Mar 12, 2025 at 1:06 AM Álvaro Herrera
> >>>> <[email protected]> wrote:
> >>>>> Hello,
> >>>>>
> >>>>> On 2025-Mar-11, Mahendra Singh Thalor wrote:
> >>>>>
> >>>>>> In map.dat file, I tried to fix this issue by adding number of
> >>>>>> characters
> >>>>>> in dbname but as per code comments, as of now, we are not
> >>>>>> supporting \n\r
> >>>>>> in dbnames so i removed handling.
> >>>>>> I will do some more study to fix this issue.
> >>>>> Yeah, I think this is saying that you should not consider the
> >>>>> contents
> >>>>> of map.dat as a shell string. After all, you're not going to
> >>>>> _execute_
> >>>>> that file via the shell.
> >>>>>
> >>>>> Maybe for map.dat you need to escape such characters somehow, so that
> >>>>> they don't appear as literal newlines/carriage returns.
> >>>>>
> >>>> I am confused.
> >>>> currently pg_dumpall plain format will abort when encountering dbname
> >>>> containing newline.
> >>>> the left dumped plain file does not contain all the cluster
> >>>> databases data.
> >>>>
> >>>>
> >>>> if pg_dumpall non-text format aborts earlier,
> >>>> it's aligned with pg_dumpall plain format?
> >>>> it's also an improvement since aborts earlier, nothing will be dumped?
> >>>>
> >>>>
> >>>> am i missing something?
> >>>>
> >>>>
> >>> I think we should fix that.
> >>>
> >>> But for the current proposal, Álvaro and I were talking this morning,
> >>> and we thought the simplest thing here would be to have the one line
> >>> format and escape NL/CRs in the database name.
> >>>
> >>>
> >>> cheers
> >>>
> >> Okay. As per discussions, we will keep one line entry for each
> >> database into map.file.
> >>
> >> Thanks all for feedback and review.
> >>
> >> Here, I am attaching updated patches for review and testing. These
> >> patches can be applied on commit a6524105d20b.
> >
> >
> >
> > I'm working through this patch set with a view to committing it.
> > Attached is some cleanup which is where I got to today, although there
> > is more to do. One thing I am wondering is why not put the
> > SimpleDatabaseOidList stuff in fe_utils/simle_list.{c,h} ? That's
> > where all the similar stuff belongs, and it feels strange to have this
> > inline in pg_restore.c. (I also don't like the name much -
> > SimpleOidStringList or maybe SimpleOidPlusStringList might be better).
> >
> >
> >
>
>
> OK, I have done that, so here is the result. The first two are you
> original patches. patch 3 adds the new list type to fe-utils, and patch
> 4 contains my cleanups and use of the new list type. Apart from some
> relatively minor cleanup, the one thing I would like to change is how
> dumps are named. If we are producing tar or custom format dumps, I think
> the file names should reflect that (oid.dmp and oid.tar rather than a
> bare oid as the filename), and pg_restore should look for those. I'm
> going to work on that tomorrow - I don't think it will be terribly
> difficult.
>
Thanks Andrew.
Here, I am attaching a delta patch for oid.tar and oid.dmp format.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] delta-0001-pg_dumpall-dump-as-tar-and-dmp-file-for-file.patch (2.7K, 2-delta-0001-pg_dumpall-dump-as-tar-and-dmp-file-for-file.patch)
download | inline diff:
From b43ae117fe3809b82abb5bc89fc62d45a5707ff6 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Sat, 29 Mar 2025 10:14:18 +0530
Subject: [PATCH] pg_dumpall - dump as .tar and .dmp file for tar and custom
format
---
src/bin/pg_dump/pg_dumpall.c | 7 ++++++-
src/bin/pg_dump/pg_restore.c | 22 +++++++++++++++++++++-
2 files changed, 27 insertions(+), 2 deletions(-)
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 6aab1bfe831..12983d973be 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -1685,7 +1685,12 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
*/
if (archDumpFormat != archNull)
{
- snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
/* Put one line entry for dboid and dbname in map file. */
fprintf(map_file, "%s %s\n", oid, dbname);
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 44a24791a6e..31cd9c84c5a 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -1164,6 +1164,8 @@ restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
db_cell; db_cell = db_cell->next)
{
char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
int n_errors;
/* ignore dbs marked for skipping */
@@ -1180,7 +1182,25 @@ restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
opts->cparams.override_dbname = NULL;
}
- snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, db_cell->oid);
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", dumpdirpath);
+
+ /*
+ * Validate database dump file. If there is .tar or .dmp file exist
+ * then consider particular file, otherwise just append dboid to the
+ * databases folder.
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", db_cell->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", dumpdirpath, db_cell->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", db_cell->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", dumpdirpath, db_cell->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, db_cell->oid);
+ }
pg_log_info("restoring database \"%s\"", db_cell->str);
--
2.39.3
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-27 21:15 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-28 22:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-29 05:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-03-30 16:50 ` Andrew Dunstan <[email protected]>
2025-03-30 22:05 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 09:34 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 2 replies; 29+ messages in thread
From: Andrew Dunstan @ 2025-03-30 16:50 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: jian he <[email protected]>; Álvaro Herrera <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-03-29 Sa 1:17 AM, Mahendra Singh Thalor wrote:
> On Sat, 29 Mar 2025 at 03:50, Andrew Dunstan <[email protected]> wrote:
>>
>> On 2025-03-27 Th 5:15 PM, Andrew Dunstan wrote:
>>> On 2025-03-19 We 2:41 AM, Mahendra Singh Thalor wrote:
>>>> On Wed, 12 Mar 2025 at 21:18, Andrew Dunstan <[email protected]>
>>>> wrote:
>>>>> On 2025-03-12 We 3:03 AM, jian he wrote:
>>>>>> On Wed, Mar 12, 2025 at 1:06 AM Álvaro Herrera
>>>>>> <[email protected]> wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> On 2025-Mar-11, Mahendra Singh Thalor wrote:
>>>>>>>
>>>>>>>> In map.dat file, I tried to fix this issue by adding number of
>>>>>>>> characters
>>>>>>>> in dbname but as per code comments, as of now, we are not
>>>>>>>> supporting \n\r
>>>>>>>> in dbnames so i removed handling.
>>>>>>>> I will do some more study to fix this issue.
>>>>>>> Yeah, I think this is saying that you should not consider the
>>>>>>> contents
>>>>>>> of map.dat as a shell string. After all, you're not going to
>>>>>>> _execute_
>>>>>>> that file via the shell.
>>>>>>>
>>>>>>> Maybe for map.dat you need to escape such characters somehow, so that
>>>>>>> they don't appear as literal newlines/carriage returns.
>>>>>>>
>>>>>> I am confused.
>>>>>> currently pg_dumpall plain format will abort when encountering dbname
>>>>>> containing newline.
>>>>>> the left dumped plain file does not contain all the cluster
>>>>>> databases data.
>>>>>>
>>>>>>
>>>>>> if pg_dumpall non-text format aborts earlier,
>>>>>> it's aligned with pg_dumpall plain format?
>>>>>> it's also an improvement since aborts earlier, nothing will be dumped?
>>>>>>
>>>>>>
>>>>>> am i missing something?
>>>>>>
>>>>>>
>>>>> I think we should fix that.
>>>>>
>>>>> But for the current proposal, Álvaro and I were talking this morning,
>>>>> and we thought the simplest thing here would be to have the one line
>>>>> format and escape NL/CRs in the database name.
>>>>>
>>>>>
>>>>> cheers
>>>>>
>>>> Okay. As per discussions, we will keep one line entry for each
>>>> database into map.file.
>>>>
>>>> Thanks all for feedback and review.
>>>>
>>>> Here, I am attaching updated patches for review and testing. These
>>>> patches can be applied on commit a6524105d20b.
>>>
>>>
>>> I'm working through this patch set with a view to committing it.
>>> Attached is some cleanup which is where I got to today, although there
>>> is more to do. One thing I am wondering is why not put the
>>> SimpleDatabaseOidList stuff in fe_utils/simle_list.{c,h} ? That's
>>> where all the similar stuff belongs, and it feels strange to have this
>>> inline in pg_restore.c. (I also don't like the name much -
>>> SimpleOidStringList or maybe SimpleOidPlusStringList might be better).
>>>
>>>
>>>
>>
>> OK, I have done that, so here is the result. The first two are you
>> original patches. patch 3 adds the new list type to fe-utils, and patch
>> 4 contains my cleanups and use of the new list type. Apart from some
>> relatively minor cleanup, the one thing I would like to change is how
>> dumps are named. If we are producing tar or custom format dumps, I think
>> the file names should reflect that (oid.dmp and oid.tar rather than a
>> bare oid as the filename), and pg_restore should look for those. I'm
>> going to work on that tomorrow - I don't think it will be terribly
>> difficult.
>>
> Thanks Andrew.
>
> Here, I am attaching a delta patch for oid.tar and oid.dmp format.
>
OK, looks good, I have incorporated that.
There are a couple of rough edges, though.
First, I see this:
andrew@ub22arm:inst $ bin/pg_restore -C -d postgres
--exclude-database=regression_dummy_seclabel
--exclude-database=regression_test_extensions
--exclude-database=regression_test_pg_dump dest
pg_restore: error: could not execute query: "ERROR: role "andrew"
already exists
"
Command was: "
--
-- Roles
--
CREATE ROLE andrew;"
pg_restore: warning: errors ignored on global.dat file restore: 1
pg_restore: error: could not execute query: ERROR: database "template1"
already exists
Command was: CREATE DATABASE template1 WITH TEMPLATE = template0
ENCODING = 'SQL_ASCII' LOCALE_PROVIDER = libc LOCALE = 'C';
pg_restore: warning: errors ignored on database "template1" restore: 1
pg_restore: error: could not execute query: ERROR: database "postgres"
already exists
Command was: CREATE DATABASE postgres WITH TEMPLATE = template0 ENCODING
= 'SQL_ASCII' LOCALE_PROVIDER = libc LOCALE = 'C';
pg_restore: warning: errors ignored on database "postgres" restore: 1
pg_restore: warning: errors ignored on restore: 3
It seems pointless to be trying to create the rolw that we are connected
as, and we also expect template1 and postgres to exist.
In a similar vein, I don't see why we are setting the --create flag in
pg_dumpall for those databases. I'm attaching a patch that is designed
to stop that, but it doesn't solve the above issues.
I also notice a bunch of these in globals.dat:
--
-- Databases
--
--
-- Database "template1" dump
--
--
-- Database "andrew" dump
--
--
-- Database "isolation_regression_brin" dump
--
--
-- Database "isolation_regression_delay_execution" dump
--
...
The patch also tries to fix this.
Lastly, this badly needs some TAP tests written.
I'm going to work on reviewing the documentation next.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 7ceaba27419..7a06e595b88 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -1639,10 +1639,9 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
fprintf(OPF, "--\n-- Databases\n--\n\n");
/*
- * If directory/tar/custom format is specified then create a subdirectory
- * under the main directory and each database dump file subdirectory will
- * be created under the subdirectory in archive mode as per single db
- * pg_dump.
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
*/
if (archDumpFormat != archNull)
{
@@ -1666,7 +1665,7 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
char *dbname = PQgetvalue(res, i, 0);
char *oid = PQgetvalue(res, i, 1);
- const char *create_opts;
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1699,7 +1698,8 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
pg_log_info("dumping database \"%s\"", dbname);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
/*
* We assume that "template1" and "postgres" already exist in the
@@ -1713,20 +1713,8 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
if (output_clean)
create_opts = "--clean --create";
- else
- {
- /* Since pg_dump won't emit a \connect command, we must */
- if (archDumpFormat == archNull)
- {
- create_opts = "";
- fprintf(OPF, "\\connect %s\n\n", dbname);
- }
- else
- {
- /* Dumping all databases so add --create option. */
- create_opts = "--create";
- }
- }
+ else if (archDumpFormat == archNull)
+ fprintf(OPF, "\\connect %s\n\n", dbname);
}
else
create_opts = "--create";
Attachments:
[text/x-patch] v20250330-0001-Move-common-pg_dump-code-related-to-connec.patch (26.2K, 2-v20250330-0001-Move-common-pg_dump-code-related-to-connec.patch)
download | inline diff:
From ed53d8b5ad82e49bb56bc5bc48ddb8426fdb4c80 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Wed, 19 Mar 2025 01:18:46 +0530
Subject: [PATCH v20250330 1/3] Move common pg_dump code related to connections
to a new file
ConnectDatabase is used by pg_dumpall, pg_restore and pg_dump so move
common code to new file.
new file name: connectdb.c
Author: Mahendra Singh Thalor <[email protected]>
---
src/bin/pg_dump/Makefile | 5 +-
src/bin/pg_dump/connectdb.c | 294 +++++++++++++++++++++++++++
src/bin/pg_dump/connectdb.h | 26 +++
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/pg_backup.h | 6 +-
src/bin/pg_dump/pg_backup_archiver.c | 6 +-
src/bin/pg_dump/pg_backup_db.c | 79 +------
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 278 +------------------------
9 files changed, 352 insertions(+), 345 deletions(-)
create mode 100644 src/bin/pg_dump/connectdb.c
create mode 100644 src/bin/pg_dump/connectdb.h
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..fa795883e9f 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -31,6 +31,7 @@ OBJS = \
compress_lz4.o \
compress_none.o \
compress_zstd.o \
+ connectdb.o \
dumputils.o \
filter.o \
parallel.o \
@@ -50,8 +51,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
new file mode 100644
index 00000000000..9e593b70e81
--- /dev/null
+++ b/src/bin/pg_dump/connectdb.c
@@ -0,0 +1,294 @@
+/*-------------------------------------------------------------------------
+ *
+ * connectdb.c
+ * This is a common file connection to the database.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/connectdb.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "connectdb.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+
+/*
+ * ConnectDatabase
+ *
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the 'connstr' is set to a connection string containing
+ * the options used and 'server_version' is set to version so that caller
+ * can use them.
+ */
+PGconn *
+ConnectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version, char *password,
+ char *override_dbname)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 8;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ if (override_dbname)
+ {
+ keywords[i] = "dbname";
+ values[i++] = override_dbname;
+ }
+
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. If requested, remember the options used, in
+ * the form of a connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If requested, then copy server version to out variable. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/*
+ * constructConnStr
+ *
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * executeQuery
+ *
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
diff --git a/src/bin/pg_dump/connectdb.h b/src/bin/pg_dump/connectdb.h
new file mode 100644
index 00000000000..6c1e1954769
--- /dev/null
+++ b/src/bin/pg_dump/connectdb.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * connectdb.h
+ * Common header file for connection to the database.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/connectdb.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONNECTDB_H
+#define CONNECTDB_H
+
+#include "pg_backup.h"
+#include "pg_backup_utils.h"
+
+extern PGconn *ConnectDatabase(const char *dbname, const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version,
+ char *password, char *override_dbname);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+#endif /* CONNECTDB_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..25989e8f16b 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -6,6 +6,7 @@ pg_dump_common_sources = files(
'compress_lz4.c',
'compress_none.c',
'compress_zstd.c',
+ 'connectdb.c',
'dumputils.c',
'filter.c',
'parallel.c',
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 658986de6f8..49bc1ee71ef 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -293,9 +293,9 @@ typedef void (*SetupWorkerPtrType) (Archive *AH);
* Main archiver interface.
*/
-extern void ConnectDatabase(Archive *AHX,
- const ConnParams *cparams,
- bool isReconnect);
+extern void ConnectDatabaseAhx(Archive *AHX,
+ const ConnParams *cparams,
+ bool isReconnect);
extern void DisconnectDatabase(Archive *AHX);
extern PGconn *GetConnection(Archive *AHX);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 1d131e5a57d..3f59f8f9d9d 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -415,7 +415,7 @@ RestoreArchive(Archive *AHX)
AHX->minRemoteVersion = 0;
AHX->maxRemoteVersion = 9999999;
- ConnectDatabase(AHX, &ropt->cparams, false);
+ ConnectDatabaseAhx(AHX, &ropt->cparams, false);
/*
* If we're talking to the DB directly, don't send comments since they
@@ -4458,7 +4458,7 @@ restore_toc_entries_postfork(ArchiveHandle *AH, TocEntry *pending_list)
/*
* Now reconnect the single parent connection.
*/
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
/* re-establish fixed state */
_doSetFixedOutputState(AH);
@@ -5076,7 +5076,7 @@ CloneArchive(ArchiveHandle *AH)
* Connect our new clone object to the database, using the same connection
* parameters used for the original connection.
*/
- ConnectDatabase((Archive *) clone, &clone->public.ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) clone, &clone->public.ropt->cparams, true);
/* re-establish fixed state */
if (AH->mode == archModeRead)
diff --git a/src/bin/pg_dump/pg_backup_db.c b/src/bin/pg_dump/pg_backup_db.c
index 71c55d2466a..5c349279beb 100644
--- a/src/bin/pg_dump/pg_backup_db.c
+++ b/src/bin/pg_dump/pg_backup_db.c
@@ -19,6 +19,7 @@
#include "common/connect.h"
#include "common/string.h"
+#include "connectdb.h"
#include "parallel.h"
#include "pg_backup_archiver.h"
#include "pg_backup_db.h"
@@ -86,9 +87,9 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* ArchiveHandle's connCancel, before closing old connection. Otherwise
* an ill-timed SIGINT could try to access a dead connection.
*/
- AH->connection = NULL; /* dodge error check in ConnectDatabase */
+ AH->connection = NULL; /* dodge error check in ConnectDatabaseAhx */
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
PQfinish(oldConn);
}
@@ -105,14 +106,13 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* username never does change, so one savedPassword is sufficient.
*/
void
-ConnectDatabase(Archive *AHX,
- const ConnParams *cparams,
- bool isReconnect)
+ConnectDatabaseAhx(Archive *AHX,
+ const ConnParams *cparams,
+ bool isReconnect)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
trivalue prompt_password;
char *password;
- bool new_pass;
if (AH->connection)
pg_fatal("already connected to a database");
@@ -125,69 +125,10 @@ ConnectDatabase(Archive *AHX,
if (prompt_password == TRI_YES && password == NULL)
password = simple_prompt("Password: ", false);
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- const char *keywords[8];
- const char *values[8];
- int i = 0;
-
- /*
- * If dbname is a connstring, its entries can override the other
- * values obtained from cparams; but in turn, override_dbname can
- * override the dbname component of it.
- */
- keywords[i] = "host";
- values[i++] = cparams->pghost;
- keywords[i] = "port";
- values[i++] = cparams->pgport;
- keywords[i] = "user";
- values[i++] = cparams->username;
- keywords[i] = "password";
- values[i++] = password;
- keywords[i] = "dbname";
- values[i++] = cparams->dbname;
- if (cparams->override_dbname)
- {
- keywords[i] = "dbname";
- values[i++] = cparams->override_dbname;
- }
- keywords[i] = "fallback_application_name";
- values[i++] = progname;
- keywords[i] = NULL;
- values[i++] = NULL;
- Assert(i <= lengthof(keywords));
-
- new_pass = false;
- AH->connection = PQconnectdbParams(keywords, values, true);
-
- if (!AH->connection)
- pg_fatal("could not connect to database");
-
- if (PQstatus(AH->connection) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(AH->connection) &&
- password == NULL &&
- prompt_password != TRI_NO)
- {
- PQfinish(AH->connection);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(AH->connection) == CONNECTION_BAD)
- {
- if (isReconnect)
- pg_fatal("reconnection failed: %s",
- PQerrorMessage(AH->connection));
- else
- pg_fatal("%s",
- PQerrorMessage(AH->connection));
- }
+ AH->connection = ConnectDatabase(cparams->dbname, NULL, cparams->pghost,
+ cparams->pgport, cparams->username,
+ prompt_password, true,
+ progname, NULL, NULL, password, cparams->override_dbname);
/* Start strict; later phases may override this. */
PQclear(ExecuteSqlQueryForSingleRow((Archive *) AH,
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 84a78625820..bfa70369c47 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -966,7 +966,7 @@ main(int argc, char **argv)
* Open the database using the Archiver, so it knows about it. Errors mean
* death.
*/
- ConnectDatabase(fout, &dopt.cparams, false);
+ ConnectDatabaseAhx(fout, &dopt.cparams, false);
setup_connection(fout, dumpencoding, dumpsnapshot, use_role);
/*
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 2ea574b0f06..573a8b61a45 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -24,11 +24,11 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
-#include "pg_backup.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -71,21 +71,14 @@ static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
static char pg_dump_bin[MAXPGPATH];
-static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -129,8 +122,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -499,19 +490,22 @@ main(int argc, char *argv[])
*/
if (pgdb)
{
- conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase(pgdb, connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
}
else
{
- conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase("postgres", connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
- conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ conn = ConnectDatabase("template1", connstr, pghost, pgport, pguser,
+ prompt_password, true,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
{
@@ -1738,256 +1732,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
--
2.34.1
[text/x-patch] v20250330-0002-add-new-list-type-simple_oid_string_list-t.patch (2.6K, 3-v20250330-0002-add-new-list-type-simple_oid_string_list-t.patch)
download | inline diff:
From 17af42d16fa7f47a7e07a0e5bb02f323b224c680 Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email protected]>
Date: Fri, 28 Mar 2025 18:10:24 -0400
Subject: [PATCH v20250330 2/3] add new list type simple_oid_string_list to
fe-utils/simple_list
---
src/fe_utils/simple_list.c | 41 ++++++++++++++++++++++++++++++
src/include/fe_utils/simple_list.h | 16 ++++++++++++
2 files changed, 57 insertions(+)
diff --git a/src/fe_utils/simple_list.c b/src/fe_utils/simple_list.c
index 483d5455594..bbcc4ef618d 100644
--- a/src/fe_utils/simple_list.c
+++ b/src/fe_utils/simple_list.c
@@ -192,3 +192,44 @@ simple_ptr_list_destroy(SimplePtrList *list)
cell = next;
}
}
+
+/*
+ * Add to an oid_string list
+ */
+void
+simple_oid_string_list_append(SimpleOidStringList * list, Oid oid, const char *str)
+{
+ SimpleOidStringListCell *cell;
+
+ cell = (SimpleOidStringListCell *)
+ pg_malloc(offsetof(SimpleOidStringListCell, str) + strlen(str) + 1);
+
+ cell->next = NULL;
+ cell->oid = oid;
+ strcpy(cell->str, str);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * Destroy an oid_string list
+ */
+void
+simple_oid_string_list_destroy(SimpleOidStringList * list)
+{
+ SimpleOidStringListCell *cell;
+
+ cell = list->head;
+ while (cell != NULL)
+ {
+ SimpleOidStringListCell *next;
+
+ next = cell->next;
+ pg_free(cell);
+ cell = next;
+ }
+}
diff --git a/src/include/fe_utils/simple_list.h b/src/include/fe_utils/simple_list.h
index 3b8e38414ec..af61545d7ff 100644
--- a/src/include/fe_utils/simple_list.h
+++ b/src/include/fe_utils/simple_list.h
@@ -55,6 +55,19 @@ typedef struct SimplePtrList
SimplePtrListCell *tail;
} SimplePtrList;
+typedef struct SimpleOidStringListCell
+{
+ struct SimpleOidStringListCell *next;
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} SimpleOidStringListCell;
+
+typedef struct SimpleOidStringList
+{
+ SimpleOidStringListCell *head;
+ SimpleOidStringListCell *tail;
+} SimpleOidStringList;
+
extern void simple_oid_list_append(SimpleOidList *list, Oid val);
extern bool simple_oid_list_member(SimpleOidList *list, Oid val);
extern void simple_oid_list_destroy(SimpleOidList *list);
@@ -68,4 +81,7 @@ extern const char *simple_string_list_not_touched(SimpleStringList *list);
extern void simple_ptr_list_append(SimplePtrList *list, void *ptr);
extern void simple_ptr_list_destroy(SimplePtrList *list);
+extern void simple_oid_string_list_append(SimpleOidStringList * list, Oid oid, const char *str);
+extern void simple_oid_string_list_destroy(SimpleOidStringList * list);
+
#endif /* SIMPLE_LIST_H */
--
2.34.1
[text/x-patch] v20250330-0003-pg_dumpall-with-directory-tar-custom-forma.patch (59.2K, 4-v20250330-0003-pg_dumpall-with-directory-tar-custom-forma.patch)
download | inline diff:
From 95ddbd6726ebb0eb573a5fbaa80aac9f5a4e7cfb Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Wed, 19 Mar 2025 01:30:12 +0530
Subject: [PATCH v20250330 3/3] pg_dumpall with directory|tar|custom format and
restore it by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat to restore all databases. If global.dat file is exist in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list (if exist)
for --exclude-database=PATTERN for pg_restore
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
if no db connection, then PATTERN=NAME matching only
for each database, we are cleaning on_exit_nicely_index list.
at the end of restore, we are giving warning with total number of errors (including global.dat,
and each database errors) and for each database, we are printing warning with dbname and total
errors.
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
Author: Mahendra Singh Thalor <[email protected]>
---
doc/src/sgml/ref/pg_dumpall.sgml | 78 ++-
doc/src/sgml/ref/pg_restore.sgml | 41 +-
src/bin/pg_dump/parallel.c | 11 +-
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 20 +-
src/bin/pg_dump/pg_backup_archiver.h | 3 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 22 +-
src/bin/pg_dump/pg_backup_utils.h | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 284 ++++++++--
src/bin/pg_dump/pg_restore.c | 750 ++++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 9 +
src/tools/pgindent/typedefs.list | 2 +
14 files changed, 1152 insertions(+), 77 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 765b30a3a66..82ea2028469 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster based on specified dump format </refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -121,10 +121,86 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can be omitted only when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify format of dump files. If we want to dump all the databases,
+ then pass this as non-plain so that dump of all databases can be taken
+ in separate subdirectory in archive format.
+ by default, this is plain format.
+
+ If non-plain mode is passed, then global.dat (global sql commands) and
+ map.dat(dboid and dbnames list of all the databases) files will be created.
+ Apart from these files, one subdirectory with databases name will be created.
+ Under this databases subdirectory, there will be files with dboid name for each
+ database and if <option>--format</option> is directory, then toc.dat and other
+ dump files will be under dboid subdirectory.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output a directory-format archive suitable for input into pg_restore. Under dboid
+ subdirectory, this will create a directory with one file for each table and large
+ object being dumped, plus a so-called Table of Contents file describing the dumped
+ objects in a machine-readable format that pg_restore can read. A directory format
+ archive can be manipulated with standard Unix tools; for example, files in an
+ uncompressed archive can be compressed with the gzip, lz4, or zstd tools. This
+ format is compressed by default using gzip and also supports parallel dumps.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive suitable for input into pg_restore. Together with the
+ directory output format, this is the most flexible output format in that it allows manual
+ selection and reordering of archived items during restore. This format is also
+ compressed by default.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive suitable for input into pg_restore. The tar format is
+ compatible with the directory format: extracting a tar-format archive produces a valid
+ directory-format archive. However, the tar format does not support compression. Also,
+ when using tar format the relative order of table data items cannot be changed during restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index c840a807ae9..f0a24134595 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore <productname>PostgreSQL</productname> database from an
+ archive file created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -37,9 +38,10 @@ PostgreSQL documentation
<title>Description</title>
<para>
- <application>pg_restore</application> is a utility for restoring a
+ <application>pg_restore</application> is a utility for restoring
<productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
database to the state it was in at the time it was saved. The
archive files also allow <application>pg_restore</application> to
@@ -140,6 +142,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from dump of <application>pg_dumpall</application>.
</para>
<para>
@@ -166,6 +170,25 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +338,16 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..a36d2a5bf84 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -326,11 +326,18 @@ getThreadLocalPQExpBuffer(void)
* pg_dump and pg_restore call this to register the cleanup handler
* as soon as they've created the ArchiveHandle.
*/
-void
+int
on_exit_close_archive(Archive *AHX)
{
shutdown_info.AHX = AHX;
- on_exit_nicely(archive_close_connection, &shutdown_info);
+ return on_exit_nicely(archive_close_connection, &shutdown_info);
+}
+
+void
+replace_on_exit_close_archive(Archive *AHX, int idx)
+{
+ shutdown_info.AHX = AHX;
+ set_on_exit_nicely_entry(archive_close_connection, &shutdown_info, idx);
}
/*
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 49bc1ee71ef..17d6e06ec25 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -311,7 +311,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 3f59f8f9d9d..54eb4728928 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -85,7 +85,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -338,9 +338,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -457,7 +462,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1293,7 +1298,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1672,7 +1677,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1692,7 +1698,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index a2064f471ed..dc045b852e9 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -385,7 +385,8 @@ struct _tocEntry
};
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
-extern void on_exit_close_archive(Archive *AHX);
+extern int on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX, int idx);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..59ece2999a8 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -61,14 +61,26 @@ set_dump_section(const char *arg, int *dumpSections)
/* Register a callback to be run when exit_nicely is invoked. */
-void
+int
on_exit_nicely(on_exit_nicely_callback function, void *arg)
{
- if (on_exit_nicely_index >= MAX_ON_EXIT_NICELY)
- pg_fatal("out of on_exit_nicely slots");
- on_exit_nicely_list[on_exit_nicely_index].function = function;
- on_exit_nicely_list[on_exit_nicely_index].arg = arg;
+ set_on_exit_nicely_entry(function, arg, on_exit_nicely_index);
on_exit_nicely_index++;
+
+ return (on_exit_nicely_index - 1);
+}
+
+void
+set_on_exit_nicely_entry(on_exit_nicely_callback function, void *arg, int i)
+{
+ if (i >= MAX_ON_EXIT_NICELY)
+ pg_fatal("out of on_exit_nicely slots");
+
+ if (i > on_exit_nicely_index)
+ pg_fatal("no entry exists on %d index into on_exit_nicely slots", i);
+
+ on_exit_nicely_list[i].function = function;
+ on_exit_nicely_list[i].arg = arg;
}
/*
diff --git a/src/bin/pg_dump/pg_backup_utils.h b/src/bin/pg_dump/pg_backup_utils.h
index ba042016879..1ce1077096d 100644
--- a/src/bin/pg_dump/pg_backup_utils.h
+++ b/src/bin/pg_dump/pg_backup_utils.h
@@ -28,7 +28,8 @@ typedef void (*on_exit_nicely_callback) (int code, void *arg);
extern const char *progname;
extern void set_dump_section(const char *arg, int *dumpSections);
-extern void on_exit_nicely(on_exit_nicely_callback function, void *arg);
+extern int on_exit_nicely(on_exit_nicely_callback function, void *arg);
+extern void set_on_exit_nicely_entry(on_exit_nicely_callback function, void *arg, int idx);
pg_noreturn extern void exit_nicely(int code);
/* In pg_dump, we modify pg_fatal to call exit_nicely instead of exit */
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index bfa70369c47..d8a62736ef1 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1219,7 +1219,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 573a8b61a45..7ceaba27419 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -64,9 +65,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -75,6 +77,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -146,6 +150,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -195,6 +200,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -244,7 +251,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -272,7 +279,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -421,6 +430,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -483,6 +507,33 @@ main(int argc, char *argv[])
if (statistics_only)
appendPQExpBufferStr(pgdumpopts, " --statistics-only");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char global_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(global_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -522,19 +573,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -634,7 +672,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -647,7 +685,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -658,12 +696,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster based on specified dump format.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -1570,10 +1610,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1587,7 +1630,7 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
@@ -1595,9 +1638,34 @@ dumpDatabases(PGconn *conn)
if (PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified then create a subdirectory
+ * under the main directory and each database dump file subdirectory will
+ * be created under the subdirectory in archive mode as per single db
+ * pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_fatal("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
+ char *oid = PQgetvalue(res, i, 1);
const char *create_opts;
int ret;
@@ -1612,6 +1680,23 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
@@ -1630,9 +1715,17 @@ dumpDatabases(PGconn *conn)
create_opts = "--clean --create";
else
{
- create_opts = "";
/* Since pg_dump won't emit a \connect command, we must */
- fprintf(OPF, "\\connect %s\n\n", dbname);
+ if (archDumpFormat == archNull)
+ {
+ create_opts = "";
+ fprintf(OPF, "\\connect %s\n\n", dbname);
+ }
+ else
+ {
+ /* Dumping all databases so add --create option. */
+ create_opts = "--create";
+ }
}
}
else
@@ -1641,19 +1734,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char global_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(global_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ global_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1663,7 +1767,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1672,17 +1777,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1827,3 +1951,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if (!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 47f7b0dd3a1..ce70b7e12b2 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,11 +41,15 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -53,18 +57,36 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num);
+static int read_one_statement(StringInfo inBuf, FILE *pfile);
+static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static void copy_or_print_global_file(const char *outfile, FILE *pfile);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimpleOidStringList * dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleOidStringList * dbname_oid_list);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
+static int on_exit_index = 0;
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -90,6 +112,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -144,6 +167,7 @@ main(int argc, char **argv)
{"with-statistics", no_argument, &with_statistics, 1},
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -172,7 +196,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -199,11 +223,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -318,6 +345,9 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -345,6 +375,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -452,6 +489,114 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL && !file_exists_in_directory(inputFileSpec, "toc.dat") &&
+ file_exists_in_directory(inputFileSpec, "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql
+ * commands. */
+
+ /*
+ * User is suggested to use single database dump for --list option.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring multiple databases by archive of pg_dumpall");
+
+ /*
+ * To restore multiple databases, -C (create database) option should
+ * be specified. Even there is single database in dump, report error
+ * because it might be possible that database hasn't created so better
+ * we report error.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("-C/--create option should be specified when restoring multiple databases by archive of pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("If db is already created and dump has single db dump, then use particular dump file.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Connect to database to execute global sql commands from global.dat
+ * file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ /*
+ * Open global.dat file and execute/append all the global sql
+ * commands.
+ */
+ n_errors = process_global_sql_commands(conn, inputFileSpec,
+ opts->filename);
+
+ if (conn)
+ PQfinish(conn);
+
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ n_errors = restoreAllDatabases(conn, inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else /* process if global.dat file does not exist. */
+ {
+ if (db_exclude_patterns.head != NULL)
+ pg_fatal("option --exclude-database can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ n_errors = restoreOneDatabase(inputFileSpec, opts, numWorkers, false, 0);
+ }
+
+ on_exit_index = 0; /* Reset index. */
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restoreOneDatabase
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restoreOneDatabase(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -459,9 +604,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then save index of exit_nicely so that we
+ * can use same slot for all the databases as we already closed the
+ * previous archive by CloseArchive.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_index = on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH, on_exit_index);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -481,25 +632,22 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n"
+ "If archive is created by pg_dumpall, then restores multiple databases also. \n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -517,6 +665,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -529,6 +678,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -569,8 +719,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be combined\n"
+ "and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -675,3 +825,569 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * read_one_statement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.dat file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+read_one_statement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+ int m;
+
+ StringInfoData q;
+
+ initStringInfo(&q);
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != '\n' && c != ';')
+ {
+ appendStringInfoChar(inBuf, (char) c);
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != ';' && c != '\n')
+ appendStringInfoChar(inBuf, (char) c);
+ else
+ break;
+ }
+ }
+
+ if (c == '\'' || c == '"')
+ {
+ appendStringInfoChar(&q, (char) c);
+ m = c;
+
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(&q, (char) c);
+
+ if (c == m)
+ {
+ appendStringInfoString(inBuf, q.data);
+ resetStringInfo(&q);
+ break;
+ }
+ }
+ }
+
+ if (c == ';')
+ {
+ appendStringInfoChar(inBuf, (char) ';');
+ break;
+ }
+
+ if (c == '\n')
+ appendStringInfoChar(inBuf, (char) '\n');
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimpleOidStringList * dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimpleOidStringListCell * db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ bool skip_db_restore = false;
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * the construct pattern matching query: SELECT 1 WHERE XXX
+ * OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from
+ * the dbname_oid_list, which is initially extracted from the
+ * map.dat file located in the backup directory. that's why we
+ * need quote_literal_cstr.
+ *
+ * If no db connection, then consider PATTERN as NAME.
+ */
+ if (pg_strcasecmp(db_cell->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, quote_literal_cstr(db_cell->str),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ db_cell->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database \"%s\" matches exclude pattern: \"%s\"", db_cell->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /* Increment count if database needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", db_cell->str);
+ db_cell->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleOidStringList * dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ /*
+ * If there is only global.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while ((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid = InvalidOid;
+ char db_oid_str[MAXPGPATH + 1] = {'\0'};
+ char dbname[MAXPGPATH + 1] = {'\0'};
+
+ /* Extract dboid. */
+ sscanf(line, "%u", &db_oid);
+ sscanf(line, "%20s", db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found database \"%s\" (OID: %u) in map.dat file while restoring.", dbname, db_oid);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making a
+ * list of all the databases.
+ */
+ simple_oid_string_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restoreAllDatabases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleOidStringList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /*
+ * If map.dat has no entry, return from here after processing global.dat
+ * file.
+ */
+ if (dbname_oid_list.head == NULL)
+ return process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+
+ /*
+ * processing pg_retsore --exclude-database=PATTERN/NAME if no connection.
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ n_errors_total = process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info("no database needs to restore out of %d databases", num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * Till now, we made a list of databases, those needs to be restored after
+ * skipping names of exclude-database. Now we can launch parallel workers
+ * to restore these databases.
+ */
+ for (SimpleOidStringListCell * db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (db_cell->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", dumpdirpath);
+
+ /*
+ * Validate database dump file. If there is .tar or .dmp file exist
+ * then consider particular file, otherwise just append dboid to the
+ * databases folder.
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", db_cell->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", dumpdirpath, db_cell->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", db_cell->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", dumpdirpath, db_cell->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, db_cell->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", db_cell->str);
+
+ /* Restore single database. */
+ n_errors = restoreOneDatabase(subdirpath, opts, numWorkers, true, count);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", db_cell->str, n_errors);
+ }
+
+ count++;
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_oid_string_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
+
+/*
+ * process_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ *
+ * returns the number of errors while processing global.dat
+ */
+static int
+process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement;
+ FILE *pfile;
+ int n_errors = 0;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ copy_or_print_global_file(outfile, pfile);
+ return 0;
+ }
+
+ /* Init sqlstatement to append commands. */
+ initStringInfo(&sqlstatement);
+
+ /* Process file till EOF and execute sql statements. */
+ while (read_one_statement(&sqlstatement, pfile) != EOF)
+ {
+ pg_log_info("executing query: %s", sqlstatement.data);
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ n_errors++;
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ /* Print a summary of ignored errors during global.dat. */
+ if (n_errors)
+ pg_log_warning("errors ignored on global.dat file restore: %d", n_errors);
+
+ fclose(pfile);
+
+ return n_errors;
+}
+
+/*
+ * copy_or_print_global_file
+ *
+ * This will copy global.dat file into out file. If "-" is used as outfile,
+ * then print commands to the stdout.
+ */
+static void
+copy_or_print_global_file(const char *outfile, FILE *pfile)
+{
+ char out_file_path[MAXPGPATH];
+ FILE *OPF;
+ int c;
+
+ /* "-" is used for stdout. */
+ if (strcmp(outfile, "-") == 0)
+ OPF = stdout;
+ else
+ {
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ OPF = fopen(out_file_path, PG_BINARY_W);
+
+ if (OPF == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", outfile);
+ }
+ }
+
+ /* Append global.dat into out file or print to the stdout. */
+ while ((c = fgetc(pfile)) != EOF)
+ fputc(c, OPF);
+
+ fclose(pfile);
+
+ /* Close out file. */
+ if (strcmp(outfile, "-") != 0)
+ fclose(OPF);
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr
+ *
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..0bbcdbe84a7
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,11 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +249,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b66cecd8799..95ec8fbb141 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2732,6 +2732,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.34.1
[text/plain] dumpall_cleanup.patch2-noci (2.0K, 5-dumpall_cleanup.patch2-noci)
download | inline diff:
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 7ceaba27419..7a06e595b88 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -1639,10 +1639,9 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
fprintf(OPF, "--\n-- Databases\n--\n\n");
/*
- * If directory/tar/custom format is specified then create a subdirectory
- * under the main directory and each database dump file subdirectory will
- * be created under the subdirectory in archive mode as per single db
- * pg_dump.
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
*/
if (archDumpFormat != archNull)
{
@@ -1666,7 +1665,7 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
char *dbname = PQgetvalue(res, i, 0);
char *oid = PQgetvalue(res, i, 1);
- const char *create_opts;
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1699,7 +1698,8 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
pg_log_info("dumping database \"%s\"", dbname);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
/*
* We assume that "template1" and "postgres" already exist in the
@@ -1713,20 +1713,8 @@ dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
if (output_clean)
create_opts = "--clean --create";
- else
- {
- /* Since pg_dump won't emit a \connect command, we must */
- if (archDumpFormat == archNull)
- {
- create_opts = "";
- fprintf(OPF, "\\connect %s\n\n", dbname);
- }
- else
- {
- /* Dumping all databases so add --create option. */
- create_opts = "--create";
- }
- }
+ else if (archDumpFormat == archNull)
+ fprintf(OPF, "\\connect %s\n\n", dbname);
}
else
create_opts = "--create";
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-27 21:15 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-28 22:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-29 05:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-30 16:50 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
@ 2025-03-30 22:05 ` Andrew Dunstan <[email protected]>
1 sibling, 0 replies; 29+ messages in thread
From: Andrew Dunstan @ 2025-03-30 22:05 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: jian he <[email protected]>; Álvaro Herrera <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-03-30 Su 12:50 PM, Andrew Dunstan wrote:
>
> On 2025-03-29 Sa 1:17 AM, Mahendra Singh Thalor wrote:
>> On Sat, 29 Mar 2025 at 03:50, Andrew Dunstan <[email protected]>
>> wrote:
>>>
>>> On 2025-03-27 Th 5:15 PM, Andrew Dunstan wrote:
>>>> On 2025-03-19 We 2:41 AM, Mahendra Singh Thalor wrote:
>>>>> On Wed, 12 Mar 2025 at 21:18, Andrew Dunstan <[email protected]>
>>>>> wrote:
>>>>>> On 2025-03-12 We 3:03 AM, jian he wrote:
>>>>>>> On Wed, Mar 12, 2025 at 1:06 AM Álvaro Herrera
>>>>>>> <[email protected]> wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> On 2025-Mar-11, Mahendra Singh Thalor wrote:
>>>>>>>>
>>>>>>>>> In map.dat file, I tried to fix this issue by adding number of
>>>>>>>>> characters
>>>>>>>>> in dbname but as per code comments, as of now, we are not
>>>>>>>>> supporting \n\r
>>>>>>>>> in dbnames so i removed handling.
>>>>>>>>> I will do some more study to fix this issue.
>>>>>>>> Yeah, I think this is saying that you should not consider the
>>>>>>>> contents
>>>>>>>> of map.dat as a shell string. After all, you're not going to
>>>>>>>> _execute_
>>>>>>>> that file via the shell.
>>>>>>>>
>>>>>>>> Maybe for map.dat you need to escape such characters somehow,
>>>>>>>> so that
>>>>>>>> they don't appear as literal newlines/carriage returns.
>>>>>>>>
>>>>>>> I am confused.
>>>>>>> currently pg_dumpall plain format will abort when encountering
>>>>>>> dbname
>>>>>>> containing newline.
>>>>>>> the left dumped plain file does not contain all the cluster
>>>>>>> databases data.
>>>>>>>
>>>>>>>
>>>>>>> if pg_dumpall non-text format aborts earlier,
>>>>>>> it's aligned with pg_dumpall plain format?
>>>>>>> it's also an improvement since aborts earlier, nothing will be
>>>>>>> dumped?
>>>>>>>
>>>>>>>
>>>>>>> am i missing something?
>>>>>>>
>>>>>>>
>>>>>> I think we should fix that.
>>>>>>
>>>>>> But for the current proposal, Álvaro and I were talking this
>>>>>> morning,
>>>>>> and we thought the simplest thing here would be to have the one line
>>>>>> format and escape NL/CRs in the database name.
>>>>>>
>>>>>>
>>>>>> cheers
>>>>>>
>>>>> Okay. As per discussions, we will keep one line entry for each
>>>>> database into map.file.
>>>>>
>>>>> Thanks all for feedback and review.
>>>>>
>>>>> Here, I am attaching updated patches for review and testing. These
>>>>> patches can be applied on commit a6524105d20b.
>>>>
>>>>
>>>> I'm working through this patch set with a view to committing it.
>>>> Attached is some cleanup which is where I got to today, although there
>>>> is more to do. One thing I am wondering is why not put the
>>>> SimpleDatabaseOidList stuff in fe_utils/simle_list.{c,h} ? That's
>>>> where all the similar stuff belongs, and it feels strange to have this
>>>> inline in pg_restore.c. (I also don't like the name much -
>>>> SimpleOidStringList or maybe SimpleOidPlusStringList might be better).
>>>>
>>>>
>>>>
>>>
>>> OK, I have done that, so here is the result. The first two are you
>>> original patches. patch 3 adds the new list type to fe-utils, and patch
>>> 4 contains my cleanups and use of the new list type. Apart from some
>>> relatively minor cleanup, the one thing I would like to change is how
>>> dumps are named. If we are producing tar or custom format dumps, I
>>> think
>>> the file names should reflect that (oid.dmp and oid.tar rather than a
>>> bare oid as the filename), and pg_restore should look for those. I'm
>>> going to work on that tomorrow - I don't think it will be terribly
>>> difficult.
>>>
>> Thanks Andrew.
>>
>> Here, I am attaching a delta patch for oid.tar and oid.dmp format.
>>
>
>
> OK, looks good, I have incorporated that.
>
> There are a couple of rough edges, though.
>
> First, I see this:
>
>
> andrew@ub22arm:inst $ bin/pg_restore -C -d postgres
> --exclude-database=regression_dummy_seclabel
> --exclude-database=regression_test_extensions
> --exclude-database=regression_test_pg_dump dest
> pg_restore: error: could not execute query: "ERROR: role "andrew"
> already exists
> "
> Command was: "
>
> --
> -- Roles
> --
>
> CREATE ROLE andrew;"
> pg_restore: warning: errors ignored on global.dat file restore: 1
> pg_restore: error: could not execute query: ERROR: database
> "template1" already exists
> Command was: CREATE DATABASE template1 WITH TEMPLATE = template0
> ENCODING = 'SQL_ASCII' LOCALE_PROVIDER = libc LOCALE = 'C';
>
>
> pg_restore: warning: errors ignored on database "template1" restore: 1
> pg_restore: error: could not execute query: ERROR: database
> "postgres" already exists
> Command was: CREATE DATABASE postgres WITH TEMPLATE = template0
> ENCODING = 'SQL_ASCII' LOCALE_PROVIDER = libc LOCALE = 'C';
>
>
> pg_restore: warning: errors ignored on database "postgres" restore: 1
> pg_restore: warning: errors ignored on restore: 3
>
>
>
> It seems pointless to be trying to create the rolw that we are
> connected as, and we also expect template1 and postgres to exist.
>
> In a similar vein, I don't see why we are setting the --create flag in
> pg_dumpall for those databases. I'm attaching a patch that is designed
> to stop that, but it doesn't solve the above issues.
>
> I also notice a bunch of these in globals.dat:
>
>
> --
> -- Databases
> --
>
> --
> -- Database "template1" dump
> --
>
> --
> -- Database "andrew" dump
> --
>
> --
> -- Database "isolation_regression_brin" dump
> --
>
> --
> -- Database "isolation_regression_delay_execution" dump
> --
>
> ...
>
>
> The patch also tries to fix this.
>
> Lastly, this badly needs some TAP tests written.
>
> I'm going to work on reviewing the documentation next.
>
>
>
I have reworked the documentation some. See attached.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 765b30a3a66..43fdab2d77e 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster using a specified dump format</refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +33,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an archive. The archive contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +52,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -121,10 +126,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>global.dat</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index c840a807ae9..f14e5866f6c 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore a <productname>PostgreSQL</productname> database or cluster
+ from an archive created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by<application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -140,6 +149,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -166,6 +177,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +348,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
Attachments:
[text/plain] dumpall-docs.patch.noci (10.1K, 2-dumpall-docs.patch.noci)
download | inline diff:
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 765b30a3a66..43fdab2d77e 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster using a specified dump format</refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +33,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an archive. The archive contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +52,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -121,10 +126,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>global.dat</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index c840a807ae9..f14e5866f6c 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore a <productname>PostgreSQL</productname> database or cluster
+ from an archive created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by<application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -140,6 +149,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -166,6 +177,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +348,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-27 21:15 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-28 22:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-29 05:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-30 16:50 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
@ 2025-03-31 09:34 ` Mahendra Singh Thalor <[email protected]>
2025-03-31 13:57 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
1 sibling, 1 reply; 29+ messages in thread
From: Mahendra Singh Thalor @ 2025-03-31 09:34 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: jian he <[email protected]>; Álvaro Herrera <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Sun, 30 Mar 2025 at 22:20, Andrew Dunstan <[email protected]> wrote:
>
>
> On 2025-03-29 Sa 1:17 AM, Mahendra Singh Thalor wrote:
> > On Sat, 29 Mar 2025 at 03:50, Andrew Dunstan <[email protected]>
wrote:
> >>
> >> On 2025-03-27 Th 5:15 PM, Andrew Dunstan wrote:
> >>> On 2025-03-19 We 2:41 AM, Mahendra Singh Thalor wrote:
> >>>> On Wed, 12 Mar 2025 at 21:18, Andrew Dunstan <[email protected]>
> >>>> wrote:
> >>>>> On 2025-03-12 We 3:03 AM, jian he wrote:
> >>>>>> On Wed, Mar 12, 2025 at 1:06 AM Álvaro Herrera
> >>>>>> <[email protected]> wrote:
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> On 2025-Mar-11, Mahendra Singh Thalor wrote:
> >>>>>>>
> >>>>>>>> In map.dat file, I tried to fix this issue by adding number of
> >>>>>>>> characters
> >>>>>>>> in dbname but as per code comments, as of now, we are not
> >>>>>>>> supporting \n\r
> >>>>>>>> in dbnames so i removed handling.
> >>>>>>>> I will do some more study to fix this issue.
> >>>>>>> Yeah, I think this is saying that you should not consider the
> >>>>>>> contents
> >>>>>>> of map.dat as a shell string. After all, you're not going to
> >>>>>>> _execute_
> >>>>>>> that file via the shell.
> >>>>>>>
> >>>>>>> Maybe for map.dat you need to escape such characters somehow, so
that
> >>>>>>> they don't appear as literal newlines/carriage returns.
> >>>>>>>
> >>>>>> I am confused.
> >>>>>> currently pg_dumpall plain format will abort when encountering
dbname
> >>>>>> containing newline.
> >>>>>> the left dumped plain file does not contain all the cluster
> >>>>>> databases data.
> >>>>>>
> >>>>>>
> >>>>>> if pg_dumpall non-text format aborts earlier,
> >>>>>> it's aligned with pg_dumpall plain format?
> >>>>>> it's also an improvement since aborts earlier, nothing will be
dumped?
> >>>>>>
> >>>>>>
> >>>>>> am i missing something?
> >>>>>>
> >>>>>>
> >>>>> I think we should fix that.
> >>>>>
> >>>>> But for the current proposal, Álvaro and I were talking this
morning,
> >>>>> and we thought the simplest thing here would be to have the one line
> >>>>> format and escape NL/CRs in the database name.
> >>>>>
> >>>>>
> >>>>> cheers
> >>>>>
> >>>> Okay. As per discussions, we will keep one line entry for each
> >>>> database into map.file.
> >>>>
> >>>> Thanks all for feedback and review.
> >>>>
> >>>> Here, I am attaching updated patches for review and testing. These
> >>>> patches can be applied on commit a6524105d20b.
> >>>
> >>>
> >>> I'm working through this patch set with a view to committing it.
> >>> Attached is some cleanup which is where I got to today, although there
> >>> is more to do. One thing I am wondering is why not put the
> >>> SimpleDatabaseOidList stuff in fe_utils/simle_list.{c,h} ? That's
> >>> where all the similar stuff belongs, and it feels strange to have this
> >>> inline in pg_restore.c. (I also don't like the name much -
> >>> SimpleOidStringList or maybe SimpleOidPlusStringList might be better).
> >>>
> >>>
> >>>
> >>
> >> OK, I have done that, so here is the result. The first two are you
> >> original patches. patch 3 adds the new list type to fe-utils, and patch
> >> 4 contains my cleanups and use of the new list type. Apart from some
> >> relatively minor cleanup, the one thing I would like to change is how
> >> dumps are named. If we are producing tar or custom format dumps, I
think
> >> the file names should reflect that (oid.dmp and oid.tar rather than a
> >> bare oid as the filename), and pg_restore should look for those. I'm
> >> going to work on that tomorrow - I don't think it will be terribly
> >> difficult.
> >>
> > Thanks Andrew.
> >
> > Here, I am attaching a delta patch for oid.tar and oid.dmp format.
> >
>
>
> OK, looks good, I have incorporated that.
>
> There are a couple of rough edges, though.
>
> First, I see this:
>
>
> andrew@ub22arm:inst $ bin/pg_restore -C -d postgres
> --exclude-database=regression_dummy_seclabel
> --exclude-database=regression_test_extensions
> --exclude-database=regression_test_pg_dump dest
> pg_restore: error: could not execute query: "ERROR: role "andrew"
> already exists
> "
> Command was: "
>
> --
> -- Roles
> --
>
> CREATE ROLE andrew;"
> pg_restore: warning: errors ignored on global.dat file restore: 1
> pg_restore: error: could not execute query: ERROR: database "template1"
> already exists
> Command was: CREATE DATABASE template1 WITH TEMPLATE = template0
> ENCODING = 'SQL_ASCII' LOCALE_PROVIDER = libc LOCALE = 'C';
>
>
> pg_restore: warning: errors ignored on database "template1" restore: 1
> pg_restore: error: could not execute query: ERROR: database "postgres"
> already exists
> Command was: CREATE DATABASE postgres WITH TEMPLATE = template0 ENCODING
> = 'SQL_ASCII' LOCALE_PROVIDER = libc LOCALE = 'C';
>
>
> pg_restore: warning: errors ignored on database "postgres" restore: 1
> pg_restore: warning: errors ignored on restore: 3
>
>
>
> It seems pointless to be trying to create the rolw that we are connected
> as, and we also expect template1 and postgres to exist.
Thanks Andrew for the updated patches.
Here, I am attaching a delta patch which solves the errors for the
already created database and we need to reset some flags also. Please have
a look over this delta patch and merge it.
If we want to skip errors for connected user (CREATE ROLE username), then
we need to handle it by comparing sql commands in
process_global_sql_commands function or we can compare errors after
executing it.
delta_0002* patch is doing some handling but this is not a proper fix.
I think we can merge delta_0001* and later, we can work on delta_0002.
>
> In a similar vein, I don't see why we are setting the --create flag in
> pg_dumpall for those databases. I'm attaching a patch that is designed
> to stop that, but it doesn't solve the above issues.
>
> I also notice a bunch of these in globals.dat:
>
>
> --
> -- Databases
> --
>
> --
> -- Database "template1" dump
> --
>
> --
> -- Database "andrew" dump
> --
>
> --
> -- Database "isolation_regression_brin" dump
> --
>
> --
> -- Database "isolation_regression_delay_execution" dump
> --
>
> ...
>
>
> The patch also tries to fix this.
>
> Lastly, this badly needs some TAP tests written.
>
> I'm going to work on reviewing the documentation next.
Thank you.
>
>
> cheers
>
>
> andrew
>
> --
> Andrew Dunstan
> EDB: https://www.enterprisedb.com
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] delta_0002-pg_restore-skip-error-for-CRETE-ROLE-username.patch (3.7K, 3-delta_0002-pg_restore-skip-error-for-CRETE-ROLE-username.patch)
download | inline diff:
From f795accf5fede15476300a43ea27135708b5d1e1 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Mon, 31 Mar 2025 14:59:13 +0530
Subject: [PATCH] pg_restore: skip error for CRETE ROLE username
---
src/bin/pg_dump/pg_restore.c | 32 +++++++++++++++++++++++++++-----
1 file changed, 27 insertions(+), 5 deletions(-)
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 3d8be43241d..8ce5b790e51 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -64,7 +64,7 @@ static int read_one_statement(StringInfo inBuf, FILE *pfile);
static int restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
static int process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
- const char *outfile);
+ const char *outfile, const char *username);
static void copy_or_print_global_file(const char *outfile, FILE *pfile);
static int get_dbnames_list_to_restore(PGconn *conn,
SimpleOidStringList * dbname_oid_list,
@@ -543,7 +543,7 @@ main(int argc, char **argv)
* commands.
*/
n_errors = process_global_sql_commands(conn, inputFileSpec,
- opts->filename);
+ opts->filename, opts->cparams.username);
if (conn)
PQfinish(conn);
@@ -1123,7 +1123,7 @@ restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
* file.
*/
if (dbname_oid_list.head == NULL)
- return process_global_sql_commands(conn, dumpdirpath, opts->filename);
+ return process_global_sql_commands(conn, dumpdirpath, opts->filename, opts->cparams.username);
pg_log_info("found total %d database names in map.dat file", num_total_db);
@@ -1153,7 +1153,7 @@ restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
db_exclude_patterns);
/* Open global.dat file and execute/append all the global sql commands. */
- n_errors_total = process_global_sql_commands(conn, dumpdirpath, opts->filename);
+ n_errors_total = process_global_sql_commands(conn, dumpdirpath, opts->filename, opts->cparams.username);
/* Close the db connection as we are done with globals and patterns. */
if (conn)
@@ -1280,11 +1280,13 @@ restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
* returns the number of errors while processing global.dat
*/
static int
-process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile, const char *username)
{
char global_file_path[MAXPGPATH];
PGresult *result;
StringInfoData sqlstatement;
+ StringInfoData rolesqlstatement;
FILE *pfile;
int n_errors = 0;
@@ -1308,10 +1310,30 @@ process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *o
/* Init sqlstatement to append commands. */
initStringInfo(&sqlstatement);
+ initStringInfo(&rolesqlstatement);
/* Process file till EOF and execute sql statements. */
while (read_one_statement(&sqlstatement, pfile) != EOF)
{
+ if (username)
+ {
+ appendStringInfoString(&rolesqlstatement, "\n\n--\n-- Roles\n--\n\nCREATE ROLE ");
+ appendStringInfoString(&rolesqlstatement, username);
+ appendStringInfoString(&rolesqlstatement, ";");
+ }
+
+ /*
+ * If this command is for "CREATE ROLE username", then skip this as
+ * current user is already created.
+ */
+ if (username && strcmp(sqlstatement.data, rolesqlstatement.data) == 0)
+ {
+ resetStringInfo(&rolesqlstatement);
+ continue;
+ }
+
+ resetStringInfo(&rolesqlstatement);
+
pg_log_info("executing query: %s", sqlstatement.data);
result = PQexec(conn, sqlstatement.data);
--
2.39.3
[application/octet-stream] delta-0001-pg_restore-skip-error-if-db-already-created.patch (2.3K, 4-delta-0001-pg_restore-skip-error-if-db-already-created.patch)
download | inline diff:
From eb1bd481d4216bb27db227410cc48edc4bf2634e Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Mon, 31 Mar 2025 14:01:41 +0530
Subject: [PATCH] pg_restore: if database is already created, then set createdb
as 0
If database is already created, then set createdb as 0 so that user
will not get errors.
Also reset some flags for each database.
as dumpData, dumpSchema, dumpStatistics
---
src/bin/pg_dump/pg_restore.c | 38 ++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index ce70b7e12b2..3d8be43241d 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -1107,6 +1107,14 @@ restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
int num_total_db;
int n_errors_total;
int count = 0;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+
+ /* Save db name. */
+ if (opts->cparams.dbname)
+ connected_db = pg_strdup(opts->cparams.dbname);
num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
@@ -1209,9 +1217,39 @@ restoreAllDatabases(PGconn *conn, const char *dumpdirpath,
pg_log_info("restoring database \"%s\"", db_cell->str);
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(db_cell->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ if (conn)
+ {
+ opts->createDB = 0;
+ PQfinish(conn);
+
+ /* Use already created database for connection. */
+ if (opts->cparams.dbname)
+ opts->cparams.dbname = pg_strdup(db_cell->str);
+ }
+ }
+
/* Restore single database. */
n_errors = restoreOneDatabase(subdirpath, opts, numWorkers, true, count);
+ /* Set opts->createDB flag. */
+ if (opts->createDB == 0)
+ {
+ opts->createDB = 1;
+ opts->cparams.dbname = pg_strdup(connected_db);
+ }
+
+ /* Reset flags for next database. */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
/* Print a summary of ignored errors during single database restore. */
if (n_errors)
{
--
2.39.3
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-27 21:15 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-28 22:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-29 05:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-30 16:50 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 09:34 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-03-31 13:57 ` Andrew Dunstan <[email protected]>
2025-03-31 16:16 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Dunstan @ 2025-03-31 13:57 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: jian he <[email protected]>; Álvaro Herrera <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-03-31 Mo 5:34 AM, Mahendra Singh Thalor wrote:
>
> >
> > There are a couple of rough edges, though.
> >
> > First, I see this:
> >
> >
> > andrew@ub22arm:inst $ bin/pg_restore -C -d postgres
> > --exclude-database=regression_dummy_seclabel
> > --exclude-database=regression_test_extensions
> > --exclude-database=regression_test_pg_dump dest
> > pg_restore: error: could not execute query: "ERROR: role "andrew"
> > already exists
> > "
> > Command was: "
> >
> > --
> > -- Roles
> > --
> >
> > CREATE ROLE andrew;"
> > pg_restore: warning: errors ignored on global.dat file restore: 1
> > pg_restore: error: could not execute query: ERROR: database "template1"
> > already exists
> > Command was: CREATE DATABASE template1 WITH TEMPLATE = template0
> > ENCODING = 'SQL_ASCII' LOCALE_PROVIDER = libc LOCALE = 'C';
> >
> >
> > pg_restore: warning: errors ignored on database "template1" restore: 1
> > pg_restore: error: could not execute query: ERROR: database "postgres"
> > already exists
> > Command was: CREATE DATABASE postgres WITH TEMPLATE = template0 ENCODING
> > = 'SQL_ASCII' LOCALE_PROVIDER = libc LOCALE = 'C';
> >
> >
> > pg_restore: warning: errors ignored on database "postgres" restore: 1
> > pg_restore: warning: errors ignored on restore: 3
> >
> >
> >
> > It seems pointless to be trying to create the rolw that we are connected
> > as, and we also expect template1 and postgres to exist.
>
> Thanks Andrew for the updated patches.
>
> Here, I am attaching a delta patch which solves the errors for the
> already created database and we need to reset some flags also. Please
> have a look over this delta patch and merge it.
>
> If we want to skip errors for connected user (CREATE ROLE username),
> then we need to handle it by comparing sql commands in
> process_global_sql_commands function or we can compare errors after
> executing it.
> delta_0002* patch is doing some handling but this is not a proper fix.
>
> I think we can merge delta_0001* and later, we can work on delta_0002.
Yes, delta 1 looks OK, except that the pstrdup() calls are probably
unnecessary. Delta 2 needs some significant surgery at least. I think we
can use it as at least a partial fix, to avoid trying to create the role
we're running as (Should use PQuser() for that rather than cparams.user).
BTW, if you're sending delta patches, make sure they don't have .patch
extensions. Otherwise, the CFBot gets upset. I usually just add .noci to
the file names.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-27 21:15 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-28 22:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-29 05:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-30 16:50 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 09:34 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-31 13:57 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
@ 2025-03-31 16:16 ` Mahendra Singh Thalor <[email protected]>
2025-03-31 17:16 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Mahendra Singh Thalor @ 2025-03-31 16:16 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: jian he <[email protected]>; Álvaro Herrera <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Mon, 31 Mar 2025 at 19:27, Andrew Dunstan <[email protected]> wrote:
>
>
> On 2025-03-31 Mo 5:34 AM, Mahendra Singh Thalor wrote:
> >
> > >
> > > There are a couple of rough edges, though.
> > >
> > > First, I see this:
> > >
> > >
> > > andrew@ub22arm:inst $ bin/pg_restore -C -d postgres
> > > --exclude-database=regression_dummy_seclabel
> > > --exclude-database=regression_test_extensions
> > > --exclude-database=regression_test_pg_dump dest
> > > pg_restore: error: could not execute query: "ERROR: role "andrew"
> > > already exists
> > > "
> > > Command was: "
> > >
> > > --
> > > -- Roles
> > > --
> > >
> > > CREATE ROLE andrew;"
> > > pg_restore: warning: errors ignored on global.dat file restore: 1
> > > pg_restore: error: could not execute query: ERROR: database "template1"
> > > already exists
> > > Command was: CREATE DATABASE template1 WITH TEMPLATE = template0
> > > ENCODING = 'SQL_ASCII' LOCALE_PROVIDER = libc LOCALE = 'C';
> > >
> > >
> > > pg_restore: warning: errors ignored on database "template1" restore: 1
> > > pg_restore: error: could not execute query: ERROR: database "postgres"
> > > already exists
> > > Command was: CREATE DATABASE postgres WITH TEMPLATE = template0 ENCODING
> > > = 'SQL_ASCII' LOCALE_PROVIDER = libc LOCALE = 'C';
> > >
> > >
> > > pg_restore: warning: errors ignored on database "postgres" restore: 1
> > > pg_restore: warning: errors ignored on restore: 3
> > >
> > >
> > >
> > > It seems pointless to be trying to create the rolw that we are connected
> > > as, and we also expect template1 and postgres to exist.
> >
> > Thanks Andrew for the updated patches.
> >
> > Here, I am attaching a delta patch which solves the errors for the
> > already created database and we need to reset some flags also. Please
> > have a look over this delta patch and merge it.
> >
> > If we want to skip errors for connected user (CREATE ROLE username),
> > then we need to handle it by comparing sql commands in
> > process_global_sql_commands function or we can compare errors after
> > executing it.
> > delta_0002* patch is doing some handling but this is not a proper fix.
> >
> > I think we can merge delta_0001* and later, we can work on delta_0002.
>
>
> Yes, delta 1 looks OK, except that the pstrdup() calls are probably
> unnecessary. Delta 2 needs some significant surgery at least. I think we
> can use it as at least a partial fix, to avoid trying to create the role
> we're running as (Should use PQuser() for that rather than cparams.user).
Thanks for the quick review.
I fixed the above comments and made 2 delta patches. Please have a
look over these.
> BTW, if you're sending delta patches, make sure they don't have .patch
> extensions. Otherwise, the CFBot gets upset. I usually just add .noci to
> the file names.
Sure. I will also use .noci. Thanks for feedback.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] delta-0001-31march-pg_restore-skip-error-if-db-already-created.noci (2.3K, 2-delta-0001-31march-pg_restore-skip-error-if-db-already-created.noci)
download
[application/octet-stream] delta-0002-31march-pg_restore-skip-error-for-CRETE-ROLE-username.noci (2.2K, 3-delta-0002-31march-pg_restore-skip-error-for-CRETE-ROLE-username.noci)
download
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-27 21:15 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-28 22:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-29 05:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-30 16:50 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 09:34 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-31 13:57 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 16:16 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
@ 2025-03-31 17:16 ` Andrew Dunstan <[email protected]>
2025-03-31 17:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Dunstan @ 2025-03-31 17:16 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: jian he <[email protected]>; Álvaro Herrera <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-03-31 Mo 12:16 PM, Mahendra Singh Thalor wrote:
> On Mon, 31 Mar 2025 at 19:27, Andrew Dunstan <[email protected]> wrote:
>>
>> On 2025-03-31 Mo 5:34 AM, Mahendra Singh Thalor wrote:
>>>> There are a couple of rough edges, though.
>>>>
>>>> First, I see this:
>>>>
>>>>
>>>> andrew@ub22arm:inst $ bin/pg_restore -C -d postgres
>>>> --exclude-database=regression_dummy_seclabel
>>>> --exclude-database=regression_test_extensions
>>>> --exclude-database=regression_test_pg_dump dest
>>>> pg_restore: error: could not execute query: "ERROR: role "andrew"
>>>> already exists
>>>> "
>>>> Command was: "
>>>>
>>>> --
>>>> -- Roles
>>>> --
>>>>
>>>> CREATE ROLE andrew;"
>>>> pg_restore: warning: errors ignored on global.dat file restore: 1
>>>> pg_restore: error: could not execute query: ERROR: database "template1"
>>>> already exists
>>>> Command was: CREATE DATABASE template1 WITH TEMPLATE = template0
>>>> ENCODING = 'SQL_ASCII' LOCALE_PROVIDER = libc LOCALE = 'C';
>>>>
>>>>
>>>> pg_restore: warning: errors ignored on database "template1" restore: 1
>>>> pg_restore: error: could not execute query: ERROR: database "postgres"
>>>> already exists
>>>> Command was: CREATE DATABASE postgres WITH TEMPLATE = template0 ENCODING
>>>> = 'SQL_ASCII' LOCALE_PROVIDER = libc LOCALE = 'C';
>>>>
>>>>
>>>> pg_restore: warning: errors ignored on database "postgres" restore: 1
>>>> pg_restore: warning: errors ignored on restore: 3
>>>>
>>>>
>>>>
>>>> It seems pointless to be trying to create the rolw that we are connected
>>>> as, and we also expect template1 and postgres to exist.
>>> Thanks Andrew for the updated patches.
>>>
>>> Here, I am attaching a delta patch which solves the errors for the
>>> already created database and we need to reset some flags also. Please
>>> have a look over this delta patch and merge it.
>>>
>>> If we want to skip errors for connected user (CREATE ROLE username),
>>> then we need to handle it by comparing sql commands in
>>> process_global_sql_commands function or we can compare errors after
>>> executing it.
>>> delta_0002* patch is doing some handling but this is not a proper fix.
>>>
>>> I think we can merge delta_0001* and later, we can work on delta_0002.
>>
>> Yes, delta 1 looks OK, except that the pstrdup() calls are probably
>> unnecessary. Delta 2 needs some significant surgery at least. I think we
>> can use it as at least a partial fix, to avoid trying to create the role
>> we're running as (Should use PQuser() for that rather than cparams.user).
> Thanks for the quick review.
>
> I fixed the above comments and made 2 delta patches. Please have a
> look over these.
>
>> BTW, if you're sending delta patches, make sure they don't have .patch
>> extensions. Otherwise, the CFBot gets upset. I usually just add .noci to
>> the file names.
> Sure. I will also use .noci. Thanks for feedback.
Thanks. Here are patches that contain (my version of) all the cleanups.
With this I get a clean restore run in my test case with no error messages.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-27 21:15 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-28 22:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-29 05:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-30 16:50 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 09:34 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-31 13:57 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 16:16 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-31 17:16 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
@ 2025-03-31 17:20 ` Andrew Dunstan <[email protected]>
2025-03-31 18:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Dunstan @ 2025-03-31 17:20 UTC (permalink / raw)
To: Mahendra Singh Thalor <[email protected]>; +Cc: jian he <[email protected]>; Álvaro Herrera <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On 2025-03-31 Mo 1:16 PM, Andrew Dunstan wrote:
>
>
>
> Thanks. Here are patches that contain (my version of) all the
> cleanups. With this I get a clean restore run in my test case with no
> error messages.
>
>
>
This time with patches
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
Attachments:
[text/x-patch] v20250331-0001-Move-common-pg_dump-code-related-to-connec.patch (26.2K, 2-v20250331-0001-Move-common-pg_dump-code-related-to-connec.patch)
download | inline diff:
From 6286701ff360ccb8c105fa5aa0a8f9bba3b1d1d7 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Wed, 19 Mar 2025 01:18:46 +0530
Subject: [PATCH v20250331 1/3] Move common pg_dump code related to connections
to a new file
ConnectDatabase is used by pg_dumpall, pg_restore and pg_dump so move
common code to new file.
new file name: connectdb.c
Author: Mahendra Singh Thalor <[email protected]>
---
src/bin/pg_dump/Makefile | 5 +-
src/bin/pg_dump/connectdb.c | 294 +++++++++++++++++++++++++++
src/bin/pg_dump/connectdb.h | 26 +++
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/pg_backup.h | 6 +-
src/bin/pg_dump/pg_backup_archiver.c | 6 +-
src/bin/pg_dump/pg_backup_db.c | 79 +------
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 278 +------------------------
9 files changed, 352 insertions(+), 345 deletions(-)
create mode 100644 src/bin/pg_dump/connectdb.c
create mode 100644 src/bin/pg_dump/connectdb.h
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..fa795883e9f 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -31,6 +31,7 @@ OBJS = \
compress_lz4.o \
compress_none.o \
compress_zstd.o \
+ connectdb.o \
dumputils.o \
filter.o \
parallel.o \
@@ -50,8 +51,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
new file mode 100644
index 00000000000..9e593b70e81
--- /dev/null
+++ b/src/bin/pg_dump/connectdb.c
@@ -0,0 +1,294 @@
+/*-------------------------------------------------------------------------
+ *
+ * connectdb.c
+ * This is a common file connection to the database.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/connectdb.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "connectdb.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+
+/*
+ * ConnectDatabase
+ *
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the 'connstr' is set to a connection string containing
+ * the options used and 'server_version' is set to version so that caller
+ * can use them.
+ */
+PGconn *
+ConnectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version, char *password,
+ char *override_dbname)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 8;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ if (override_dbname)
+ {
+ keywords[i] = "dbname";
+ values[i++] = override_dbname;
+ }
+
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. If requested, remember the options used, in
+ * the form of a connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If requested, then copy server version to out variable. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/*
+ * constructConnStr
+ *
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * executeQuery
+ *
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
diff --git a/src/bin/pg_dump/connectdb.h b/src/bin/pg_dump/connectdb.h
new file mode 100644
index 00000000000..6c1e1954769
--- /dev/null
+++ b/src/bin/pg_dump/connectdb.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * connectdb.h
+ * Common header file for connection to the database.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/connectdb.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONNECTDB_H
+#define CONNECTDB_H
+
+#include "pg_backup.h"
+#include "pg_backup_utils.h"
+
+extern PGconn *ConnectDatabase(const char *dbname, const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version,
+ char *password, char *override_dbname);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+#endif /* CONNECTDB_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..25989e8f16b 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -6,6 +6,7 @@ pg_dump_common_sources = files(
'compress_lz4.c',
'compress_none.c',
'compress_zstd.c',
+ 'connectdb.c',
'dumputils.c',
'filter.c',
'parallel.c',
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 658986de6f8..49bc1ee71ef 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -293,9 +293,9 @@ typedef void (*SetupWorkerPtrType) (Archive *AH);
* Main archiver interface.
*/
-extern void ConnectDatabase(Archive *AHX,
- const ConnParams *cparams,
- bool isReconnect);
+extern void ConnectDatabaseAhx(Archive *AHX,
+ const ConnParams *cparams,
+ bool isReconnect);
extern void DisconnectDatabase(Archive *AHX);
extern PGconn *GetConnection(Archive *AHX);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 1d131e5a57d..3f59f8f9d9d 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -415,7 +415,7 @@ RestoreArchive(Archive *AHX)
AHX->minRemoteVersion = 0;
AHX->maxRemoteVersion = 9999999;
- ConnectDatabase(AHX, &ropt->cparams, false);
+ ConnectDatabaseAhx(AHX, &ropt->cparams, false);
/*
* If we're talking to the DB directly, don't send comments since they
@@ -4458,7 +4458,7 @@ restore_toc_entries_postfork(ArchiveHandle *AH, TocEntry *pending_list)
/*
* Now reconnect the single parent connection.
*/
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
/* re-establish fixed state */
_doSetFixedOutputState(AH);
@@ -5076,7 +5076,7 @@ CloneArchive(ArchiveHandle *AH)
* Connect our new clone object to the database, using the same connection
* parameters used for the original connection.
*/
- ConnectDatabase((Archive *) clone, &clone->public.ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) clone, &clone->public.ropt->cparams, true);
/* re-establish fixed state */
if (AH->mode == archModeRead)
diff --git a/src/bin/pg_dump/pg_backup_db.c b/src/bin/pg_dump/pg_backup_db.c
index 71c55d2466a..5c349279beb 100644
--- a/src/bin/pg_dump/pg_backup_db.c
+++ b/src/bin/pg_dump/pg_backup_db.c
@@ -19,6 +19,7 @@
#include "common/connect.h"
#include "common/string.h"
+#include "connectdb.h"
#include "parallel.h"
#include "pg_backup_archiver.h"
#include "pg_backup_db.h"
@@ -86,9 +87,9 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* ArchiveHandle's connCancel, before closing old connection. Otherwise
* an ill-timed SIGINT could try to access a dead connection.
*/
- AH->connection = NULL; /* dodge error check in ConnectDatabase */
+ AH->connection = NULL; /* dodge error check in ConnectDatabaseAhx */
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
PQfinish(oldConn);
}
@@ -105,14 +106,13 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* username never does change, so one savedPassword is sufficient.
*/
void
-ConnectDatabase(Archive *AHX,
- const ConnParams *cparams,
- bool isReconnect)
+ConnectDatabaseAhx(Archive *AHX,
+ const ConnParams *cparams,
+ bool isReconnect)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
trivalue prompt_password;
char *password;
- bool new_pass;
if (AH->connection)
pg_fatal("already connected to a database");
@@ -125,69 +125,10 @@ ConnectDatabase(Archive *AHX,
if (prompt_password == TRI_YES && password == NULL)
password = simple_prompt("Password: ", false);
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- const char *keywords[8];
- const char *values[8];
- int i = 0;
-
- /*
- * If dbname is a connstring, its entries can override the other
- * values obtained from cparams; but in turn, override_dbname can
- * override the dbname component of it.
- */
- keywords[i] = "host";
- values[i++] = cparams->pghost;
- keywords[i] = "port";
- values[i++] = cparams->pgport;
- keywords[i] = "user";
- values[i++] = cparams->username;
- keywords[i] = "password";
- values[i++] = password;
- keywords[i] = "dbname";
- values[i++] = cparams->dbname;
- if (cparams->override_dbname)
- {
- keywords[i] = "dbname";
- values[i++] = cparams->override_dbname;
- }
- keywords[i] = "fallback_application_name";
- values[i++] = progname;
- keywords[i] = NULL;
- values[i++] = NULL;
- Assert(i <= lengthof(keywords));
-
- new_pass = false;
- AH->connection = PQconnectdbParams(keywords, values, true);
-
- if (!AH->connection)
- pg_fatal("could not connect to database");
-
- if (PQstatus(AH->connection) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(AH->connection) &&
- password == NULL &&
- prompt_password != TRI_NO)
- {
- PQfinish(AH->connection);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(AH->connection) == CONNECTION_BAD)
- {
- if (isReconnect)
- pg_fatal("reconnection failed: %s",
- PQerrorMessage(AH->connection));
- else
- pg_fatal("%s",
- PQerrorMessage(AH->connection));
- }
+ AH->connection = ConnectDatabase(cparams->dbname, NULL, cparams->pghost,
+ cparams->pgport, cparams->username,
+ prompt_password, true,
+ progname, NULL, NULL, password, cparams->override_dbname);
/* Start strict; later phases may override this. */
PQclear(ExecuteSqlQueryForSingleRow((Archive *) AH,
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 4ca34be230c..f84ea6ecc48 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -966,7 +966,7 @@ main(int argc, char **argv)
* Open the database using the Archiver, so it knows about it. Errors mean
* death.
*/
- ConnectDatabase(fout, &dopt.cparams, false);
+ ConnectDatabaseAhx(fout, &dopt.cparams, false);
setup_connection(fout, dumpencoding, dumpsnapshot, use_role);
/*
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 2ea574b0f06..573a8b61a45 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -24,11 +24,11 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
-#include "pg_backup.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -71,21 +71,14 @@ static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
static char pg_dump_bin[MAXPGPATH];
-static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -129,8 +122,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -499,19 +490,22 @@ main(int argc, char *argv[])
*/
if (pgdb)
{
- conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase(pgdb, connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
}
else
{
- conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase("postgres", connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
- conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ conn = ConnectDatabase("template1", connstr, pghost, pgport, pguser,
+ prompt_password, true,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
{
@@ -1738,256 +1732,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
--
2.34.1
[text/x-patch] v20250331-0002-add-new-list-type-simple_oid_string_list-t.patch (2.6K, 3-v20250331-0002-add-new-list-type-simple_oid_string_list-t.patch)
download | inline diff:
From f172c7db61a75d4bbb586d597ae8e9e44d02642c Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email protected]>
Date: Fri, 28 Mar 2025 18:10:24 -0400
Subject: [PATCH v20250331 2/3] add new list type simple_oid_string_list to
fe-utils/simple_list
---
src/fe_utils/simple_list.c | 41 ++++++++++++++++++++++++++++++
src/include/fe_utils/simple_list.h | 16 ++++++++++++
2 files changed, 57 insertions(+)
diff --git a/src/fe_utils/simple_list.c b/src/fe_utils/simple_list.c
index 483d5455594..bbcc4ef618d 100644
--- a/src/fe_utils/simple_list.c
+++ b/src/fe_utils/simple_list.c
@@ -192,3 +192,44 @@ simple_ptr_list_destroy(SimplePtrList *list)
cell = next;
}
}
+
+/*
+ * Add to an oid_string list
+ */
+void
+simple_oid_string_list_append(SimpleOidStringList * list, Oid oid, const char *str)
+{
+ SimpleOidStringListCell *cell;
+
+ cell = (SimpleOidStringListCell *)
+ pg_malloc(offsetof(SimpleOidStringListCell, str) + strlen(str) + 1);
+
+ cell->next = NULL;
+ cell->oid = oid;
+ strcpy(cell->str, str);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * Destroy an oid_string list
+ */
+void
+simple_oid_string_list_destroy(SimpleOidStringList * list)
+{
+ SimpleOidStringListCell *cell;
+
+ cell = list->head;
+ while (cell != NULL)
+ {
+ SimpleOidStringListCell *next;
+
+ next = cell->next;
+ pg_free(cell);
+ cell = next;
+ }
+}
diff --git a/src/include/fe_utils/simple_list.h b/src/include/fe_utils/simple_list.h
index 3b8e38414ec..af61545d7ff 100644
--- a/src/include/fe_utils/simple_list.h
+++ b/src/include/fe_utils/simple_list.h
@@ -55,6 +55,19 @@ typedef struct SimplePtrList
SimplePtrListCell *tail;
} SimplePtrList;
+typedef struct SimpleOidStringListCell
+{
+ struct SimpleOidStringListCell *next;
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} SimpleOidStringListCell;
+
+typedef struct SimpleOidStringList
+{
+ SimpleOidStringListCell *head;
+ SimpleOidStringListCell *tail;
+} SimpleOidStringList;
+
extern void simple_oid_list_append(SimpleOidList *list, Oid val);
extern bool simple_oid_list_member(SimpleOidList *list, Oid val);
extern void simple_oid_list_destroy(SimpleOidList *list);
@@ -68,4 +81,7 @@ extern const char *simple_string_list_not_touched(SimpleStringList *list);
extern void simple_ptr_list_append(SimplePtrList *list, void *ptr);
extern void simple_ptr_list_destroy(SimplePtrList *list);
+extern void simple_oid_string_list_append(SimpleOidStringList * list, Oid oid, const char *str);
+extern void simple_oid_string_list_destroy(SimpleOidStringList * list);
+
#endif /* SIMPLE_LIST_H */
--
2.34.1
[text/x-patch] v20250331-0003-pg_dumpall-with-directory-tar-custom-forma.patch (64.3K, 4-v20250331-0003-pg_dumpall-with-directory-tar-custom-forma.patch)
download | inline diff:
From 00c534f0e4f501cadb84f2d26b78c92995971f3b Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Wed, 19 Mar 2025 01:30:12 +0530
Subject: [PATCH v20250331 3/3] pg_dumpall with directory|tar|custom format and
restore it by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat to restore all databases. If global.dat file is exist in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list (if exist)
for --exclude-database=PATTERN for pg_restore
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
if no db connection, then PATTERN=NAME matching only
for each database, we are cleaning on_exit_nicely_index list.
at the end of restore, we are giving warning with total number of errors (including global.dat,
and each database errors) and for each database, we are printing warning with dbname and total
errors.
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
Author: Mahendra Singh Thalor <[email protected]>
---
doc/src/sgml/ref/pg_dumpall.sgml | 86 ++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/parallel.c | 11 +-
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 20 +-
src/bin/pg_dump/pg_backup_archiver.h | 3 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_backup_utils.c | 22 +-
src/bin/pg_dump/pg_backup_utils.h | 3 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 295 ++++++++--
src/bin/pg_dump/pg_restore.c | 802 ++++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 9 +
src/tools/pgindent/typedefs.list | 2 +
14 files changed, 1231 insertions(+), 94 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 765b30a3a66..43fdab2d77e 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster using a specified dump format</refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +33,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an archive. The archive contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +52,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -121,10 +126,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>global.dat</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index c840a807ae9..f14e5866f6c 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore a <productname>PostgreSQL</productname> database or cluster
+ from an archive created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by<application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -140,6 +149,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -166,6 +177,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +348,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..a36d2a5bf84 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -326,11 +326,18 @@ getThreadLocalPQExpBuffer(void)
* pg_dump and pg_restore call this to register the cleanup handler
* as soon as they've created the ArchiveHandle.
*/
-void
+int
on_exit_close_archive(Archive *AHX)
{
shutdown_info.AHX = AHX;
- on_exit_nicely(archive_close_connection, &shutdown_info);
+ return on_exit_nicely(archive_close_connection, &shutdown_info);
+}
+
+void
+replace_on_exit_close_archive(Archive *AHX, int idx)
+{
+ shutdown_info.AHX = AHX;
+ set_on_exit_nicely_entry(archive_close_connection, &shutdown_info, idx);
}
/*
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 49bc1ee71ef..17d6e06ec25 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -311,7 +311,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 3f59f8f9d9d..54eb4728928 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -85,7 +85,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -338,9 +338,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -457,7 +462,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1293,7 +1298,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1672,7 +1677,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1692,7 +1698,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index a2064f471ed..dc045b852e9 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -385,7 +385,8 @@ struct _tocEntry
};
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
-extern void on_exit_close_archive(Archive *AHX);
+extern int on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX, int idx);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..59ece2999a8 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -61,14 +61,26 @@ set_dump_section(const char *arg, int *dumpSections)
/* Register a callback to be run when exit_nicely is invoked. */
-void
+int
on_exit_nicely(on_exit_nicely_callback function, void *arg)
{
- if (on_exit_nicely_index >= MAX_ON_EXIT_NICELY)
- pg_fatal("out of on_exit_nicely slots");
- on_exit_nicely_list[on_exit_nicely_index].function = function;
- on_exit_nicely_list[on_exit_nicely_index].arg = arg;
+ set_on_exit_nicely_entry(function, arg, on_exit_nicely_index);
on_exit_nicely_index++;
+
+ return (on_exit_nicely_index - 1);
+}
+
+void
+set_on_exit_nicely_entry(on_exit_nicely_callback function, void *arg, int i)
+{
+ if (i >= MAX_ON_EXIT_NICELY)
+ pg_fatal("out of on_exit_nicely slots");
+
+ if (i > on_exit_nicely_index)
+ pg_fatal("no entry exists on %d index into on_exit_nicely slots", i);
+
+ on_exit_nicely_list[i].function = function;
+ on_exit_nicely_list[i].arg = arg;
}
/*
diff --git a/src/bin/pg_dump/pg_backup_utils.h b/src/bin/pg_dump/pg_backup_utils.h
index ba042016879..1ce1077096d 100644
--- a/src/bin/pg_dump/pg_backup_utils.h
+++ b/src/bin/pg_dump/pg_backup_utils.h
@@ -28,7 +28,8 @@ typedef void (*on_exit_nicely_callback) (int code, void *arg);
extern const char *progname;
extern void set_dump_section(const char *arg, int *dumpSections);
-extern void on_exit_nicely(on_exit_nicely_callback function, void *arg);
+extern int on_exit_nicely(on_exit_nicely_callback function, void *arg);
+extern void set_on_exit_nicely_entry(on_exit_nicely_callback function, void *arg, int idx);
pg_noreturn extern void exit_nicely(int code);
/* In pg_dump, we modify pg_fatal to call exit_nicely instead of exit */
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index f84ea6ecc48..832a6af7091 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1219,7 +1219,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 573a8b61a45..9d08c6ca0e6 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -64,9 +65,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -75,6 +77,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -146,6 +150,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -195,6 +200,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -244,7 +251,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -272,7 +279,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -421,6 +430,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -483,6 +507,33 @@ main(int argc, char *argv[])
if (statistics_only)
appendPQExpBufferStr(pgdumpopts, " --statistics-only");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char global_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(global_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -522,19 +573,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -634,7 +672,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -647,7 +685,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -658,12 +696,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster based on specified dump format.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -969,9 +1009,6 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
@@ -1485,6 +1522,7 @@ dumpUserConfig(PGconn *conn, const char *username)
{
PQExpBuffer buf = createPQExpBuffer();
PGresult *res;
+ static bool header_done = false;
printfPQExpBuffer(buf, "SELECT unnest(setconfig) FROM pg_db_role_setting "
"WHERE setdatabase = 0 AND setrole = "
@@ -1496,7 +1534,13 @@ dumpUserConfig(PGconn *conn, const char *username)
res = executeQuery(conn, buf->data);
if (PQntuples(res) > 0)
+ {
+ if (!header_done)
+ fprintf(OPF, "\n--\n-- User Configurations\n--\n");
+ header_done = true;
+
fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", username);
+ }
for (int i = 0; i < PQntuples(res); i++)
{
@@ -1570,10 +1614,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1587,18 +1634,42 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
- if (PQntuples(res) > 0)
+ if (archDumpFormat == archNull && PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_fatal("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1612,9 +1683,27 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
/*
* We assume that "template1" and "postgres" already exist in the
@@ -1628,12 +1717,9 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
- else
- {
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
fprintf(OPF, "\\connect %s\n\n", dbname);
- }
}
else
create_opts = "--create";
@@ -1641,19 +1727,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char global_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(global_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ global_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1663,7 +1760,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1672,17 +1770,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1827,3 +1944,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if (!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 47f7b0dd3a1..5ba0c4b768f 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,11 +41,15 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -53,18 +57,36 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num);
+static int read_one_statement(StringInfo inBuf, FILE *pfile);
+static int restore_all_databases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static void copy_or_print_global_file(const char *outfile, FILE *pfile);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimpleOidStringList * dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleOidStringList * dbname_oid_list);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
+static int on_exit_index = 0;
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -90,6 +112,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -144,6 +167,7 @@ main(int argc, char **argv)
{"with-statistics", no_argument, &with_statistics, 1},
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -172,7 +196,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -199,11 +223,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -318,6 +345,9 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -345,6 +375,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -452,6 +489,115 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL && !file_exists_in_directory(inputFileSpec, "toc.dat") &&
+ file_exists_in_directory(inputFileSpec, "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql
+ * commands. */
+
+ /*
+ * User is suggested to use single database dump for --list option.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring multiple databases by archive of pg_dumpall");
+
+ /*
+ * To restore multiple databases, -C (create database) option should
+ * be specified. Even there is single database in dump, report error
+ * because it might be possible that database hasn't created so better
+ * we report error.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("-C/--create option should be specified when restoring multiple databases by archive of pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("If db is already created and dump has single db dump, then use particular dump file.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Connect to database to execute global sql commands from global.dat
+ * file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ /*
+ * Open global.dat file and execute/append all the global sql
+ * commands.
+ */
+ n_errors = process_global_sql_commands(conn, inputFileSpec,
+ opts->filename);
+
+ if (conn)
+ PQfinish(conn);
+
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ n_errors = restore_all_databases(conn, inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else /* process if global.dat file does not exist. */
+ {
+ if (db_exclude_patterns.head != NULL)
+ pg_fatal("option --exclude-database can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, 0);
+ }
+
+ on_exit_index = 0; /* Reset index. */
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -459,9 +605,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then save index of exit_nicely so that we
+ * can use same slot for all the databases as we already closed the
+ * previous archive by CloseArchive.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_index = on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH, on_exit_index);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -481,25 +633,22 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n"
+ "If archive is created by pg_dumpall, then restores multiple databases also. \n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -517,6 +666,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -529,6 +679,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -569,8 +720,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be combined\n"
+ "and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -675,3 +826,620 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * read_one_statement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.dat file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+read_one_statement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+ int m;
+
+ StringInfoData q;
+
+ initStringInfo(&q);
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != '\n' && c != ';')
+ {
+ appendStringInfoChar(inBuf, (char) c);
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != ';' && c != '\n')
+ appendStringInfoChar(inBuf, (char) c);
+ else
+ break;
+ }
+ }
+
+ if (c == '\'' || c == '"')
+ {
+ appendStringInfoChar(&q, (char) c);
+ m = c;
+
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(&q, (char) c);
+
+ if (c == m)
+ {
+ appendStringInfoString(inBuf, q.data);
+ resetStringInfo(&q);
+ break;
+ }
+ }
+ }
+
+ if (c == ';')
+ {
+ appendStringInfoChar(inBuf, (char) ';');
+ break;
+ }
+
+ if (c == '\n')
+ appendStringInfoChar(inBuf, (char) '\n');
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimpleOidStringList * dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimpleOidStringListCell * db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ bool skip_db_restore = false;
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * the construct pattern matching query: SELECT 1 WHERE XXX
+ * OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from
+ * the dbname_oid_list, which is initially extracted from the
+ * map.dat file located in the backup directory. that's why we
+ * need quote_literal_cstr.
+ *
+ * If no db connection, then consider PATTERN as NAME.
+ */
+ if (pg_strcasecmp(db_cell->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, quote_literal_cstr(db_cell->str),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ db_cell->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database \"%s\" matches exclude pattern: \"%s\"", db_cell->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /* Increment count if database needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", db_cell->str);
+ db_cell->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleOidStringList * dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ /*
+ * If there is only global.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while ((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid = InvalidOid;
+ char db_oid_str[MAXPGPATH + 1] = {'\0'};
+ char dbname[MAXPGPATH + 1] = {'\0'};
+
+ /* Extract dboid. */
+ sscanf(line, "%u", &db_oid);
+ sscanf(line, "%20s", db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found database \"%s\" (OID: %u) in map.dat file while restoring.", dbname, db_oid);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making a
+ * list of all the databases.
+ */
+ simple_oid_string_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleOidStringList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /*
+ * If map.dat has no entry, return from here after processing global.dat
+ * file.
+ */
+ if (dbname_oid_list.head == NULL)
+ return process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+
+ /*
+ * processing pg_restore --exclude-database=PATTERN/NAME if no connection.
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ n_errors_total = process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info("no database needs to restore out of %d databases", num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * Till now, we made a list of databases, those needs to be restored after
+ * skipping names of exclude-database. Now we can launch parallel workers
+ * to restore these databases.
+ */
+ for (SimpleOidStringListCell * db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (db_cell->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", dumpdirpath);
+
+ /*
+ * Validate database dump file. If there is .tar or .dmp file exist
+ * then consider particular file, otherwise just append dboid to the
+ * databases folder.
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", db_cell->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", dumpdirpath, db_cell->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", db_cell->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", dumpdirpath, db_cell->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, db_cell->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", db_cell->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(db_cell->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = db_cell->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, count);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", db_cell->str, n_errors);
+ }
+
+ count++;
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_oid_string_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
+
+/*
+ * process_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ *
+ * returns the number of errors while processing global.dat
+ */
+static int
+process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement, user_create;
+ FILE *pfile;
+ int n_errors = 0;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ copy_or_print_global_file(outfile, pfile);
+ return 0;
+ }
+
+ /* Init sqlstatement to append commands. */
+ initStringInfo(&sqlstatement);
+
+ /* creation statement for our current role */
+ initStringInfo(&user_create);
+ appendStringInfoString(&user_create, "CREATE ROLE ");
+ /* should use fmtId here, but we don't know the encoding */
+ appendStringInfoString(&user_create, PQuser(conn));
+ appendStringInfoString(&user_create, ";");
+
+ /* Process file till EOF and execute sql statements. */
+ while (read_one_statement(&sqlstatement, pfile) != EOF)
+ {
+ /* don't try to create the role we are connected as */
+ if (strstr(sqlstatement.data, user_create.data))
+ continue;
+
+ pg_log_info("executing query: %s", sqlstatement.data);
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ n_errors++;
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ /* Print a summary of ignored errors during global.dat. */
+ if (n_errors)
+ pg_log_warning("errors ignored on global.dat file restore: %d", n_errors);
+
+ fclose(pfile);
+
+ return n_errors;
+}
+
+/*
+ * copy_or_print_global_file
+ *
+ * This will copy global.dat file into out file. If "-" is used as outfile,
+ * then print commands to the stdout.
+ */
+static void
+copy_or_print_global_file(const char *outfile, FILE *pfile)
+{
+ char out_file_path[MAXPGPATH];
+ FILE *OPF;
+ int c;
+
+ /* "-" is used for stdout. */
+ if (strcmp(outfile, "-") == 0)
+ OPF = stdout;
+ else
+ {
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ OPF = fopen(out_file_path, PG_BINARY_W);
+
+ if (OPF == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", outfile);
+ }
+ }
+
+ /* Append global.dat into out file or print to the stdout. */
+ while ((c = fgetc(pfile)) != EOF)
+ fputc(c, OPF);
+
+ fclose(pfile);
+
+ /* Close out file. */
+ if (strcmp(outfile, "-") != 0)
+ fclose(OPF);
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr
+ *
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..0bbcdbe84a7
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,11 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +249,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b66cecd8799..95ec8fbb141 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2732,6 +2732,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.34.1
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-27 21:15 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-28 22:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-29 05:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-30 16:50 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 09:34 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-31 13:57 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 16:16 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-31 17:16 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 17:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
@ 2025-03-31 18:12 ` Álvaro Herrera <[email protected]>
2025-04-01 05:59 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
0 siblings, 1 reply; 29+ messages in thread
From: Álvaro Herrera @ 2025-03-31 18:12 UTC (permalink / raw)
To: Andrew Dunstan <[email protected]>; +Cc: Mahendra Singh Thalor <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
Hi
FWIW I don't think the on_exit_nicely business is in final shape just
yet. We're doing something super strange and novel about keeping track
of an array index, so that we can modify it later. Or something like
that, I think? That doesn't sound all that nice to me. Elsewhere it
was suggested that we need some way to keep track of the list of things
that need cleanup (a list of connections IIRC?) -- perhaps in a
thread-local variable or a global or something -- and we install the
cleanup function once, and that reads from the variable. The program
can add things to the list, or remove them, at will; and we don't need
to modify the cleanup function in any way.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 29+ messages in thread
* Re: Non-text mode for pg_dumpall
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 14:11 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
2025-03-12 07:03 ` Re: Non-text mode for pg_dumpall jian he <[email protected]>
2025-03-12 15:48 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-27 21:15 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-28 22:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-29 05:17 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-30 16:50 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 09:34 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-31 13:57 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 16:16 ` Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-31 17:16 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 17:20 ` Re: Non-text mode for pg_dumpall Andrew Dunstan <[email protected]>
2025-03-31 18:12 ` Re: Non-text mode for pg_dumpall Álvaro Herrera <[email protected]>
@ 2025-04-01 05:59 ` Mahendra Singh Thalor <[email protected]>
0 siblings, 0 replies; 29+ messages in thread
From: Mahendra Singh Thalor @ 2025-04-01 05:59 UTC (permalink / raw)
To: Álvaro Herrera <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; jian he <[email protected]>; Srinath Reddy <[email protected]>; [email protected]
On Mon, 31 Mar 2025 at 23:43, Álvaro Herrera <[email protected]> wrote:
>
> Hi
>
> FWIW I don't think the on_exit_nicely business is in final shape just
> yet. We're doing something super strange and novel about keeping track
> of an array index, so that we can modify it later. Or something like
> that, I think? That doesn't sound all that nice to me. Elsewhere it
> was suggested that we need some way to keep track of the list of things
> that need cleanup (a list of connections IIRC?) -- perhaps in a
> thread-local variable or a global or something -- and we install the
> cleanup function once, and that reads from the variable. The program
> can add things to the list, or remove them, at will; and we don't need
> to modify the cleanup function in any way.
>
> --
> Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
Thanks Álvaro for the feedback.
I removed the old handling of on_exit_nicely_list from the last patch
set and added one simple function to just update the archive handle in
shutdown_info. (shutdown_info.AHX = AHX;)
For first database, we will add entry into on_exit_nicely_list array
and for rest database, we will update only shutdown_info as we already
closed connection for previous database.With this fix, we will not
touch entry of on_exit_nicely_list for each database.
Here, I am attaching updated patches.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] v25_0002-add-new-list-type-simple_oid_string_list-to-fe-utils.patch (2.6K, 2-v25_0002-add-new-list-type-simple_oid_string_list-to-fe-utils.patch)
download | inline diff:
From a1701016ca6647df22847fb87b369d94bd60ece9 Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email protected]>
Date: Fri, 28 Mar 2025 18:10:24 -0400
Subject: [PATCH 2/4] add new list type simple_oid_string_list to
fe-utils/simple_list
---
src/fe_utils/simple_list.c | 41 ++++++++++++++++++++++++++++++
src/include/fe_utils/simple_list.h | 16 ++++++++++++
2 files changed, 57 insertions(+)
diff --git a/src/fe_utils/simple_list.c b/src/fe_utils/simple_list.c
index 483d5455594..bbcc4ef618d 100644
--- a/src/fe_utils/simple_list.c
+++ b/src/fe_utils/simple_list.c
@@ -192,3 +192,44 @@ simple_ptr_list_destroy(SimplePtrList *list)
cell = next;
}
}
+
+/*
+ * Add to an oid_string list
+ */
+void
+simple_oid_string_list_append(SimpleOidStringList * list, Oid oid, const char *str)
+{
+ SimpleOidStringListCell *cell;
+
+ cell = (SimpleOidStringListCell *)
+ pg_malloc(offsetof(SimpleOidStringListCell, str) + strlen(str) + 1);
+
+ cell->next = NULL;
+ cell->oid = oid;
+ strcpy(cell->str, str);
+
+ if (list->tail)
+ list->tail->next = cell;
+ else
+ list->head = cell;
+ list->tail = cell;
+}
+
+/*
+ * Destroy an oid_string list
+ */
+void
+simple_oid_string_list_destroy(SimpleOidStringList * list)
+{
+ SimpleOidStringListCell *cell;
+
+ cell = list->head;
+ while (cell != NULL)
+ {
+ SimpleOidStringListCell *next;
+
+ next = cell->next;
+ pg_free(cell);
+ cell = next;
+ }
+}
diff --git a/src/include/fe_utils/simple_list.h b/src/include/fe_utils/simple_list.h
index 3b8e38414ec..af61545d7ff 100644
--- a/src/include/fe_utils/simple_list.h
+++ b/src/include/fe_utils/simple_list.h
@@ -55,6 +55,19 @@ typedef struct SimplePtrList
SimplePtrListCell *tail;
} SimplePtrList;
+typedef struct SimpleOidStringListCell
+{
+ struct SimpleOidStringListCell *next;
+ Oid oid;
+ char str[FLEXIBLE_ARRAY_MEMBER]; /* null-terminated string here */
+} SimpleOidStringListCell;
+
+typedef struct SimpleOidStringList
+{
+ SimpleOidStringListCell *head;
+ SimpleOidStringListCell *tail;
+} SimpleOidStringList;
+
extern void simple_oid_list_append(SimpleOidList *list, Oid val);
extern bool simple_oid_list_member(SimpleOidList *list, Oid val);
extern void simple_oid_list_destroy(SimpleOidList *list);
@@ -68,4 +81,7 @@ extern const char *simple_string_list_not_touched(SimpleStringList *list);
extern void simple_ptr_list_append(SimplePtrList *list, void *ptr);
extern void simple_ptr_list_destroy(SimplePtrList *list);
+extern void simple_oid_string_list_append(SimpleOidStringList * list, Oid oid, const char *str);
+extern void simple_oid_string_list_destroy(SimpleOidStringList * list);
+
#endif /* SIMPLE_LIST_H */
--
2.39.3
[application/octet-stream] v25_0001-Move-common-pg_dump-code-related-to-connections.patch (26.2K, 3-v25_0001-Move-common-pg_dump-code-related-to-connections.patch)
download | inline diff:
From 259644a2b0d22b286c0f599927135325e8905894 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Wed, 19 Mar 2025 01:18:46 +0530
Subject: [PATCH 1/4] Move common pg_dump code related to connections to a new
file
ConnectDatabase is used by pg_dumpall, pg_restore and pg_dump so move
common code to new file.
new file name: connectdb.c
Author: Mahendra Singh Thalor <[email protected]>
---
src/bin/pg_dump/Makefile | 5 +-
src/bin/pg_dump/connectdb.c | 294 +++++++++++++++++++++++++++
src/bin/pg_dump/connectdb.h | 26 +++
src/bin/pg_dump/meson.build | 1 +
src/bin/pg_dump/pg_backup.h | 6 +-
src/bin/pg_dump/pg_backup_archiver.c | 6 +-
src/bin/pg_dump/pg_backup_db.c | 79 +------
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 278 +------------------------
9 files changed, 352 insertions(+), 345 deletions(-)
create mode 100644 src/bin/pg_dump/connectdb.c
create mode 100644 src/bin/pg_dump/connectdb.h
diff --git a/src/bin/pg_dump/Makefile b/src/bin/pg_dump/Makefile
index 233ad15ca75..fa795883e9f 100644
--- a/src/bin/pg_dump/Makefile
+++ b/src/bin/pg_dump/Makefile
@@ -31,6 +31,7 @@ OBJS = \
compress_lz4.o \
compress_none.o \
compress_zstd.o \
+ connectdb.o \
dumputils.o \
filter.o \
parallel.o \
@@ -50,8 +51,8 @@ pg_dump: pg_dump.o common.o pg_dump_sort.o $(OBJS) | submake-libpq submake-libpg
pg_restore: pg_restore.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
$(CC) $(CFLAGS) pg_restore.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
-pg_dumpall: pg_dumpall.o dumputils.o filter.o $(WIN32RES) | submake-libpq submake-libpgport submake-libpgfeutils
- $(CC) $(CFLAGS) pg_dumpall.o dumputils.o filter.o $(WIN32RES) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
+pg_dumpall: pg_dumpall.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
+ $(CC) $(CFLAGS) pg_dumpall.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_dump$(X) '$(DESTDIR)$(bindir)'/pg_dump$(X)
diff --git a/src/bin/pg_dump/connectdb.c b/src/bin/pg_dump/connectdb.c
new file mode 100644
index 00000000000..9e593b70e81
--- /dev/null
+++ b/src/bin/pg_dump/connectdb.c
@@ -0,0 +1,294 @@
+/*-------------------------------------------------------------------------
+ *
+ * connectdb.c
+ * This is a common file connection to the database.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/connectdb.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres_fe.h"
+
+#include "common/connect.h"
+#include "common/logging.h"
+#include "common/string.h"
+#include "connectdb.h"
+#include "dumputils.h"
+#include "fe_utils/string_utils.h"
+
+static char *constructConnStr(const char **keywords, const char **values);
+
+/*
+ * ConnectDatabase
+ *
+ * Make a database connection with the given parameters. An
+ * interactive password prompt is automatically issued if required.
+ *
+ * If fail_on_error is false, we return NULL without printing any message
+ * on failure, but preserve any prompted password for the next try.
+ *
+ * On success, the 'connstr' is set to a connection string containing
+ * the options used and 'server_version' is set to version so that caller
+ * can use them.
+ */
+PGconn *
+ConnectDatabase(const char *dbname, const char *connection_string,
+ const char *pghost, const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error, const char *progname,
+ const char **connstr, int *server_version, char *password,
+ char *override_dbname)
+{
+ PGconn *conn;
+ bool new_pass;
+ const char *remoteversion_str;
+ int my_version;
+ const char **keywords = NULL;
+ const char **values = NULL;
+ PQconninfoOption *conn_opts = NULL;
+ int server_version_temp;
+
+ if (prompt_password == TRI_YES && !password)
+ password = simple_prompt("Password: ", false);
+
+ /*
+ * Start the connection. Loop until we have a password if requested by
+ * backend.
+ */
+ do
+ {
+ int argcount = 8;
+ PQconninfoOption *conn_opt;
+ char *err_msg = NULL;
+ int i = 0;
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /*
+ * Merge the connection info inputs given in form of connection string
+ * and other options. Explicitly discard any dbname value in the
+ * connection string; otherwise, PQconnectdbParams() would interpret
+ * that value as being itself a connection string.
+ */
+ if (connection_string)
+ {
+ conn_opts = PQconninfoParse(connection_string, &err_msg);
+ if (conn_opts == NULL)
+ pg_fatal("%s", err_msg);
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ argcount++;
+ }
+
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+
+ for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
+ {
+ if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
+ strcmp(conn_opt->keyword, "dbname") != 0)
+ {
+ keywords[i] = conn_opt->keyword;
+ values[i] = conn_opt->val;
+ i++;
+ }
+ }
+ }
+ else
+ {
+ keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
+ values = pg_malloc0((argcount + 1) * sizeof(*values));
+ }
+
+ if (pghost)
+ {
+ keywords[i] = "host";
+ values[i] = pghost;
+ i++;
+ }
+ if (pgport)
+ {
+ keywords[i] = "port";
+ values[i] = pgport;
+ i++;
+ }
+ if (pguser)
+ {
+ keywords[i] = "user";
+ values[i] = pguser;
+ i++;
+ }
+ if (password)
+ {
+ keywords[i] = "password";
+ values[i] = password;
+ i++;
+ }
+ if (dbname)
+ {
+ keywords[i] = "dbname";
+ values[i] = dbname;
+ i++;
+ }
+ if (override_dbname)
+ {
+ keywords[i] = "dbname";
+ values[i++] = override_dbname;
+ }
+
+ keywords[i] = "fallback_application_name";
+ values[i] = progname;
+ i++;
+
+ new_pass = false;
+ conn = PQconnectdbParams(keywords, values, true);
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", dbname);
+
+ if (PQstatus(conn) == CONNECTION_BAD &&
+ PQconnectionNeedsPassword(conn) &&
+ !password &&
+ prompt_password != TRI_NO)
+ {
+ PQfinish(conn);
+ password = simple_prompt("Password: ", false);
+ new_pass = true;
+ }
+ } while (new_pass);
+
+ /* check to see that the backend connection was successfully made */
+ if (PQstatus(conn) == CONNECTION_BAD)
+ {
+ if (fail_on_error)
+ pg_fatal("%s", PQerrorMessage(conn));
+ else
+ {
+ PQfinish(conn);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ return NULL;
+ }
+ }
+
+ /*
+ * Ok, connected successfully. If requested, remember the options used, in
+ * the form of a connection string.
+ */
+ if (connstr)
+ *connstr = constructConnStr(keywords, values);
+
+ free(keywords);
+ free(values);
+ PQconninfoFree(conn_opts);
+
+ /* Check version */
+ remoteversion_str = PQparameterStatus(conn, "server_version");
+ if (!remoteversion_str)
+ pg_fatal("could not get server version");
+
+ server_version_temp = PQserverVersion(conn);
+ if (server_version_temp == 0)
+ pg_fatal("could not parse server version \"%s\"",
+ remoteversion_str);
+
+ /* If requested, then copy server version to out variable. */
+ if (server_version)
+ *server_version = server_version_temp;
+
+ my_version = PG_VERSION_NUM;
+
+ /*
+ * We allow the server to be back to 9.2, and up to any minor release of
+ * our own major version. (See also version check in pg_dump.c.)
+ */
+ if (my_version != server_version_temp
+ && (server_version_temp < 90200 ||
+ (server_version_temp / 100) > (my_version / 100)))
+ {
+ pg_log_error("aborting because of server version mismatch");
+ pg_log_error_detail("server version: %s; %s version: %s",
+ remoteversion_str, progname, PG_VERSION);
+ exit_nicely(1);
+ }
+
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
+
+ return conn;
+}
+
+/*
+ * constructConnStr
+ *
+ * Construct a connection string from the given keyword/value pairs. It is
+ * used to pass the connection options to the pg_dump subprocess.
+ *
+ * The following parameters are excluded:
+ * dbname - varies in each pg_dump invocation
+ * password - it's not secure to pass a password on the command line
+ * fallback_application_name - we'll let pg_dump set it
+ */
+static char *
+constructConnStr(const char **keywords, const char **values)
+{
+ PQExpBuffer buf = createPQExpBuffer();
+ char *connstr;
+ int i;
+ bool firstkeyword = true;
+
+ /* Construct a new connection string in key='value' format. */
+ for (i = 0; keywords[i] != NULL; i++)
+ {
+ if (strcmp(keywords[i], "dbname") == 0 ||
+ strcmp(keywords[i], "password") == 0 ||
+ strcmp(keywords[i], "fallback_application_name") == 0)
+ continue;
+
+ if (!firstkeyword)
+ appendPQExpBufferChar(buf, ' ');
+ firstkeyword = false;
+ appendPQExpBuffer(buf, "%s=", keywords[i]);
+ appendConnStrVal(buf, values[i]);
+ }
+
+ connstr = pg_strdup(buf->data);
+ destroyPQExpBuffer(buf);
+ return connstr;
+}
+
+/*
+ * executeQuery
+ *
+ * Run a query, return the results, exit program on failure.
+ */
+PGresult *
+executeQuery(PGconn *conn, const char *query)
+{
+ PGresult *res;
+
+ pg_log_info("executing %s", query);
+
+ res = PQexec(conn, query);
+ if (!res ||
+ PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ pg_log_error("query failed: %s", PQerrorMessage(conn));
+ pg_log_error_detail("Query was: %s", query);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ return res;
+}
diff --git a/src/bin/pg_dump/connectdb.h b/src/bin/pg_dump/connectdb.h
new file mode 100644
index 00000000000..6c1e1954769
--- /dev/null
+++ b/src/bin/pg_dump/connectdb.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * connectdb.h
+ * Common header file for connection to the database.
+ *
+ * Portions Copyright (c) 1996-2025, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/bin/pg_dump/connectdb.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONNECTDB_H
+#define CONNECTDB_H
+
+#include "pg_backup.h"
+#include "pg_backup_utils.h"
+
+extern PGconn *ConnectDatabase(const char *dbname, const char *connection_string, const char *pghost,
+ const char *pgport, const char *pguser,
+ trivalue prompt_password, bool fail_on_error,
+ const char *progname, const char **connstr, int *server_version,
+ char *password, char *override_dbname);
+extern PGresult *executeQuery(PGconn *conn, const char *query);
+#endif /* CONNECTDB_H */
diff --git a/src/bin/pg_dump/meson.build b/src/bin/pg_dump/meson.build
index 603ba6cfbf0..25989e8f16b 100644
--- a/src/bin/pg_dump/meson.build
+++ b/src/bin/pg_dump/meson.build
@@ -6,6 +6,7 @@ pg_dump_common_sources = files(
'compress_lz4.c',
'compress_none.c',
'compress_zstd.c',
+ 'connectdb.c',
'dumputils.c',
'filter.c',
'parallel.c',
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 658986de6f8..49bc1ee71ef 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -293,9 +293,9 @@ typedef void (*SetupWorkerPtrType) (Archive *AH);
* Main archiver interface.
*/
-extern void ConnectDatabase(Archive *AHX,
- const ConnParams *cparams,
- bool isReconnect);
+extern void ConnectDatabaseAhx(Archive *AHX,
+ const ConnParams *cparams,
+ bool isReconnect);
extern void DisconnectDatabase(Archive *AHX);
extern PGconn *GetConnection(Archive *AHX);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 1d131e5a57d..3f59f8f9d9d 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -415,7 +415,7 @@ RestoreArchive(Archive *AHX)
AHX->minRemoteVersion = 0;
AHX->maxRemoteVersion = 9999999;
- ConnectDatabase(AHX, &ropt->cparams, false);
+ ConnectDatabaseAhx(AHX, &ropt->cparams, false);
/*
* If we're talking to the DB directly, don't send comments since they
@@ -4458,7 +4458,7 @@ restore_toc_entries_postfork(ArchiveHandle *AH, TocEntry *pending_list)
/*
* Now reconnect the single parent connection.
*/
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
/* re-establish fixed state */
_doSetFixedOutputState(AH);
@@ -5076,7 +5076,7 @@ CloneArchive(ArchiveHandle *AH)
* Connect our new clone object to the database, using the same connection
* parameters used for the original connection.
*/
- ConnectDatabase((Archive *) clone, &clone->public.ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) clone, &clone->public.ropt->cparams, true);
/* re-establish fixed state */
if (AH->mode == archModeRead)
diff --git a/src/bin/pg_dump/pg_backup_db.c b/src/bin/pg_dump/pg_backup_db.c
index 71c55d2466a..5c349279beb 100644
--- a/src/bin/pg_dump/pg_backup_db.c
+++ b/src/bin/pg_dump/pg_backup_db.c
@@ -19,6 +19,7 @@
#include "common/connect.h"
#include "common/string.h"
+#include "connectdb.h"
#include "parallel.h"
#include "pg_backup_archiver.h"
#include "pg_backup_db.h"
@@ -86,9 +87,9 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* ArchiveHandle's connCancel, before closing old connection. Otherwise
* an ill-timed SIGINT could try to access a dead connection.
*/
- AH->connection = NULL; /* dodge error check in ConnectDatabase */
+ AH->connection = NULL; /* dodge error check in ConnectDatabaseAhx */
- ConnectDatabase((Archive *) AH, &ropt->cparams, true);
+ ConnectDatabaseAhx((Archive *) AH, &ropt->cparams, true);
PQfinish(oldConn);
}
@@ -105,14 +106,13 @@ ReconnectToServer(ArchiveHandle *AH, const char *dbname)
* username never does change, so one savedPassword is sufficient.
*/
void
-ConnectDatabase(Archive *AHX,
- const ConnParams *cparams,
- bool isReconnect)
+ConnectDatabaseAhx(Archive *AHX,
+ const ConnParams *cparams,
+ bool isReconnect)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
trivalue prompt_password;
char *password;
- bool new_pass;
if (AH->connection)
pg_fatal("already connected to a database");
@@ -125,69 +125,10 @@ ConnectDatabase(Archive *AHX,
if (prompt_password == TRI_YES && password == NULL)
password = simple_prompt("Password: ", false);
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- const char *keywords[8];
- const char *values[8];
- int i = 0;
-
- /*
- * If dbname is a connstring, its entries can override the other
- * values obtained from cparams; but in turn, override_dbname can
- * override the dbname component of it.
- */
- keywords[i] = "host";
- values[i++] = cparams->pghost;
- keywords[i] = "port";
- values[i++] = cparams->pgport;
- keywords[i] = "user";
- values[i++] = cparams->username;
- keywords[i] = "password";
- values[i++] = password;
- keywords[i] = "dbname";
- values[i++] = cparams->dbname;
- if (cparams->override_dbname)
- {
- keywords[i] = "dbname";
- values[i++] = cparams->override_dbname;
- }
- keywords[i] = "fallback_application_name";
- values[i++] = progname;
- keywords[i] = NULL;
- values[i++] = NULL;
- Assert(i <= lengthof(keywords));
-
- new_pass = false;
- AH->connection = PQconnectdbParams(keywords, values, true);
-
- if (!AH->connection)
- pg_fatal("could not connect to database");
-
- if (PQstatus(AH->connection) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(AH->connection) &&
- password == NULL &&
- prompt_password != TRI_NO)
- {
- PQfinish(AH->connection);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(AH->connection) == CONNECTION_BAD)
- {
- if (isReconnect)
- pg_fatal("reconnection failed: %s",
- PQerrorMessage(AH->connection));
- else
- pg_fatal("%s",
- PQerrorMessage(AH->connection));
- }
+ AH->connection = ConnectDatabase(cparams->dbname, NULL, cparams->pghost,
+ cparams->pgport, cparams->username,
+ prompt_password, true,
+ progname, NULL, NULL, password, cparams->override_dbname);
/* Start strict; later phases may override this. */
PQclear(ExecuteSqlQueryForSingleRow((Archive *) AH,
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 4ca34be230c..f84ea6ecc48 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -966,7 +966,7 @@ main(int argc, char **argv)
* Open the database using the Archiver, so it knows about it. Errors mean
* death.
*/
- ConnectDatabase(fout, &dopt.cparams, false);
+ ConnectDatabaseAhx(fout, &dopt.cparams, false);
setup_connection(fout, dumpencoding, dumpsnapshot, use_role);
/*
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 2ea574b0f06..573a8b61a45 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -24,11 +24,11 @@
#include "common/hashfn_unstable.h"
#include "common/logging.h"
#include "common/string.h"
+#include "connectdb.h"
#include "dumputils.h"
#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
-#include "pg_backup.h"
/* version string we expect back from pg_dump */
#define PGDUMP_VERSIONSTR "pg_dump (PostgreSQL) " PG_VERSION "\n"
@@ -71,21 +71,14 @@ static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
PQExpBuffer buffer);
-static PGconn *connectDatabase(const char *dbname,
- const char *connection_string, const char *pghost,
- const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error);
-static char *constructConnStr(const char **keywords, const char **values);
-static PGresult *executeQuery(PGconn *conn, const char *query);
static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
static char pg_dump_bin[MAXPGPATH];
-static const char *progname;
static PQExpBuffer pgdumpopts;
-static char *connstr = "";
+static const char *connstr = "";
static bool output_clean = false;
static bool skip_acls = false;
static bool verbose = false;
@@ -129,8 +122,6 @@ static char *filename = NULL;
static SimpleStringList database_exclude_patterns = {NULL, NULL};
static SimpleStringList database_exclude_names = {NULL, NULL};
-#define exit_nicely(code) exit(code)
-
int
main(int argc, char *argv[])
{
@@ -499,19 +490,22 @@ main(int argc, char *argv[])
*/
if (pgdb)
{
- conn = connectDatabase(pgdb, connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase(pgdb, connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
pg_fatal("could not connect to database \"%s\"", pgdb);
}
else
{
- conn = connectDatabase("postgres", connstr, pghost, pgport, pguser,
- prompt_password, false);
+ conn = ConnectDatabase("postgres", connstr, pghost, pgport, pguser,
+ prompt_password, false,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
- conn = connectDatabase("template1", connstr, pghost, pgport, pguser,
- prompt_password, true);
+ conn = ConnectDatabase("template1", connstr, pghost, pgport, pguser,
+ prompt_password, true,
+ progname, &connstr, &server_version, NULL, NULL);
if (!conn)
{
@@ -1738,256 +1732,6 @@ buildShSecLabels(PGconn *conn, const char *catalog_name, Oid objectId,
destroyPQExpBuffer(sql);
}
-/*
- * Make a database connection with the given parameters. An
- * interactive password prompt is automatically issued if required.
- *
- * If fail_on_error is false, we return NULL without printing any message
- * on failure, but preserve any prompted password for the next try.
- *
- * On success, the global variable 'connstr' is set to a connection string
- * containing the options used.
- */
-static PGconn *
-connectDatabase(const char *dbname, const char *connection_string,
- const char *pghost, const char *pgport, const char *pguser,
- trivalue prompt_password, bool fail_on_error)
-{
- PGconn *conn;
- bool new_pass;
- const char *remoteversion_str;
- int my_version;
- const char **keywords = NULL;
- const char **values = NULL;
- PQconninfoOption *conn_opts = NULL;
- static char *password = NULL;
-
- if (prompt_password == TRI_YES && !password)
- password = simple_prompt("Password: ", false);
-
- /*
- * Start the connection. Loop until we have a password if requested by
- * backend.
- */
- do
- {
- int argcount = 6;
- PQconninfoOption *conn_opt;
- char *err_msg = NULL;
- int i = 0;
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /*
- * Merge the connection info inputs given in form of connection string
- * and other options. Explicitly discard any dbname value in the
- * connection string; otherwise, PQconnectdbParams() would interpret
- * that value as being itself a connection string.
- */
- if (connection_string)
- {
- conn_opts = PQconninfoParse(connection_string, &err_msg);
- if (conn_opts == NULL)
- pg_fatal("%s", err_msg);
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- argcount++;
- }
-
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
-
- for (conn_opt = conn_opts; conn_opt->keyword != NULL; conn_opt++)
- {
- if (conn_opt->val != NULL && conn_opt->val[0] != '\0' &&
- strcmp(conn_opt->keyword, "dbname") != 0)
- {
- keywords[i] = conn_opt->keyword;
- values[i] = conn_opt->val;
- i++;
- }
- }
- }
- else
- {
- keywords = pg_malloc0((argcount + 1) * sizeof(*keywords));
- values = pg_malloc0((argcount + 1) * sizeof(*values));
- }
-
- if (pghost)
- {
- keywords[i] = "host";
- values[i] = pghost;
- i++;
- }
- if (pgport)
- {
- keywords[i] = "port";
- values[i] = pgport;
- i++;
- }
- if (pguser)
- {
- keywords[i] = "user";
- values[i] = pguser;
- i++;
- }
- if (password)
- {
- keywords[i] = "password";
- values[i] = password;
- i++;
- }
- if (dbname)
- {
- keywords[i] = "dbname";
- values[i] = dbname;
- i++;
- }
- keywords[i] = "fallback_application_name";
- values[i] = progname;
- i++;
-
- new_pass = false;
- conn = PQconnectdbParams(keywords, values, true);
-
- if (!conn)
- pg_fatal("could not connect to database \"%s\"", dbname);
-
- if (PQstatus(conn) == CONNECTION_BAD &&
- PQconnectionNeedsPassword(conn) &&
- !password &&
- prompt_password != TRI_NO)
- {
- PQfinish(conn);
- password = simple_prompt("Password: ", false);
- new_pass = true;
- }
- } while (new_pass);
-
- /* check to see that the backend connection was successfully made */
- if (PQstatus(conn) == CONNECTION_BAD)
- {
- if (fail_on_error)
- pg_fatal("%s", PQerrorMessage(conn));
- else
- {
- PQfinish(conn);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- return NULL;
- }
- }
-
- /*
- * Ok, connected successfully. Remember the options used, in the form of a
- * connection string.
- */
- connstr = constructConnStr(keywords, values);
-
- free(keywords);
- free(values);
- PQconninfoFree(conn_opts);
-
- /* Check version */
- remoteversion_str = PQparameterStatus(conn, "server_version");
- if (!remoteversion_str)
- pg_fatal("could not get server version");
- server_version = PQserverVersion(conn);
- if (server_version == 0)
- pg_fatal("could not parse server version \"%s\"",
- remoteversion_str);
-
- my_version = PG_VERSION_NUM;
-
- /*
- * We allow the server to be back to 9.2, and up to any minor release of
- * our own major version. (See also version check in pg_dump.c.)
- */
- if (my_version != server_version
- && (server_version < 90200 ||
- (server_version / 100) > (my_version / 100)))
- {
- pg_log_error("aborting because of server version mismatch");
- pg_log_error_detail("server version: %s; %s version: %s",
- remoteversion_str, progname, PG_VERSION);
- exit_nicely(1);
- }
-
- PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL));
-
- return conn;
-}
-
-/* ----------
- * Construct a connection string from the given keyword/value pairs. It is
- * used to pass the connection options to the pg_dump subprocess.
- *
- * The following parameters are excluded:
- * dbname - varies in each pg_dump invocation
- * password - it's not secure to pass a password on the command line
- * fallback_application_name - we'll let pg_dump set it
- * ----------
- */
-static char *
-constructConnStr(const char **keywords, const char **values)
-{
- PQExpBuffer buf = createPQExpBuffer();
- char *connstr;
- int i;
- bool firstkeyword = true;
-
- /* Construct a new connection string in key='value' format. */
- for (i = 0; keywords[i] != NULL; i++)
- {
- if (strcmp(keywords[i], "dbname") == 0 ||
- strcmp(keywords[i], "password") == 0 ||
- strcmp(keywords[i], "fallback_application_name") == 0)
- continue;
-
- if (!firstkeyword)
- appendPQExpBufferChar(buf, ' ');
- firstkeyword = false;
- appendPQExpBuffer(buf, "%s=", keywords[i]);
- appendConnStrVal(buf, values[i]);
- }
-
- connstr = pg_strdup(buf->data);
- destroyPQExpBuffer(buf);
- return connstr;
-}
-
-/*
- * Run a query, return the results, exit program on failure.
- */
-static PGresult *
-executeQuery(PGconn *conn, const char *query)
-{
- PGresult *res;
-
- pg_log_info("executing %s", query);
-
- res = PQexec(conn, query);
- if (!res ||
- PQresultStatus(res) != PGRES_TUPLES_OK)
- {
- pg_log_error("query failed: %s", PQerrorMessage(conn));
- pg_log_error_detail("Query was: %s", query);
- PQfinish(conn);
- exit_nicely(1);
- }
-
- return res;
-}
-
/*
* As above for a SQL command (which returns nothing).
*/
--
2.39.3
[application/octet-stream] v25_0004-update-AX-handle-for-each-database-for-cleanup.patch (2.6K, 4-v25_0004-update-AX-handle-for-each-database-for-cleanup.patch)
download | inline diff:
From 8450086079480f3784d26c6c85973b84cfa8b484 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Tue, 1 Apr 2025 11:08:38 +0530
Subject: [PATCH 4/4] update AX handle for each database for cleanup.
---
src/bin/pg_dump/parallel.c | 10 ++++++++++
src/bin/pg_dump/pg_backup_archiver.h | 1 +
src/bin/pg_dump/pg_restore.c | 10 +++++-----
3 files changed, 16 insertions(+), 5 deletions(-)
diff --git a/src/bin/pg_dump/parallel.c b/src/bin/pg_dump/parallel.c
index 086adcdc502..5974d6706fd 100644
--- a/src/bin/pg_dump/parallel.c
+++ b/src/bin/pg_dump/parallel.c
@@ -333,6 +333,16 @@ on_exit_close_archive(Archive *AHX)
on_exit_nicely(archive_close_connection, &shutdown_info);
}
+/*
+ * When pg_restore restores multiple databases, then update already added entry
+ * into array for cleanup.
+ */
+void
+replace_on_exit_close_archive(Archive *AHX)
+{
+ shutdown_info.AHX = AHX;
+}
+
/*
* on_exit_nicely handler for shutting down database connections and
* worker processes cleanly.
diff --git a/src/bin/pg_dump/pg_backup_archiver.h b/src/bin/pg_dump/pg_backup_archiver.h
index a2064f471ed..ed0238cca47 100644
--- a/src/bin/pg_dump/pg_backup_archiver.h
+++ b/src/bin/pg_dump/pg_backup_archiver.h
@@ -386,6 +386,7 @@ struct _tocEntry
extern int parallel_restore(ArchiveHandle *AH, TocEntry *te);
extern void on_exit_close_archive(Archive *AHX);
+extern void replace_on_exit_close_archive(Archive *AHX);
extern void warn_or_exit_horribly(ArchiveHandle *AH, const char *fmt,...) pg_attribute_printf(2, 3);
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index aa8887a4eb0..a142b744222 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -603,14 +603,14 @@ restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
* it's still NULL, the cleanup function will just be a no-op. If we are
- * restoring multiple databases, then save index of exit_nicely so that we
- * can use same slot for all the databases as we already closed the
- * previous archive by CloseArchive.
+ * restoring multiple databases, then only update AX handle for cleanup as
+ * previous entry was already in array and we had closed previous
+ * connection so we can use same slot from array.
*/
if (!append_data || num == 0)
on_exit_close_archive(AH);
- //else
- //replace_on_exit_close_archive(AH);
+ else
+ replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
--
2.39.3
[application/octet-stream] v25_0003-pg_dumpall-with-directory-tar-custom-format.patch (60.8K, 5-v25_0003-pg_dumpall-with-directory-tar-custom-format.patch)
download | inline diff:
From 10411c96b0294d414deaf475ad82a1dd7821be36 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <[email protected]>
Date: Tue, 1 Apr 2025 10:48:52 +0530
Subject: [PATCH 3/4] pg_dumpall with directory|tar|custom format and restore
it by pg_restore
new option to pg_dumpall:
-F, --format=d|t|c|p output file format ( plain text (default))
Ex:1 ./pg_dumpall --format=directory --file=dumpDirName
Ex:2 ./pg_dumpall --format=tar --file=dumpDirName
Ex:3 ./pg_dumpall --format=custom --file=dumpDirName
Ex:4 ./pg_dumpall --format=plain --file=dumpDirName
dumps are as:
global.dat ::: global sql commands in simple plain format
map.dat. ::: dboid dbname ---entries for all databases in simple text form
databases. :::
subdir dboid1 -> toc.dat and data files in archive format
subdir dboid2. -> toc.dat and data files in archive format
etc
---------------------------------------------------------------------------
NOTE:
if needed, restore single db by particular subdir
Ex: ./pg_restore --format=directory -d postgres dumpDirName/databases/5
-- here, 5 is the dboid of postgres db
-- to get dboid, refer dbname in map.file
--------------------------------------------------------------------------
new options to pg_restore:
-g, --globals-only restore only global objects, no databases
--exclude-database=PATTERN exclude database whose name matches pattern
When we give -g/--globals-only option, then only restore globals, no db restoring.
Design:
When --format=d|t|c is specified and there is no toc.dat in main directory, then check
for global.dat to restore all databases. If global.dat file is exist in directory,
then first restore all globals from global.dat and then restore all databases one by one
from map.dat list (if exist)
for --exclude-database=PATTERN for pg_restore
as of now, SELECT 1 WHERE XXX OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
if no db connection, then PATTERN=NAME matching only
At the end of restore, we are giving warning with total number of errors (including global.dat,
and each database errors) and for each database, we are printing warning with dbname and total
errors.
thread:
https://www.postgresql.org/message-id/flat/CAKYtNAp9vOtydXL3_pnGJ%2BTetZtN%3DFYSnZSMCqXceU3mkHPxPg%40mail.gmail.com#066433cb5ae007cbe35fefddf796d52f
Author: Mahendra Singh Thalor <[email protected]>
---
doc/src/sgml/ref/pg_dumpall.sgml | 86 ++-
doc/src/sgml/ref/pg_restore.sgml | 66 ++-
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_backup_archiver.c | 20 +-
src/bin/pg_dump/pg_backup_tar.c | 2 +-
src/bin/pg_dump/pg_dump.c | 2 +-
src/bin/pg_dump/pg_dumpall.c | 295 ++++++++--
src/bin/pg_dump/pg_restore.c | 799 ++++++++++++++++++++++++++-
src/bin/pg_dump/t/001_basic.pl | 9 +
src/tools/pgindent/typedefs.list | 2 +
10 files changed, 1198 insertions(+), 85 deletions(-)
mode change 100644 => 100755 src/bin/pg_dump/t/001_basic.pl
diff --git a/doc/src/sgml/ref/pg_dumpall.sgml b/doc/src/sgml/ref/pg_dumpall.sgml
index 765b30a3a66..43fdab2d77e 100644
--- a/doc/src/sgml/ref/pg_dumpall.sgml
+++ b/doc/src/sgml/ref/pg_dumpall.sgml
@@ -16,7 +16,7 @@ PostgreSQL documentation
<refnamediv>
<refname>pg_dumpall</refname>
- <refpurpose>extract a <productname>PostgreSQL</productname> database cluster into a script file</refpurpose>
+ <refpurpose>extract a <productname>PostgreSQL</productname> database cluster using a specified dump format</refpurpose>
</refnamediv>
<refsynopsisdiv>
@@ -33,7 +33,7 @@ PostgreSQL documentation
<para>
<application>pg_dumpall</application> is a utility for writing out
(<quote>dumping</quote>) all <productname>PostgreSQL</productname> databases
- of a cluster into one script file. The script file contains
+ of a cluster into an archive. The archive contains
<acronym>SQL</acronym> commands that can be used as input to <xref
linkend="app-psql"/> to restore the databases. It does this by
calling <xref linkend="app-pgdump"/> for each database in the cluster.
@@ -52,11 +52,16 @@ PostgreSQL documentation
</para>
<para>
- The SQL script will be written to the standard output. Use the
+ Plain text SQL scripts will be written to the standard output. Use the
<option>-f</option>/<option>--file</option> option or shell operators to
redirect it into a file.
</para>
+ <para>
+ Archives in other formats will be placed in a directory named using the
+ <option>-f</option>/<option>--file</option>, which is required in this case.
+ </para>
+
<para>
<application>pg_dumpall</application> needs to connect several
times to the <productname>PostgreSQL</productname> server (once per
@@ -121,10 +126,85 @@ PostgreSQL documentation
<para>
Send output to the specified file. If this is omitted, the
standard output is used.
+ Note: This option can only be omitted when <option>--format</option> is plain
</para>
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-F <replaceable class="parameter">format</replaceable></option></term>
+ <term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the format of dump files. In plain format, all the dump data is
+ sent in a single text stream. This is the default.
+
+ In all other modes, <application>pg_dumpall</application> first creates two files:
+ <filename>global.dat</filename> and <filename>map.dat</filename>, in the directory
+ specified by <option>--file</option>.
+ The first file contains global data, such as roles and tablespaces. The second
+ contains a mapping between database oids and names. These files are used by
+ <application>pg_restore</application>. Data for individual databases is placed in
+ <filename>databases</filename> subdirectory, named using the database's <type>oid</type>.
+
+ <variablelist>
+ <varlistentry>
+ <term><literal>d</literal></term>
+ <term><literal>directory</literal></term>
+ <listitem>
+ <para>
+ Output directory-format archives for each database,
+ suitable for input into pg_restore. The directory
+ will have database <type>oid</type> as its name.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>p</literal></term>
+ <term><literal>plain</literal></term>
+ <listitem>
+ <para>
+ Output a plain-text SQL script file (the default).
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>c</literal></term>
+ <term><literal>custom</literal></term>
+ <listitem>
+ <para>
+ Output a custom-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.dmp</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><literal>t</literal></term>
+ <term><literal>tar</literal></term>
+ <listitem>
+ <para>
+ Output a tar-format archive for each database,
+ suitable for input into pg_restore. The archive
+ will be named <filename>dboid.tar</filename> where <type>dboid</type> is the
+ <type>oid</type> of the database.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ Note: see <xref linkend="app-pgdump"/> for details
+ of how the various non plain text archives work.
+
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--filter=<replaceable class="parameter">filename</replaceable></option></term>
<listitem>
diff --git a/doc/src/sgml/ref/pg_restore.sgml b/doc/src/sgml/ref/pg_restore.sgml
index c840a807ae9..f14e5866f6c 100644
--- a/doc/src/sgml/ref/pg_restore.sgml
+++ b/doc/src/sgml/ref/pg_restore.sgml
@@ -18,8 +18,9 @@ PostgreSQL documentation
<refname>pg_restore</refname>
<refpurpose>
- restore a <productname>PostgreSQL</productname> database from an
- archive file created by <application>pg_dump</application>
+ restore a <productname>PostgreSQL</productname> database or cluster
+ from an archive created by <application>pg_dump</application> or
+ <application>pg_dumpall</application>
</refpurpose>
</refnamediv>
@@ -38,13 +39,14 @@ PostgreSQL documentation
<para>
<application>pg_restore</application> is a utility for restoring a
- <productname>PostgreSQL</productname> database from an archive
- created by <xref linkend="app-pgdump"/> in one of the non-plain-text
+ <productname>PostgreSQL</productname> database or cluster from an archive
+ created by <xref linkend="app-pgdump"/> or
+ <xref linkend="app-pg-dumpall"/> in one of the non-plain-text
formats. It will issue the commands necessary to reconstruct the
- database to the state it was in at the time it was saved. The
- archive files also allow <application>pg_restore</application> to
+ database or cluster to the state it was in at the time it was saved. The
+ archives also allow <application>pg_restore</application> to
be selective about what is restored, or even to reorder the items
- prior to being restored. The archive files are designed to be
+ prior to being restored. The archive formats are designed to be
portable across architectures.
</para>
@@ -52,10 +54,17 @@ PostgreSQL documentation
<application>pg_restore</application> can operate in two modes.
If a database name is specified, <application>pg_restore</application>
connects to that database and restores archive contents directly into
- the database. Otherwise, a script containing the SQL
- commands necessary to rebuild the database is created and written
+ the database.
+ When restoring from a dump made by<application>pg_dumpall</application>,
+ each database will be created and then the restoration will be run in that
+ database.
+
+ Otherwise, when a database name is not specified, a script containing the SQL
+ commands necessary to rebuild the database or cluster is created and written
to a file or standard output. This script output is equivalent to
- the plain text output format of <application>pg_dump</application>.
+ the plain text output format of <application>pg_dump</application> or
+ <application>pg_dumpall</application>.
+
Some of the options controlling the output are therefore analogous to
<application>pg_dump</application> options.
</para>
@@ -140,6 +149,8 @@ PostgreSQL documentation
commands that mention this database.
Access privileges for the database itself are also restored,
unless <option>--no-acl</option> is specified.
+ <option>--create</option> is required when restoring multiple databases
+ from an archive created by <application>pg_dumpall</application>.
</para>
<para>
@@ -166,6 +177,28 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--exclude-database=<replaceable class="parameter">pattern</replaceable></option></term>
+ <listitem>
+ <para>
+ Do not restore databases whose name matches
+ <replaceable class="parameter">pattern</replaceable>.
+ Multiple patterns can be excluded by writing multiple
+ <option>--exclude-database</option> switches. The
+ <replaceable class="parameter">pattern</replaceable> parameter is
+ interpreted as a pattern according to the same rules used by
+ <application>psql</application>'s <literal>\d</literal>
+ commands (see <xref linkend="app-psql-patterns"/>),
+ so multiple databases can also be excluded by writing wildcard
+ characters in the pattern. When using wildcards, be careful to
+ quote the pattern if needed to prevent shell wildcard expansion.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--exit-on-error</option></term>
@@ -315,6 +348,19 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-g</option></term>
+ <term><option>--globals-only</option></term>
+ <listitem>
+ <para>
+ Restore only global objects (roles and tablespaces), no databases.
+ </para>
+ <para>
+ This option is only relevant when restoring from an archive made using <application>pg_dumpall</application>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-I <replaceable class="parameter">index</replaceable></option></term>
<term><option>--index=<replaceable class="parameter">index</replaceable></option></term>
diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h
index 49bc1ee71ef..17d6e06ec25 100644
--- a/src/bin/pg_dump/pg_backup.h
+++ b/src/bin/pg_dump/pg_backup.h
@@ -311,7 +311,7 @@ extern void SetArchiveOptions(Archive *AH, DumpOptions *dopt, RestoreOptions *ro
extern void ProcessArchiveRestoreOptions(Archive *AHX);
-extern void RestoreArchive(Archive *AHX);
+extern void RestoreArchive(Archive *AHX, bool append_data);
/* Open an existing archive */
extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c
index 3f59f8f9d9d..54eb4728928 100644
--- a/src/bin/pg_dump/pg_backup_archiver.c
+++ b/src/bin/pg_dump/pg_backup_archiver.c
@@ -85,7 +85,7 @@ static int RestoringToDB(ArchiveHandle *AH);
static void dump_lo_buf(ArchiveHandle *AH);
static void dumpTimestamp(ArchiveHandle *AH, const char *msg, time_t tim);
static void SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec);
+ const pg_compress_specification compression_spec, bool append_data);
static CompressFileHandle *SaveOutput(ArchiveHandle *AH);
static void RestoreOutput(ArchiveHandle *AH, CompressFileHandle *savedOutput);
@@ -338,9 +338,14 @@ ProcessArchiveRestoreOptions(Archive *AHX)
StrictNamesCheck(ropt);
}
-/* Public */
+/*
+ * RestoreArchive
+ *
+ * If append_data is set, then append data into file as we are restoring dump
+ * of multiple databases which was taken by pg_dumpall.
+ */
void
-RestoreArchive(Archive *AHX)
+RestoreArchive(Archive *AHX, bool append_data)
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
RestoreOptions *ropt = AH->public.ropt;
@@ -457,7 +462,7 @@ RestoreArchive(Archive *AHX)
*/
sav = SaveOutput(AH);
if (ropt->filename || ropt->compression_spec.algorithm != PG_COMPRESSION_NONE)
- SetOutput(AH, ropt->filename, ropt->compression_spec);
+ SetOutput(AH, ropt->filename, ropt->compression_spec, append_data);
ahprintf(AH, "--\n-- PostgreSQL database dump\n--\n\n");
@@ -1293,7 +1298,7 @@ PrintTOCSummary(Archive *AHX)
sav = SaveOutput(AH);
if (ropt->filename)
- SetOutput(AH, ropt->filename, out_compression_spec);
+ SetOutput(AH, ropt->filename, out_compression_spec, false);
if (strftime(stamp_str, sizeof(stamp_str), PGDUMP_STRFTIME_FMT,
localtime(&AH->createDate)) == 0)
@@ -1672,7 +1677,8 @@ archprintf(Archive *AH, const char *fmt,...)
static void
SetOutput(ArchiveHandle *AH, const char *filename,
- const pg_compress_specification compression_spec)
+ const pg_compress_specification compression_spec,
+ bool append_data)
{
CompressFileHandle *CFH;
const char *mode;
@@ -1692,7 +1698,7 @@ SetOutput(ArchiveHandle *AH, const char *filename,
else
fn = fileno(stdout);
- if (AH->mode == archModeAppend)
+ if (append_data || AH->mode == archModeAppend)
mode = PG_BINARY_A;
else
mode = PG_BINARY_W;
diff --git a/src/bin/pg_dump/pg_backup_tar.c b/src/bin/pg_dump/pg_backup_tar.c
index b5ba3b46dd9..d94d0de2a5d 100644
--- a/src/bin/pg_dump/pg_backup_tar.c
+++ b/src/bin/pg_dump/pg_backup_tar.c
@@ -826,7 +826,7 @@ _CloseArchive(ArchiveHandle *AH)
savVerbose = AH->public.verbose;
AH->public.verbose = 0;
- RestoreArchive((Archive *) AH);
+ RestoreArchive((Archive *) AH, false);
SetArchiveOptions((Archive *) AH, savDopt, savRopt);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index f84ea6ecc48..832a6af7091 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -1219,7 +1219,7 @@ main(int argc, char **argv)
* right now.
*/
if (plainText)
- RestoreArchive(fout);
+ RestoreArchive(fout, false);
CloseArchive(fout);
diff --git a/src/bin/pg_dump/pg_dumpall.c b/src/bin/pg_dump/pg_dumpall.c
index 573a8b61a45..9d08c6ca0e6 100644
--- a/src/bin/pg_dump/pg_dumpall.c
+++ b/src/bin/pg_dump/pg_dumpall.c
@@ -15,6 +15,7 @@
#include "postgres_fe.h"
+#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@@ -64,9 +65,10 @@ static void dropTablespaces(PGconn *conn);
static void dumpTablespaces(PGconn *conn);
static void dropDBs(PGconn *conn);
static void dumpUserConfig(PGconn *conn, const char *username);
-static void dumpDatabases(PGconn *conn);
+static void dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat);
static void dumpTimestamp(const char *msg);
-static int runPgDump(const char *dbname, const char *create_opts);
+static int runPgDump(const char *dbname, const char *create_opts,
+ char *dbfile, ArchiveFormat archDumpFormat);
static void buildShSecLabels(PGconn *conn,
const char *catalog_name, Oid objectId,
const char *objtype, const char *objname,
@@ -75,6 +77,8 @@ static void executeCommand(PGconn *conn, const char *query);
static void expand_dbname_patterns(PGconn *conn, SimpleStringList *patterns,
SimpleStringList *names);
static void read_dumpall_filters(const char *filename, SimpleStringList *pattern);
+static void create_or_open_dir(const char *dirname);
+static ArchiveFormat parseDumpFormat(const char *format);
static char pg_dump_bin[MAXPGPATH];
static PQExpBuffer pgdumpopts;
@@ -146,6 +150,7 @@ main(int argc, char *argv[])
{"password", no_argument, NULL, 'W'},
{"no-privileges", no_argument, NULL, 'x'},
{"no-acl", no_argument, NULL, 'x'},
+ {"format", required_argument, NULL, 'F'},
/*
* the following options don't have an equivalent short option letter
@@ -195,6 +200,8 @@ main(int argc, char *argv[])
char *pgdb = NULL;
char *use_role = NULL;
const char *dumpencoding = NULL;
+ ArchiveFormat archDumpFormat = archNull;
+ const char *formatName = "p";
trivalue prompt_password = TRI_DEFAULT;
bool data_only = false;
bool globals_only = false;
@@ -244,7 +251,7 @@ main(int argc, char *argv[])
pgdumpopts = createPQExpBuffer();
- while ((c = getopt_long(argc, argv, "acd:E:f:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
+ while ((c = getopt_long(argc, argv, "acd:E:f:F:gh:l:Op:rsS:tU:vwWx", long_options, &optindex)) != -1)
{
switch (c)
{
@@ -272,7 +279,9 @@ main(int argc, char *argv[])
appendPQExpBufferStr(pgdumpopts, " -f ");
appendShellString(pgdumpopts, filename);
break;
-
+ case 'F':
+ formatName = pg_strdup(optarg);
+ break;
case 'g':
globals_only = true;
break;
@@ -421,6 +430,21 @@ main(int argc, char *argv[])
exit_nicely(1);
}
+ /* Get format for dump. */
+ archDumpFormat = parseDumpFormat(formatName);
+
+ /*
+ * If a non-plain format is specified, a file name is also required as the
+ * path to the main directory.
+ */
+ if (archDumpFormat != archNull &&
+ (!filename || strcmp(filename, "") == 0))
+ {
+ pg_log_error("option -F/--format=d|c|t requires option -f/--file");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/*
* If password values are not required in the dump, switch to using
* pg_roles which is equally useful, just more likely to have unrestricted
@@ -483,6 +507,33 @@ main(int argc, char *argv[])
if (statistics_only)
appendPQExpBufferStr(pgdumpopts, " --statistics-only");
+ /*
+ * Open the output file if required, otherwise use stdout. If required,
+ * then create new directory and global.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char global_path[MAXPGPATH];
+
+ /* Create new directory or accept the empty existing directory. */
+ create_or_open_dir(filename);
+
+ snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
+
+ OPF = fopen(global_path, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open global.dat file: %s", strerror(errno));
+ }
+ else if (filename)
+ {
+ OPF = fopen(filename, PG_BINARY_W);
+ if (!OPF)
+ pg_fatal("could not open output file \"%s\": %m",
+ filename);
+ }
+ else
+ OPF = stdout;
+
/*
* If there was a database specified on the command line, use that,
* otherwise try to connect to database "postgres", and failing that
@@ -522,19 +573,6 @@ main(int argc, char *argv[])
expand_dbname_patterns(conn, &database_exclude_patterns,
&database_exclude_names);
- /*
- * Open the output file if required, otherwise use stdout
- */
- if (filename)
- {
- OPF = fopen(filename, PG_BINARY_W);
- if (!OPF)
- pg_fatal("could not open output file \"%s\": %m",
- filename);
- }
- else
- OPF = stdout;
-
/*
* Set the client encoding if requested.
*/
@@ -634,7 +672,7 @@ main(int argc, char *argv[])
}
if (!globals_only && !roles_only && !tablespaces_only)
- dumpDatabases(conn);
+ dumpDatabases(conn, archDumpFormat);
PQfinish(conn);
@@ -647,7 +685,7 @@ main(int argc, char *argv[])
fclose(OPF);
/* sync the resulting file, errors are not fatal */
- if (dosync)
+ if (dosync && (archDumpFormat == archNull))
(void) fsync_fname(filename, false);
}
@@ -658,12 +696,14 @@ main(int argc, char *argv[])
static void
help(void)
{
- printf(_("%s extracts a PostgreSQL database cluster into an SQL script file.\n\n"), progname);
+ printf(_("%s extracts a PostgreSQL database cluster based on specified dump format.\n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]...\n"), progname);
printf(_("\nGeneral options:\n"));
printf(_(" -f, --file=FILENAME output file name\n"));
+ printf(_(" -F, --format=c|d|t|p output file format (custom, directory, tar,\n"
+ " plain text (default))\n"));
printf(_(" -v, --verbose verbose mode\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
@@ -969,9 +1009,6 @@ dumpRoles(PGconn *conn)
* We do it this way because config settings for roles could mention the
* names of other roles.
*/
- if (PQntuples(res) > 0)
- fprintf(OPF, "\n--\n-- User Configurations\n--\n");
-
for (i = 0; i < PQntuples(res); i++)
dumpUserConfig(conn, PQgetvalue(res, i, i_rolname));
@@ -1485,6 +1522,7 @@ dumpUserConfig(PGconn *conn, const char *username)
{
PQExpBuffer buf = createPQExpBuffer();
PGresult *res;
+ static bool header_done = false;
printfPQExpBuffer(buf, "SELECT unnest(setconfig) FROM pg_db_role_setting "
"WHERE setdatabase = 0 AND setrole = "
@@ -1496,7 +1534,13 @@ dumpUserConfig(PGconn *conn, const char *username)
res = executeQuery(conn, buf->data);
if (PQntuples(res) > 0)
+ {
+ if (!header_done)
+ fprintf(OPF, "\n--\n-- User Configurations\n--\n");
+ header_done = true;
+
fprintf(OPF, "\n--\n-- User Config \"%s\"\n--\n\n", username);
+ }
for (int i = 0; i < PQntuples(res); i++)
{
@@ -1570,10 +1614,13 @@ expand_dbname_patterns(PGconn *conn,
* Dump contents of databases.
*/
static void
-dumpDatabases(PGconn *conn)
+dumpDatabases(PGconn *conn, ArchiveFormat archDumpFormat)
{
PGresult *res;
int i;
+ char db_subdir[MAXPGPATH];
+ char dbfilepath[MAXPGPATH];
+ FILE *map_file = NULL;
/*
* Skip databases marked not datallowconn, since we'd be unable to connect
@@ -1587,18 +1634,42 @@ dumpDatabases(PGconn *conn)
* doesn't have some failure mode with --clean.
*/
res = executeQuery(conn,
- "SELECT datname "
+ "SELECT datname, oid "
"FROM pg_database d "
"WHERE datallowconn AND datconnlimit != -2 "
"ORDER BY (datname <> 'template1'), datname");
- if (PQntuples(res) > 0)
+ if (archDumpFormat == archNull && PQntuples(res) > 0)
fprintf(OPF, "--\n-- Databases\n--\n\n");
+ /*
+ * If directory/tar/custom format is specified, create a subdirectory
+ * under the main directory and each database dump file or subdirectory
+ * will be created in that subdirectory by pg_dump.
+ */
+ if (archDumpFormat != archNull)
+ {
+ char map_file_path[MAXPGPATH];
+
+ snprintf(db_subdir, MAXPGPATH, "%s/databases", filename);
+
+ /* Create a subdirectory with 'databases' name under main directory. */
+ if (mkdir(db_subdir, 0755) != 0)
+ pg_fatal("could not create subdirectory \"%s\": %m", db_subdir);
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", filename);
+
+ /* Create a map file (to store dboid and dbname) */
+ map_file = fopen(map_file_path, PG_BINARY_W);
+ if (!map_file)
+ pg_fatal("could not open map file: %s", strerror(errno));
+ }
+
for (i = 0; i < PQntuples(res); i++)
{
char *dbname = PQgetvalue(res, i, 0);
- const char *create_opts;
+ char *oid = PQgetvalue(res, i, 1);
+ const char *create_opts = "";
int ret;
/* Skip template0, even if it's not marked !datallowconn. */
@@ -1612,9 +1683,27 @@ dumpDatabases(PGconn *conn)
continue;
}
+ /*
+ * If this is not a plain format dump, then append dboid and dbname to
+ * the map.dat file.
+ */
+ if (archDumpFormat != archNull)
+ {
+ if (archDumpFormat == archCustom)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".dmp", db_subdir, oid);
+ else if (archDumpFormat == archTar)
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\".tar", db_subdir, oid);
+ else
+ snprintf(dbfilepath, MAXPGPATH, "\"%s\"/\"%s\"", db_subdir, oid);
+
+ /* Put one line entry for dboid and dbname in map file. */
+ fprintf(map_file, "%s %s\n", oid, dbname);
+ }
+
pg_log_info("dumping database \"%s\"", dbname);
- fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
+ if (archDumpFormat == archNull)
+ fprintf(OPF, "--\n-- Database \"%s\" dump\n--\n\n", dbname);
/*
* We assume that "template1" and "postgres" already exist in the
@@ -1628,12 +1717,9 @@ dumpDatabases(PGconn *conn)
{
if (output_clean)
create_opts = "--clean --create";
- else
- {
- create_opts = "";
- /* Since pg_dump won't emit a \connect command, we must */
+ /* Since pg_dump won't emit a \connect command, we must */
+ else if (archDumpFormat == archNull)
fprintf(OPF, "\\connect %s\n\n", dbname);
- }
}
else
create_opts = "--create";
@@ -1641,19 +1727,30 @@ dumpDatabases(PGconn *conn)
if (filename)
fclose(OPF);
- ret = runPgDump(dbname, create_opts);
+ ret = runPgDump(dbname, create_opts, dbfilepath, archDumpFormat);
if (ret != 0)
pg_fatal("pg_dump failed on database \"%s\", exiting", dbname);
if (filename)
{
- OPF = fopen(filename, PG_BINARY_A);
+ char global_path[MAXPGPATH];
+
+ if (archDumpFormat != archNull)
+ snprintf(global_path, MAXPGPATH, "%s/global.dat", filename);
+ else
+ snprintf(global_path, MAXPGPATH, "%s", filename);
+
+ OPF = fopen(global_path, PG_BINARY_A);
if (!OPF)
pg_fatal("could not re-open the output file \"%s\": %m",
- filename);
+ global_path);
}
}
+ /* Close map file */
+ if (archDumpFormat != archNull)
+ fclose(map_file);
+
PQclear(res);
}
@@ -1663,7 +1760,8 @@ dumpDatabases(PGconn *conn)
* Run pg_dump on dbname, with specified options.
*/
static int
-runPgDump(const char *dbname, const char *create_opts)
+runPgDump(const char *dbname, const char *create_opts, char *dbfile,
+ ArchiveFormat archDumpFormat)
{
PQExpBufferData connstrbuf;
PQExpBufferData cmd;
@@ -1672,17 +1770,36 @@ runPgDump(const char *dbname, const char *create_opts)
initPQExpBuffer(&connstrbuf);
initPQExpBuffer(&cmd);
- printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
- pgdumpopts->data, create_opts);
-
/*
- * If we have a filename, use the undocumented plain-append pg_dump
- * format.
+ * If this is not a plain format dump, then append file name and dump
+ * format to the pg_dump command to get archive dump.
*/
- if (filename)
- appendPQExpBufferStr(&cmd, " -Fa ");
+ if (archDumpFormat != archNull)
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" -f %s %s", pg_dump_bin,
+ dbfile, create_opts);
+
+ if (archDumpFormat == archDirectory)
+ appendPQExpBufferStr(&cmd, " --format=directory ");
+ else if (archDumpFormat == archCustom)
+ appendPQExpBufferStr(&cmd, " --format=custom ");
+ else if (archDumpFormat == archTar)
+ appendPQExpBufferStr(&cmd, " --format=tar ");
+ }
else
- appendPQExpBufferStr(&cmd, " -Fp ");
+ {
+ printfPQExpBuffer(&cmd, "\"%s\" %s %s", pg_dump_bin,
+ pgdumpopts->data, create_opts);
+
+ /*
+ * If we have a filename, use the undocumented plain-append pg_dump
+ * format.
+ */
+ if (filename)
+ appendPQExpBufferStr(&cmd, " -Fa ");
+ else
+ appendPQExpBufferStr(&cmd, " -Fp ");
+ }
/*
* Append the database name to the already-constructed stem of connection
@@ -1827,3 +1944,91 @@ read_dumpall_filters(const char *filename, SimpleStringList *pattern)
filter_free(&fstate);
}
+
+/*
+ * create_or_open_dir
+ *
+ * This will create a new directory with given name. If there is already same
+ * empty directory exist, then use it.
+ */
+static void
+create_or_open_dir(const char *dirname)
+{
+ struct stat st;
+ bool is_empty = false;
+
+ /* we accept an empty existing directory */
+ if (stat(dirname, &st) == 0 && S_ISDIR(st.st_mode))
+ {
+ DIR *dir = opendir(dirname);
+
+ if (dir)
+ {
+ struct dirent *d;
+
+ is_empty = true;
+
+ while (errno = 0, (d = readdir(dir)))
+ {
+ if (strcmp(d->d_name, ".") != 0 && strcmp(d->d_name, "..") != 0)
+ {
+ is_empty = false;
+ break;
+ }
+ }
+
+ if (errno)
+ pg_fatal("could not read directory \"%s\": %m",
+ dirname);
+
+ if (closedir(dir))
+ pg_fatal("could not close directory \"%s\": %m",
+ dirname);
+ }
+
+ if (!is_empty)
+ {
+ pg_log_error("directory \"%s\" exists but is not empty", dirname);
+ pg_log_error_hint("If you want to dump data on this directory, either remove or empty "
+ "this directory \"%s\" or run %s "
+ "with an argument other than \"%s\".",
+ dirname, progname, dirname);
+ exit_nicely(1);
+ }
+ }
+ else if (mkdir(dirname, 0700) < 0)
+ pg_fatal("could not create directory \"%s\": %m", dirname);
+}
+
+/*
+ * parseDumpFormat
+ *
+ * This will validate dump formats.
+ */
+static ArchiveFormat
+parseDumpFormat(const char *format)
+{
+ ArchiveFormat archDumpFormat;
+
+ if (pg_strcasecmp(format, "c") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "custom") == 0)
+ archDumpFormat = archCustom;
+ else if (pg_strcasecmp(format, "d") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "directory") == 0)
+ archDumpFormat = archDirectory;
+ else if (pg_strcasecmp(format, "p") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "plain") == 0)
+ archDumpFormat = archNull;
+ else if (pg_strcasecmp(format, "t") == 0)
+ archDumpFormat = archTar;
+ else if (pg_strcasecmp(format, "tar") == 0)
+ archDumpFormat = archTar;
+ else
+ pg_fatal("unrecognized archive format \"%s\"; please specify \"c\", \"d\", \"p\", or \"t\"",
+ format);
+
+ return archDumpFormat;
+}
diff --git a/src/bin/pg_dump/pg_restore.c b/src/bin/pg_dump/pg_restore.c
index 47f7b0dd3a1..aa8887a4eb0 100644
--- a/src/bin/pg_dump/pg_restore.c
+++ b/src/bin/pg_dump/pg_restore.c
@@ -2,7 +2,7 @@
*
* pg_restore.c
* pg_restore is an utility extracting postgres database definitions
- * from a backup archive created by pg_dump using the archiver
+ * from a backup archive created by pg_dump/pg_dumpall using the archiver
* interface.
*
* pg_restore will read the backup archive and
@@ -41,11 +41,15 @@
#include "postgres_fe.h"
#include <ctype.h>
+#include <sys/stat.h>
#ifdef HAVE_TERMIOS_H
#include <termios.h>
#endif
+#include "common/string.h"
+#include "connectdb.h"
#include "fe_utils/option_utils.h"
+#include "fe_utils/string_utils.h"
#include "filter.h"
#include "getopt_long.h"
#include "parallel.h"
@@ -53,18 +57,35 @@
static void usage(const char *progname);
static void read_restore_filters(const char *filename, RestoreOptions *opts);
+static bool file_exists_in_directory(const char *dir, const char *filename);
+static int restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num);
+static int read_one_statement(StringInfo inBuf, FILE *pfile);
+static int restore_all_databases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts, int numWorkers);
+static int process_global_sql_commands(PGconn *conn, const char *dumpdirpath,
+ const char *outfile);
+static void copy_or_print_global_file(const char *outfile, FILE *pfile);
+static int get_dbnames_list_to_restore(PGconn *conn,
+ SimpleOidStringList * dbname_oid_list,
+ SimpleStringList db_exclude_patterns);
+static int get_dbname_oid_list_from_mfile(const char *dumpdirpath,
+ SimpleOidStringList * dbname_oid_list);
+static size_t quote_literal_internal(char *dst, const char *src, size_t len);
+static char *quote_literal_cstr(const char *rawstr);
int
main(int argc, char **argv)
{
RestoreOptions *opts;
int c;
- int exit_code;
int numWorkers = 1;
- Archive *AH;
char *inputFileSpec;
bool data_only = false;
bool schema_only = false;
+ int n_errors = 0;
+ bool globals_only = false;
+ SimpleStringList db_exclude_patterns = {NULL, NULL};
static int disable_triggers = 0;
static int enable_row_security = 0;
static int if_exists = 0;
@@ -90,6 +111,7 @@ main(int argc, char **argv)
{"clean", 0, NULL, 'c'},
{"create", 0, NULL, 'C'},
{"data-only", 0, NULL, 'a'},
+ {"globals-only", 0, NULL, 'g'},
{"dbname", 1, NULL, 'd'},
{"exit-on-error", 0, NULL, 'e'},
{"exclude-schema", 1, NULL, 'N'},
@@ -144,6 +166,7 @@ main(int argc, char **argv)
{"with-statistics", no_argument, &with_statistics, 1},
{"statistics-only", no_argument, &statistics_only, 1},
{"filter", required_argument, NULL, 4},
+ {"exclude-database", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -172,7 +195,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "acCd:ef:F:h:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
+ while ((c = getopt_long(argc, argv, "acCd:ef:F:gh:I:j:lL:n:N:Op:P:RsS:t:T:U:vwWx1",
cmdopts, NULL)) != -1)
{
switch (c)
@@ -199,11 +222,14 @@ main(int argc, char **argv)
if (strlen(optarg) != 0)
opts->formatName = pg_strdup(optarg);
break;
+ case 'g':
+ /* restore only global.dat file from directory */
+ globals_only = true;
+ break;
case 'h':
if (strlen(optarg) != 0)
opts->cparams.pghost = pg_strdup(optarg);
break;
-
case 'j': /* number of restore jobs */
if (!option_parse_int(optarg, "-j/--jobs", 1,
PG_MAX_JOBS,
@@ -318,6 +344,9 @@ main(int argc, char **argv)
exit(1);
opts->exit_on_error = true;
break;
+ case 6: /* database patterns to skip */
+ simple_string_list_append(&db_exclude_patterns, optarg);
+ break;
default:
/* getopt_long already emitted a complaint */
@@ -345,6 +374,13 @@ main(int argc, char **argv)
if (!opts->cparams.dbname && !opts->filename && !opts->tocSummary)
pg_fatal("one of -d/--dbname and -f/--file must be specified");
+ if (db_exclude_patterns.head != NULL && globals_only)
+ {
+ pg_log_error("option --exclude-database cannot be used together with -g/--globals-only");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ exit_nicely(1);
+ }
+
/* Should get at most one of -d and -f, else user is confused */
if (opts->cparams.dbname)
{
@@ -452,6 +488,113 @@ main(int argc, char **argv)
opts->formatName);
}
+ /*
+ * If toc.dat file does not present in current path, then check for
+ * global.dat. If global.dat file is present, then restore all the
+ * databases from map.dat(if exist) file list and skip restoring for
+ * --exclude-database patterns.
+ */
+ if (inputFileSpec != NULL && !file_exists_in_directory(inputFileSpec, "toc.dat") &&
+ file_exists_in_directory(inputFileSpec, "global.dat"))
+ {
+ PGconn *conn = NULL; /* Connection to restore global sql
+ * commands. */
+
+ /*
+ * User is suggested to use single database dump for --list option.
+ */
+ if (opts->tocSummary)
+ pg_fatal("option -l/--list cannot be used when restoring multiple databases by archive of pg_dumpall");
+
+ /*
+ * To restore multiple databases, -C (create database) option should
+ * be specified. Even there is single database in dump, report error
+ * because it might be possible that database hasn't created so better
+ * we report error.
+ */
+ if (!globals_only && opts->createDB != 1)
+ {
+ pg_log_error("-C/--create option should be specified when restoring multiple databases by archive of pg_dumpall");
+ pg_log_error_hint("Try \"%s --help\" for more information.", progname);
+ pg_log_error_hint("If db is already created and dump has single db dump, then use particular dump file.");
+ exit_nicely(1);
+ }
+
+ /*
+ * Connect to database to execute global sql commands from global.dat
+ * file.
+ */
+ if (opts->cparams.dbname)
+ {
+ conn = ConnectDatabase(opts->cparams.dbname, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+
+ if (!conn)
+ pg_fatal("could not connect to database \"%s\"", opts->cparams.dbname);
+ }
+
+ /* If globals-only, then return from here. */
+ if (globals_only)
+ {
+ /*
+ * Open global.dat file and execute/append all the global sql
+ * commands.
+ */
+ n_errors = process_global_sql_commands(conn, inputFileSpec,
+ opts->filename);
+
+ if (conn)
+ PQfinish(conn);
+
+ pg_log_info("databases restoring is skipped as -g/--globals-only option is specified");
+ }
+ else
+ {
+ /* Now restore all the databases from map.dat file. */
+ n_errors = restore_all_databases(conn, inputFileSpec, db_exclude_patterns,
+ opts, numWorkers);
+ }
+
+ /* Free db pattern list. */
+ simple_string_list_destroy(&db_exclude_patterns);
+ }
+ else /* process if global.dat file does not exist. */
+ {
+ if (db_exclude_patterns.head != NULL)
+ pg_fatal("option --exclude-database can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ if (globals_only)
+ pg_fatal("option -g/--globals-only can be used only when restoring multiple databases by archive of pg_dumpall");
+
+ n_errors = restore_one_database(inputFileSpec, opts, numWorkers, false, 0);
+ }
+
+ /* Done, print a summary of ignored errors during restore. */
+ if (n_errors)
+ {
+ pg_log_warning("errors ignored on restore: %d", n_errors);
+ return 1;
+ }
+
+ return 0;
+}
+
+/*
+ * restore_one_database
+ *
+ * This will restore one database using toc.dat file.
+ *
+ * returns the number of errors while doing restore.
+ */
+static int
+restore_one_database(const char *inputFileSpec, RestoreOptions *opts,
+ int numWorkers, bool append_data, int num)
+{
+ Archive *AH;
+ int n_errors;
+
AH = OpenArchive(inputFileSpec, opts->format);
SetArchiveOptions(AH, NULL, opts);
@@ -459,9 +602,15 @@ main(int argc, char **argv)
/*
* We don't have a connection yet but that doesn't matter. The connection
* is initialized to NULL and if we terminate through exit_nicely() while
- * it's still NULL, the cleanup function will just be a no-op.
+ * it's still NULL, the cleanup function will just be a no-op. If we are
+ * restoring multiple databases, then save index of exit_nicely so that we
+ * can use same slot for all the databases as we already closed the
+ * previous archive by CloseArchive.
*/
- on_exit_close_archive(AH);
+ if (!append_data || num == 0)
+ on_exit_close_archive(AH);
+ //else
+ //replace_on_exit_close_archive(AH);
/* Let the archiver know how noisy to be */
AH->verbose = opts->verbose;
@@ -481,25 +630,22 @@ main(int argc, char **argv)
else
{
ProcessArchiveRestoreOptions(AH);
- RestoreArchive(AH);
+ RestoreArchive(AH, append_data);
}
- /* done, print a summary of ignored errors */
- if (AH->n_errors)
- pg_log_warning("errors ignored on restore: %d", AH->n_errors);
+ n_errors = AH->n_errors;
/* AH may be freed in CloseArchive? */
- exit_code = AH->n_errors ? 1 : 0;
-
CloseArchive(AH);
- return exit_code;
+ return n_errors;
}
static void
usage(const char *progname)
{
- printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n\n"), progname);
+ printf(_("%s restores a PostgreSQL database from an archive created by pg_dump.\n"
+ "If archive is created by pg_dumpall, then restores multiple databases also. \n\n"), progname);
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... [FILE]\n"), progname);
@@ -517,6 +663,7 @@ usage(const char *progname)
printf(_(" -c, --clean clean (drop) database objects before recreating\n"));
printf(_(" -C, --create create the target database\n"));
printf(_(" -e, --exit-on-error exit on error, default is to continue\n"));
+ printf(_(" -g, --globals-only restore only global objects, no databases\n"));
printf(_(" -I, --index=NAME restore named index\n"));
printf(_(" -j, --jobs=NUM use this many parallel jobs to restore\n"));
printf(_(" -L, --use-list=FILENAME use table of contents from this file for\n"
@@ -529,6 +676,7 @@ usage(const char *progname)
printf(_(" -S, --superuser=NAME superuser user name to use for disabling triggers\n"));
printf(_(" -t, --table=NAME restore named relation (table, view, etc.)\n"));
printf(_(" -T, --trigger=NAME restore named trigger\n"));
+ printf(_(" --exclude-database=PATTERN exclude databases whose name matches with pattern\n"));
printf(_(" -x, --no-privileges skip restoration of access privileges (grant/revoke)\n"));
printf(_(" -1, --single-transaction restore as a single transaction\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
@@ -569,8 +717,8 @@ usage(const char *progname)
printf(_(" --role=ROLENAME do SET ROLE before restore\n"));
printf(_("\n"
- "The options -I, -n, -N, -P, -t, -T, and --section can be combined and specified\n"
- "multiple times to select multiple objects.\n"));
+ "The options -I, -n, -N, -P, -t, -T, --section, and --exclude-database can be combined\n"
+ "and specified multiple times to select multiple objects.\n"));
printf(_("\nIf no input file name is supplied, then standard input is used.\n\n"));
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
@@ -675,3 +823,620 @@ read_restore_filters(const char *filename, RestoreOptions *opts)
filter_free(&fstate);
}
+
+/*
+ * file_exists_in_directory
+ *
+ * Returns true if file exist in current directory.
+ */
+static bool
+file_exists_in_directory(const char *dir, const char *filename)
+{
+ struct stat st;
+ char buf[MAXPGPATH];
+
+ if (snprintf(buf, MAXPGPATH, "%s/%s", dir, filename) >= MAXPGPATH)
+ pg_fatal("directory name too long: \"%s\"", dir);
+
+ return (stat(buf, &st) == 0 && S_ISREG(st.st_mode));
+}
+
+/*
+ * read_one_statement
+ *
+ * This will start reading from passed file pointer using fgetc and read till
+ * semicolon(sql statement terminator for global.dat file)
+ *
+ * EOF is returned if end-of-file input is seen; time to shut down.
+ */
+
+static int
+read_one_statement(StringInfo inBuf, FILE *pfile)
+{
+ int c; /* character read from getc() */
+ int m;
+
+ StringInfoData q;
+
+ initStringInfo(&q);
+
+ resetStringInfo(inBuf);
+
+ /*
+ * Read characters until EOF or the appropriate delimiter is seen.
+ */
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != '\n' && c != ';')
+ {
+ appendStringInfoChar(inBuf, (char) c);
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ if (c != '\'' && c != '"' && c != ';' && c != '\n')
+ appendStringInfoChar(inBuf, (char) c);
+ else
+ break;
+ }
+ }
+
+ if (c == '\'' || c == '"')
+ {
+ appendStringInfoChar(&q, (char) c);
+ m = c;
+
+ while ((c = fgetc(pfile)) != EOF)
+ {
+ appendStringInfoChar(&q, (char) c);
+
+ if (c == m)
+ {
+ appendStringInfoString(inBuf, q.data);
+ resetStringInfo(&q);
+ break;
+ }
+ }
+ }
+
+ if (c == ';')
+ {
+ appendStringInfoChar(inBuf, (char) ';');
+ break;
+ }
+
+ if (c == '\n')
+ appendStringInfoChar(inBuf, (char) '\n');
+ }
+
+ /* No input before EOF signal means time to quit. */
+ if (c == EOF && inBuf->len == 0)
+ return EOF;
+
+ /* Add '\0' to make it look the same as message case. */
+ appendStringInfoChar(inBuf, (char) '\0');
+
+ return 'Q';
+}
+
+/*
+ * get_dbnames_list_to_restore
+ *
+ * This will mark for skipping any entries from dbname_oid_list that pattern match an
+ * entry in the db_exclude_patterns list.
+ *
+ * Returns the number of database to be restored.
+ *
+ */
+static int
+get_dbnames_list_to_restore(PGconn *conn,
+ SimpleOidStringList * dbname_oid_list,
+ SimpleStringList db_exclude_patterns)
+{
+ int count_db = 0;
+ PQExpBuffer query;
+ PGresult *res;
+
+ query = createPQExpBuffer();
+
+ if (!conn)
+ pg_log_info("considering PATTERN as NAME for --exclude-database option as no db connection while doing pg_restore.");
+
+ /*
+ * Process one by one all dbnames and if specified to skip restoring, then
+ * remove dbname from list.
+ */
+ for (SimpleOidStringListCell * db_cell = dbname_oid_list->head;
+ db_cell; db_cell = db_cell->next)
+ {
+ bool skip_db_restore = false;
+
+ for (SimpleStringListCell *pat_cell = db_exclude_patterns.head; pat_cell; pat_cell = pat_cell->next)
+ {
+ /*
+ * the construct pattern matching query: SELECT 1 WHERE XXX
+ * OPERATOR(pg_catalog.~) '^(PATTERN)$' COLLATE pg_catalog.default
+ *
+ * XXX represents the string literal database name derived from
+ * the dbname_oid_list, which is initially extracted from the
+ * map.dat file located in the backup directory. that's why we
+ * need quote_literal_cstr.
+ *
+ * If no db connection, then consider PATTERN as NAME.
+ */
+ if (pg_strcasecmp(db_cell->str, pat_cell->val) == 0)
+ skip_db_restore = true;
+ else if (conn)
+ {
+ int dotcnt;
+
+ appendPQExpBufferStr(query, "SELECT 1 ");
+ processSQLNamePattern(conn, query, pat_cell->val, false,
+ false, NULL, quote_literal_cstr(db_cell->str),
+ NULL, NULL, NULL, &dotcnt);
+
+ if (dotcnt > 0)
+ {
+ pg_log_error("improper qualified name (too many dotted names): %s",
+ db_cell->str);
+ PQfinish(conn);
+ exit_nicely(1);
+ }
+
+ res = executeQuery(conn, query->data);
+
+ if ((PQresultStatus(res) == PGRES_TUPLES_OK) && PQntuples(res))
+ {
+ skip_db_restore = true;
+ pg_log_info("database \"%s\" matches exclude pattern: \"%s\"", db_cell->str, pat_cell->val);
+ }
+
+ PQclear(res);
+ resetPQExpBuffer(query);
+ }
+
+ if (skip_db_restore)
+ break;
+ }
+
+ /* Increment count if database needs to be restored. */
+ if (skip_db_restore)
+ {
+ pg_log_info("excluding database \"%s\"", db_cell->str);
+ db_cell->oid = InvalidOid;
+ }
+ else
+ {
+ count_db++;
+ }
+ }
+
+ return count_db;
+}
+
+/*
+ * get_dbname_oid_list_from_mfile
+ *
+ * Open map.dat file and read line by line and then prepare a list of database
+ * names and corresponding db_oid.
+ *
+ * Returns, total number of database names in map.dat file.
+ */
+static int
+get_dbname_oid_list_from_mfile(const char *dumpdirpath, SimpleOidStringList * dbname_oid_list)
+{
+ FILE *pfile;
+ char map_file_path[MAXPGPATH];
+ char line[MAXPGPATH];
+ int count = 0;
+
+ /*
+ * If there is only global.dat file in dump, then return from here as
+ * there is no database to restore.
+ */
+ if (!file_exists_in_directory(dumpdirpath, "map.dat"))
+ {
+ pg_log_info("databases restoring is skipped as map.dat file is not present in \"%s\"", dumpdirpath);
+ return 0;
+ }
+
+ snprintf(map_file_path, MAXPGPATH, "%s/map.dat", dumpdirpath);
+
+ /* Open map.dat file. */
+ pfile = fopen(map_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open map.dat file: \"%s\"", map_file_path);
+
+ /* Append all the dbname and db_oid to the list. */
+ while ((fgets(line, MAXPGPATH, pfile)) != NULL)
+ {
+ Oid db_oid = InvalidOid;
+ char db_oid_str[MAXPGPATH + 1] = {'\0'};
+ char dbname[MAXPGPATH + 1] = {'\0'};
+
+ /* Extract dboid. */
+ sscanf(line, "%u", &db_oid);
+ sscanf(line, "%20s", db_oid_str);
+
+ /* Now copy dbname. */
+ strcpy(dbname, line + strlen(db_oid_str) + 1);
+
+ /* Remove \n from dbanme. */
+ dbname[strlen(dbname) - 1] = '\0';
+
+ pg_log_info("found database \"%s\" (OID: %u) in map.dat file while restoring.", dbname, db_oid);
+
+ /* Report error and exit if the file has any corrupted data. */
+ if (!OidIsValid(db_oid) || strlen(dbname) == 0)
+ pg_fatal("invalid entry in map.dat file at line : %d", count + 1);
+
+ /*
+ * XXX : before adding dbname into list, we can verify that this db
+ * needs to skipped for restore or not but as of now, we are making a
+ * list of all the databases.
+ */
+ simple_oid_string_list_append(dbname_oid_list, db_oid, dbname);
+ count++;
+ }
+
+ /* Close map.dat file. */
+ fclose(pfile);
+
+ return count;
+}
+
+/*
+ * restore_all_databases
+ *
+ * This will restore databases those dumps are present in
+ * directory based on map.dat file mapping.
+ *
+ * This will skip restoring for databases that are specified with
+ * exclude-database option.
+ *
+ * returns, number of errors while doing restore.
+ */
+static int
+restore_all_databases(PGconn *conn, const char *dumpdirpath,
+ SimpleStringList db_exclude_patterns, RestoreOptions *opts,
+ int numWorkers)
+{
+ SimpleOidStringList dbname_oid_list = {NULL, NULL};
+ int num_db_restore = 0;
+ int num_total_db;
+ int n_errors_total;
+ int count = 0;
+ char *connected_db = NULL;
+ bool dumpData = opts->dumpData;
+ bool dumpSchema = opts->dumpSchema;
+ bool dumpStatistics = opts->dumpSchema;
+
+ /* Save db name to reuse it for all the database. */
+ if (opts->cparams.dbname)
+ connected_db = opts->cparams.dbname;
+
+ num_total_db = get_dbname_oid_list_from_mfile(dumpdirpath, &dbname_oid_list);
+
+ /*
+ * If map.dat has no entry, return from here after processing global.dat
+ * file.
+ */
+ if (dbname_oid_list.head == NULL)
+ return process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ pg_log_info("found total %d database names in map.dat file", num_total_db);
+
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("postgres", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+
+ /* Try with template1. */
+ if (!conn)
+ {
+ pg_log_info("trying to connect database \"template1\" as failed to connect to database \"postgres\" to dump into out file");
+
+ conn = ConnectDatabase("template1", NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ }
+ }
+
+ /*
+ * processing pg_restore --exclude-database=PATTERN/NAME if no connection.
+ */
+ num_db_restore = get_dbnames_list_to_restore(conn, &dbname_oid_list,
+ db_exclude_patterns);
+
+ /* Open global.dat file and execute/append all the global sql commands. */
+ n_errors_total = process_global_sql_commands(conn, dumpdirpath, opts->filename);
+
+ /* Close the db connection as we are done with globals and patterns. */
+ if (conn)
+ PQfinish(conn);
+
+ /* Exit if no db needs to be restored. */
+ if (dbname_oid_list.head == NULL || num_db_restore == 0)
+ {
+ pg_log_info("no database needs to restore out of %d databases", num_total_db);
+ return n_errors_total;
+ }
+
+ pg_log_info("needs to restore %d databases out of %d databases", num_db_restore, num_total_db);
+
+ /*
+ * Till now, we made a list of databases, those needs to be restored after
+ * skipping names of exclude-database. Now we can launch parallel workers
+ * to restore these databases.
+ */
+ for (SimpleOidStringListCell * db_cell = dbname_oid_list.head;
+ db_cell; db_cell = db_cell->next)
+ {
+ char subdirpath[MAXPGPATH];
+ char subdirdbpath[MAXPGPATH];
+ char dbfilename[MAXPGPATH];
+ int n_errors;
+
+ /* ignore dbs marked for skipping */
+ if (db_cell->oid == InvalidOid)
+ continue;
+
+ /*
+ * We need to reset override_dbname so that objects can be restored
+ * into already created database. (used with -d/--dbname option)
+ */
+ if (opts->cparams.override_dbname)
+ {
+ pfree(opts->cparams.override_dbname);
+ opts->cparams.override_dbname = NULL;
+ }
+
+ snprintf(subdirdbpath, MAXPGPATH, "%s/databases", dumpdirpath);
+
+ /*
+ * Validate database dump file. If there is .tar or .dmp file exist
+ * then consider particular file, otherwise just append dboid to the
+ * databases folder.
+ */
+ snprintf(dbfilename, MAXPGPATH, "%u.tar", db_cell->oid);
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.tar", dumpdirpath, db_cell->oid);
+ else
+ {
+ snprintf(dbfilename, MAXPGPATH, "%u.dmp", db_cell->oid);
+
+ if (file_exists_in_directory(subdirdbpath, dbfilename))
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u.dmp", dumpdirpath, db_cell->oid);
+ else
+ snprintf(subdirpath, MAXPGPATH, "%s/databases/%u", dumpdirpath, db_cell->oid);
+ }
+
+ pg_log_info("restoring database \"%s\"", db_cell->str);
+
+ /* If database is already created, then don't set createDB flag. */
+ if (opts->cparams.dbname)
+ {
+ PGconn *test_conn;
+
+ test_conn = ConnectDatabase(db_cell->str, NULL, opts->cparams.pghost,
+ opts->cparams.pgport, opts->cparams.username, TRI_DEFAULT,
+ false, progname, NULL, NULL, NULL, NULL);
+ if (test_conn)
+ {
+ PQfinish(test_conn);
+
+ /* Use already created database for connection. */
+ opts->createDB = 0;
+ opts->cparams.dbname = db_cell->str;
+ }
+ else
+ {
+ /* we'll have to create it */
+ opts->createDB = 1;
+ opts->cparams.dbname = connected_db;
+ }
+ }
+
+ /*
+ * Reset flags - might have been reset in pg_backup_archiver.c by the
+ * previous restore.
+ */
+ opts->dumpData = dumpData;
+ opts->dumpSchema = dumpSchema;
+ opts->dumpStatistics = dumpStatistics;
+
+ /* Restore single database. */
+ n_errors = restore_one_database(subdirpath, opts, numWorkers, true, count);
+
+ /* Print a summary of ignored errors during single database restore. */
+ if (n_errors)
+ {
+ n_errors_total += n_errors;
+ pg_log_warning("errors ignored on database \"%s\" restore: %d", db_cell->str, n_errors);
+ }
+
+ count++;
+ }
+
+ /* Log number of processed databases. */
+ pg_log_info("number of restored databases are %d", num_db_restore);
+
+ /* Free dbname and dboid list. */
+ simple_oid_string_list_destroy(&dbname_oid_list);
+
+ return n_errors_total;
+}
+
+/*
+ * process_global_sql_commands
+ *
+ * This will open global.dat file and will execute all global sql commands one
+ * by one statement.
+ * Semicolon is considered as statement terminator. If outfile is passed, then
+ * this will copy all sql commands into outfile rather then executing them.
+ *
+ * returns the number of errors while processing global.dat
+ */
+static int
+process_global_sql_commands(PGconn *conn, const char *dumpdirpath, const char *outfile)
+{
+ char global_file_path[MAXPGPATH];
+ PGresult *result;
+ StringInfoData sqlstatement, user_create;
+ FILE *pfile;
+ int n_errors = 0;
+
+ snprintf(global_file_path, MAXPGPATH, "%s/global.dat", dumpdirpath);
+
+ /* Open global.dat file. */
+ pfile = fopen(global_file_path, PG_BINARY_R);
+
+ if (pfile == NULL)
+ pg_fatal("could not open global.dat file: \"%s\"", global_file_path);
+
+ /*
+ * If outfile is given, then just copy all global.dat file data into
+ * outfile.
+ */
+ if (outfile)
+ {
+ copy_or_print_global_file(outfile, pfile);
+ return 0;
+ }
+
+ /* Init sqlstatement to append commands. */
+ initStringInfo(&sqlstatement);
+
+ /* creation statement for our current role */
+ initStringInfo(&user_create);
+ appendStringInfoString(&user_create, "CREATE ROLE ");
+ /* should use fmtId here, but we don't know the encoding */
+ appendStringInfoString(&user_create, PQuser(conn));
+ appendStringInfoString(&user_create, ";");
+
+ /* Process file till EOF and execute sql statements. */
+ while (read_one_statement(&sqlstatement, pfile) != EOF)
+ {
+ /* don't try to create the role we are connected as */
+ if (strstr(sqlstatement.data, user_create.data))
+ continue;
+
+ pg_log_info("executing query: %s", sqlstatement.data);
+ result = PQexec(conn, sqlstatement.data);
+
+ switch (PQresultStatus(result))
+ {
+ case PGRES_COMMAND_OK:
+ case PGRES_TUPLES_OK:
+ case PGRES_EMPTY_QUERY:
+ break;
+ default:
+ n_errors++;
+ pg_log_error("could not execute query: \"%s\" \nCommand was: \"%s\"", PQerrorMessage(conn), sqlstatement.data);
+ }
+ PQclear(result);
+ }
+
+ /* Print a summary of ignored errors during global.dat. */
+ if (n_errors)
+ pg_log_warning("errors ignored on global.dat file restore: %d", n_errors);
+
+ fclose(pfile);
+
+ return n_errors;
+}
+
+/*
+ * copy_or_print_global_file
+ *
+ * This will copy global.dat file into out file. If "-" is used as outfile,
+ * then print commands to the stdout.
+ */
+static void
+copy_or_print_global_file(const char *outfile, FILE *pfile)
+{
+ char out_file_path[MAXPGPATH];
+ FILE *OPF;
+ int c;
+
+ /* "-" is used for stdout. */
+ if (strcmp(outfile, "-") == 0)
+ OPF = stdout;
+ else
+ {
+ snprintf(out_file_path, MAXPGPATH, "%s", outfile);
+ OPF = fopen(out_file_path, PG_BINARY_W);
+
+ if (OPF == NULL)
+ {
+ fclose(pfile);
+ pg_fatal("could not open file: \"%s\"", outfile);
+ }
+ }
+
+ /* Append global.dat into out file or print to the stdout. */
+ while ((c = fgetc(pfile)) != EOF)
+ fputc(c, OPF);
+
+ fclose(pfile);
+
+ /* Close out file. */
+ if (strcmp(outfile, "-") != 0)
+ fclose(OPF);
+}
+
+/*
+ * quote_literal_internal
+ */
+static size_t
+quote_literal_internal(char *dst, const char *src, size_t len)
+{
+ const char *s;
+ char *savedst = dst;
+
+ for (s = src; s < src + len; s++)
+ {
+ if (*s == '\\')
+ {
+ *dst++ = ESCAPE_STRING_SYNTAX;
+ break;
+ }
+ }
+
+ *dst++ = '\'';
+ while (len-- > 0)
+ {
+ if (SQL_STR_DOUBLE(*src, true))
+ *dst++ = *src;
+ *dst++ = *src++;
+ }
+ *dst++ = '\'';
+
+ return dst - savedst;
+}
+
+/*
+ * quote_literal_cstr
+ *
+ * returns a properly quoted literal
+ * copied from src/backend/utils/adt/quote.c
+ */
+static char *
+quote_literal_cstr(const char *rawstr)
+{
+ char *result;
+ int len;
+ int newlen;
+
+ len = strlen(rawstr);
+
+ /* We make a worst-case result area; wasting a little space is OK */
+ result = pg_malloc(len * 2 + 3 + 1);
+
+ newlen = quote_literal_internal(result, rawstr, len);
+ result[newlen] = '\0';
+
+ return result;
+}
diff --git a/src/bin/pg_dump/t/001_basic.pl b/src/bin/pg_dump/t/001_basic.pl
old mode 100644
new mode 100755
index 37d893d5e6a..0bbcdbe84a7
--- a/src/bin/pg_dump/t/001_basic.pl
+++ b/src/bin/pg_dump/t/001_basic.pl
@@ -237,6 +237,11 @@ command_fails_like(
'pg_restore: options -C\/--create and -1\/--single-transaction cannot be used together'
);
+command_fails_like(
+ [ 'pg_restore', '--exclude-database=foo', '--globals-only', '-d', 'xxx' ],
+ qr/\Qpg_restore: error: option --exclude-database cannot be used together with -g\/--globals-only\E/,
+ 'pg_restore: option --exclude-database cannot be used together with -g/--globals-only');
+
# also fails for -r and -t, but it seems pointless to add more tests for those.
command_fails_like(
[ 'pg_dumpall', '--exclude-database=foo', '--globals-only' ],
@@ -244,4 +249,8 @@ command_fails_like(
'pg_dumpall: option --exclude-database cannot be used together with -g/--globals-only'
);
+command_fails_like(
+ [ 'pg_dumpall', '--format', 'x' ],
+ qr/\Qpg_dumpall: error: unrecognized archive format "x";\E/,
+ 'pg_dumpall: unrecognized archive format');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b66cecd8799..95ec8fbb141 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2732,6 +2732,8 @@ ShutdownMode
SignTSVector
SimpleActionList
SimpleActionListCell
+SimpleDatabaseOidList
+SimpleDatabaseOidListCell
SimpleEcontextStackEntry
SimpleOidList
SimpleOidListCell
--
2.39.3
^ permalink raw reply [nested|flat] 29+ messages in thread
end of thread, other threads:[~2025-04-01 05:59 UTC | newest]
Thread overview: 29+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-03-05 01:12 Re: Non-text mode for pg_dumpall Mahendra Singh Thalor <[email protected]>
2025-03-05 15:12 ` Álvaro Herrera <[email protected]>
2025-03-05 16:48 ` Mahendra Singh Thalor <[email protected]>
2025-03-10 07:54 ` jian he <[email protected]>
2025-03-11 14:11 ` Mahendra Singh Thalor <[email protected]>
2025-03-11 14:42 ` Álvaro Herrera <[email protected]>
2025-03-11 15:41 ` Mahendra Singh Thalor <[email protected]>
2025-03-11 17:05 ` Álvaro Herrera <[email protected]>
2025-03-11 17:52 ` Dagfinn Ilmari Mannsåker <[email protected]>
2025-03-11 20:16 ` Andrew Dunstan <[email protected]>
2025-03-11 21:03 ` Álvaro Herrera <[email protected]>
2025-03-11 22:37 ` Andrew Dunstan <[email protected]>
2025-03-11 23:14 ` Isaac Morland <[email protected]>
2025-03-12 07:24 ` Laurenz Albe <[email protected]>
2025-03-12 07:03 ` jian he <[email protected]>
2025-03-12 15:48 ` Andrew Dunstan <[email protected]>
2025-03-19 06:41 ` Mahendra Singh Thalor <[email protected]>
2025-03-27 21:15 ` Andrew Dunstan <[email protected]>
2025-03-28 22:20 ` Andrew Dunstan <[email protected]>
2025-03-29 05:17 ` Mahendra Singh Thalor <[email protected]>
2025-03-30 16:50 ` Andrew Dunstan <[email protected]>
2025-03-30 22:05 ` Andrew Dunstan <[email protected]>
2025-03-31 09:34 ` Mahendra Singh Thalor <[email protected]>
2025-03-31 13:57 ` Andrew Dunstan <[email protected]>
2025-03-31 16:16 ` Mahendra Singh Thalor <[email protected]>
2025-03-31 17:16 ` Andrew Dunstan <[email protected]>
2025-03-31 17:20 ` Andrew Dunstan <[email protected]>
2025-03-31 18:12 ` Álvaro Herrera <[email protected]>
2025-04-01 05:59 ` Mahendra Singh Thalor <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox