public inbox for [email protected]
help / color / mirror / Atom feedFrom: Justin Pryzby <[email protected]>
To: Jan Wieck <[email protected]>
Cc: Tom Lane <[email protected]>
Cc: Bruce Momjian <[email protected]>
Cc: Zhihong Yu <[email protected]>
Cc: Andrew Dunstan <[email protected]>
Cc: Magnus Hagander <[email protected]>
Cc: Robins Tharakan <[email protected]>
Cc: Peter Eisentraut <[email protected]>
Cc: [email protected]
Subject: Re: pg_upgrade failing for 200+ million Large Objects
Date: Sat, 11 Dec 2021 16:43:08 -0600
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
On Wed, Mar 24, 2021 at 12:05:27PM -0400, Jan Wieck wrote:
> On 3/24/21 12:04 PM, Jan Wieck wrote:
> > In any case I changed the options so that they behave the same way, the
> > existing -o and -O (for old/new postmaster options) work. I don't think
> > it would be wise to have option forwarding work differently between
> > options for postmaster and options for pg_dump/pg_restore.
>
> Attaching the actual diff might help.
I think the original issue with XIDs was fixed by 74cf7d46a.
Are you still planning to progress the patches addressing huge memory use of
pg_restore?
Note this other, old thread on -general, which I believe has variations on the
same patches.
https://www.postgresql.org/message-id/flat/[email protected]
There was discussion about using pg_restore --single. Note that that was used
at some point in the past: see 12ee6ec71 and 861ad67bd.
The immediate problem is that --single conflicts with --create.
I cleaned up a patch I'd written to work around that. It preserves DB settings
and passes pg_upgrade's test. It's probably not portable as written, but if need be
could pass an empty file instead of /dev/null...
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 3628bd74a7..9c504aff79 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -364,6 +364,16 @@ create_new_objects(void)
DbInfo *old_db = &old_cluster.dbarr.dbs[dbnum];
const char *create_opts;
+ PQExpBufferData connstr,
+ escaped_connstr;
+
+ initPQExpBuffer(&connstr);
+ initPQExpBuffer(&escaped_connstr);
+ appendPQExpBufferStr(&connstr, "dbname=");
+ appendConnStrVal(&connstr, old_db->db_name);
+ appendShellString(&escaped_connstr, connstr.data);
+ termPQExpBuffer(&connstr);
+
/* Skip template1 in this pass */
if (strcmp(old_db->db_name, "template1") == 0)
continue;
@@ -378,18 +388,31 @@ create_new_objects(void)
* propagate its database-level properties.
*/
if (strcmp(old_db->db_name, "postgres") == 0)
- create_opts = "--clean --create";
+ create_opts = "--clean";
else
- create_opts = "--create";
+ create_opts = "";
+ /* Create the DB but exclude all objects */
parallel_exec_prog(log_file_name,
NULL,
"\"%s/pg_restore\" %s %s --exit-on-error --verbose "
+ "--create -L /dev/null "
"--dbname template1 \"%s\"",
new_cluster.bindir,
cluster_conn_opts(&new_cluster),
create_opts,
sql_file_name);
+
+ parallel_exec_prog(log_file_name,
+ NULL,
+ "\"%s/pg_restore\" %s %s --exit-on-error --verbose --single "
+ "--dbname=%s \"%s\"",
+ new_cluster.bindir,
+ cluster_conn_opts(&new_cluster),
+ create_opts,
+ escaped_connstr.data,
+ sql_file_name);
+
}
/* reap all children */
view thread (49+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: pg_upgrade failing for 200+ million Large Objects
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox