Reply-To: jesper.pedersen@redhat.com
Subject: Re: pg_upgrade: Pass -j down to vacuumdb
To: Michael Paquier <michael@paquier.xyz>,
 "Jamison, Kirk" <k.jamison@jp.fujitsu.com>
Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>,
 "fabriziomello@gmail.com" <fabriziomello@gmail.com>,
 Tom Lane <tgl@sss.pgh.pa.us>, Alvaro Herrera <alvherre@2ndquadrant.com>,
 "pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>,
 Bruce Momjian <bruce@momjian.us>
References: <5b7d95ba-8920-357d-1ec6-1084ce34a9e6@redhat.com>
 <D09B13F772D2274BB348A310EE3027C64137E4@g01jpexmbkw24>
 <c448acf6-7c38-3d6d-c663-5387ccffb906@redhat.com>
 <D09B13F772D2274BB348A310EE3027C6413F1A@g01jpexmbkw24>
 <388de5bc-858b-4686-d2e5-d0b21981dd72@redhat.com>
 <CAFcNs+qCPbzErgcmTDqW2qzkt7kzarjWqD48VimKD4mXM0nutA@mail.gmail.com>
 <4ce565f9-d21f-039f-8f5a-2e16e6aa1ae4@redhat.com>
 <e103a8a1-a6f1-5709-6ac6-95684fd5bff2@2ndquadrant.com>
 <be5cd0aa-304f-1b9f-68b4-0c747c67f73b@redhat.com>
 <D09B13F772D2274BB348A310EE3027C64146AF@g01jpexmbkw24>
 <20190129050830.GE3121@paquier.xyz>
From: Jesper Pedersen <jesper.pedersen@redhat.com>
Organization: Red Hat
Message-ID: <37538daf-5f5b-afed-b3f5-a64e154e6433@redhat.com>
Date: Tue, 29 Jan 2019 15:19:28 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.4.0
MIME-Version: 1.0
In-Reply-To: <20190129050830.GE3121@paquier.xyz>
Content-Type: multipart/mixed;
 boundary="------------C6D2C4D34E9C414DDE6F7784"
Content-Language: en-US
Precedence: bulk

This is a multi-part message in MIME format.
--------------C6D2C4D34E9C414DDE6F7784
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

On 1/29/19 12:08 AM, Michael Paquier wrote:
> On Tue, Jan 29, 2019 at 12:35:30AM +0000, Jamison, Kirk wrote:
>> I just checked the patch.
>> As per advice, you removed the versioning and specified --jobs.
>> The patch still applies, builds and passed the tests successfully.
> 
> I would document the optional VACUUM_OPTS on the page of pg_upgrade.
> If Peter thinks it is fine to not do so, that's fine for me as well.
> 

I added most of the documentation back, as requested by Kirk.

> It seems to me that the latest patch sent is incorrect for multiple
> reasons:
> 1) You still enforce -j to use the number of jobs that the caller of
> pg_upgrade provides, and we agreed that both things are separate
> concepts upthread, no?  What has been suggested by Alvaro is to add a
> comment so as one can use VACUUM_OPTS with -j optionally, instead of
> suggesting a full-fledged vacuumdb command which depends on what
> pg_upgrade uses.  So there is no actual need for the if/else
> complication business.

I think it is ok for the echo command to highlight to the user that 
running --analyze-only using the same amount of jobs will give a faster 
result.

> 2) Perhaps we need to worry about the second vacuumdb --all command,
> which may want custom options, which are not necessarily the same as
> the options of the first command?  I don't think we need to care as it
> applies only to an upgraded cluster using something older 8.4, just
> wondering..

I think that --all --analyze-in-stages is what most people want. And 
with the $VACUUMDB_OPTS variable people have an option to put in more, 
such as -j X.

Best regards,
  Jesper


--------------C6D2C4D34E9C414DDE6F7784
Content-Type: text/x-patch;
 name="v6_0001-Highlight-that-the-jobs-option-isn-t-passed-down-to-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v6_0001-Highlight-that-the-jobs-option-isn-t-passed-down-to-";
 filename*1=".patch"

From b02e1f414e309361595ab3fe84fb6f66bcf63265 Mon Sep 17 00:00:00 2001
From: jesperpedersen <jesper.pedersen@redhat.com>
Date: Fri, 21 Dec 2018 05:05:31 -0500
Subject: [PATCH] Highlight that the --jobs option isn't passed down to
 vacuumdb by default.

Author: Jesper Pedersen <jesper.pedersen@redhat.com>
---
 doc/src/sgml/ref/pgupgrade.sgml |  6 +++++-
 src/bin/pg_upgrade/check.c      | 28 ++++++++++++++++++++++------
 2 files changed, 27 insertions(+), 7 deletions(-)

diff --git a/doc/src/sgml/ref/pgupgrade.sgml b/doc/src/sgml/ref/pgupgrade.sgml
index 7e1afaf0a5..84a7ccb6f1 100644
--- a/doc/src/sgml/ref/pgupgrade.sgml
+++ b/doc/src/sgml/ref/pgupgrade.sgml
@@ -395,7 +395,11 @@ NET STOP postgresql-&majorversion;
      in parallel;  a good place to start is the maximum of the number of
      CPU cores and tablespaces.  This option can dramatically reduce the
      time to upgrade a multi-database server running on a multiprocessor
-     machine.
+     machine. Note that this option isn't passed to the
+     <application>vacuumdb</application> application by default.
+     The system environment variable <literal>VACUUMDB_OPTS</literal> can be
+     used to pass extra options to the <application>vacuumdb</application>
+     application during the script execution.
     </para>
 
     <para>
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index fc5aa7010f..847e604088 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -495,15 +495,31 @@ create_script_for_cluster_analyze(char **analyze_script_file_name)
 			ECHO_QUOTE, ECHO_QUOTE);
 	fprintf(script, "echo %sthis script and run:%s\n",
 			ECHO_QUOTE, ECHO_QUOTE);
-	fprintf(script, "echo %s    \"%s/vacuumdb\" %s--all %s%s\n", ECHO_QUOTE,
-			new_cluster.bindir, user_specification.data,
-	/* Did we copy the free space files? */
-			(GET_MAJOR_VERSION(old_cluster.major_version) >= 804) ?
-			"--analyze-only" : "--analyze", ECHO_QUOTE);
+	if (user_opts.jobs <= 1)
+	{
+		fprintf(script, "echo %s    \"%s/vacuumdb\" %s--all %s%s\n", ECHO_QUOTE,
+				new_cluster.bindir, user_specification.data,
+				/* Did we copy the free space files? */
+				(GET_MAJOR_VERSION(old_cluster.major_version) >= 804) ?
+				"--analyze-only" : "--analyze", ECHO_QUOTE);
+	}
+	else
+	{
+		fprintf(script, "echo %s    \"%s/vacuumdb\" %s--jobs %d --all %s%s\n", ECHO_QUOTE,
+				new_cluster.bindir, user_specification.data, user_opts.jobs,
+				/* Did we copy the free space files? */
+				(GET_MAJOR_VERSION(old_cluster.major_version) >= 804) ?
+				"--analyze-only" : "--analyze", ECHO_QUOTE);
+	}
 	fprintf(script, "echo%s\n\n", ECHO_BLANK);
 
-	fprintf(script, "\"%s/vacuumdb\" %s--all --analyze-in-stages\n",
+#ifndef WIN32
+	fprintf(script, "\"%s/vacuumdb\" %s$VACUUMDB_OPTS --all --analyze-in-stages\n",
 			new_cluster.bindir, user_specification.data);
+#else
+	fprintf(script, "\"%s\\vacuumdb.exe\" %s%%VACUUMDB_OPTS%% --all --analyze-in-stages\n",
+			new_cluster.bindir, user_specification.data);
+#endif
 	/* Did we copy the free space files? */
 	if (GET_MAJOR_VERSION(old_cluster.major_version) < 804)
 		fprintf(script, "\"%s/vacuumdb\" %s--all\n", new_cluster.bindir,
-- 
2.17.2


--------------C6D2C4D34E9C414DDE6F7784--