Subject: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

From: Michael Paquier <[email-masked]>
Date: Sat, 15 May 2021 11:26:22 +0900

Hi,
(Adding Andrew in CC for the buildfarm and PostgresNode parts.)

$subject has been around for a couple of years now, with the following
threads:
https://www.postgresql.org/message-id/[email-masked]
https://www.postgresql.org/message-id/[email-masked]

An advantage of moving to TAP is that we can then remove the support
for upgrades within the MSVC scripts, and also remove pg_upgrade's
test.sh that has accumulated tweaks that are solved by the TAP tests,
resulting in cleanup:
 8 files changed, 230 insertions(+), 403 deletions(-)

Based on the past discussions, there were two obstacles preventing to
do this switch:
- Support for tests with older versions, something where the gap as
been closed thanks to Andrew's work in 4c4eaf3d.
- Buildfarm support, and I am not sure how things need to be extended
there.

Another thing to note is that HEAD uses oldbindir, bindir and libdir
to track the location of the old and new libraries and binaries.  With
the infrastructure in place, once can define only an install path for
a PostgresNode, so this version uses only two variables:
- oldinstall, for the installation path of the version to upgrade
from.
- oldsrc, to point to the source of the old version.

It is not difficult to switch to one approach or the other, but
reducing the logic to a minimum number of variables is a good deal to
take IMO.

I have been testing this patch a bit with older versions, down to 12,
and that was logically working, and PostgresNode may need more to be
able to work with ~11, as documented in get_new_node().  And I have
not tested that with MSVC yet.  Anyway, attached is a new patch to
make the discussion move on.  Even if there is still work to be done
here, would people here still support this switch?

Thanks,
--
Michael
From 557e1bd4cea5e1081ea77e364c3e7f37ad3c5268 Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email-masked]>
Date: Sat, 15 May 2021 10:44:18 +0900
Subject: [PATCH] Switch tests of pg_upgrade to use TAP

---
 src/bin/pg_upgrade/Makefile            |  17 +-
 src/bin/pg_upgrade/TESTING             |   9 +-
 src/bin/pg_upgrade/t/001_basic.pl      |   9 +
 src/bin/pg_upgrade/t/002_pg_upgrade.pl | 188 +++++++++++++++++
 src/bin/pg_upgrade/test.sh             | 272 -------------------------
 src/test/perl/PostgresNode.pm          |  25 +++
 doc/src/sgml/install-windows.sgml      |   1 -
 src/tools/msvc/vcregress.pl            | 112 +---------
 8 files changed, 230 insertions(+), 403 deletions(-)
 create mode 100644 src/bin/pg_upgrade/t/001_basic.pl
 create mode 100644 src/bin/pg_upgrade/t/002_pg_upgrade.pl
 delete mode 100644 src/bin/pg_upgrade/test.sh

diff --git a/src/bin/pg_upgrade/Makefile b/src/bin/pg_upgrade/Makefile
index 44d06be5a6..fa8dee0a9c 100644
--- a/src/bin/pg_upgrade/Makefile
+++ b/src/bin/pg_upgrade/Makefile
@@ -49,17 +49,8 @@ clean distclean maintainer-clean:
 	       pg_upgrade_dump_globals.sql \
 	       pg_upgrade_dump_*.custom pg_upgrade_*.log
 
-# When $(MAKE) is present, make automatically infers that this is a
-# recursive make. which is not actually what we want here, as that
-# e.g. prevents output synchronization from working (as make thinks
-# that the subsidiary make knows how to deal with that itself, but
-# we're invoking a shell script that doesn't know). Referencing
-# $(MAKE) indirectly avoids that behaviour.
-# See https://www.gnu.org/software/make/manual/html_node/MAKE-Variable.html#MAKE-Variable
-NOTSUBMAKEMAKE=$(MAKE)
+check:
+	$(prove_check)
 
-check: test.sh all temp-install
-	MAKE=$(NOTSUBMAKEMAKE) $(with_temp_install) bindir=$(abs_top_builddir)/tmp_install/$(bindir) EXTRA_REGRESS_OPTS="$(EXTRA_REGRESS_OPTS)" $(SHELL) $<
-
-# installcheck is not supported because there's no meaningful way to test
-# pg_upgrade against a single already-running server
+installcheck:
+	$(prove_installcheck)
diff --git a/src/bin/pg_upgrade/TESTING b/src/bin/pg_upgrade/TESTING
index e69874b42d..b589bcaf6d 100644
--- a/src/bin/pg_upgrade/TESTING
+++ b/src/bin/pg_upgrade/TESTING
@@ -9,14 +9,11 @@ an upgrade from the version in this source tree to a new instance of
 the same version.
 
 To test an upgrade from a different version, you must have a built
-source tree for the old version as well as this version, and you
-must have done "make install" for both versions.  Then do:
+source tree for the old version as well as this version.  Then do:
 
 export oldsrc=...somewhere/postgresql	(old version's source tree)
-export oldbindir=...otherversion/bin	(old version's installed bin dir)
-export bindir=...thisversion/bin	(this version's installed bin dir)
-export libdir=...thisversion/lib	(this version's installed lib dir)
-sh test.sh
+export oldinstall=...otherversion/bin	(old version's install base path)
+make check
 
 In this case, you will have to manually eyeball the resulting dump
 diff for version-specific differences, as explained below.
diff --git a/src/bin/pg_upgrade/t/001_basic.pl b/src/bin/pg_upgrade/t/001_basic.pl
new file mode 100644
index 0000000000..605a7f622f
--- /dev/null
+++ b/src/bin/pg_upgrade/t/001_basic.pl
@@ -0,0 +1,9 @@
+use strict;
+use warnings;
+
+use TestLib;
+use Test::More tests => 8;
+
+program_help_ok('pg_upgrade');
+program_version_ok('pg_upgrade');
+program_options_handling_ok('pg_upgrade');
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
new file mode 100644
index 0000000000..263aa9525b
--- /dev/null
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -0,0 +1,188 @@
+# Set of tests for pg_upgrade.
+use strict;
+use warnings;
+
+use Cwd qw(abs_path getcwd);
+use File::Basename qw(dirname);
+
+use PostgresNode;
+use TestLib;
+use Test::More tests => 4;
+
+# Generate a database with a name made of a range of ASCII characters.
+sub generate_db
+{
+	my ($node, $from_char, $to_char) = @_;
+
+	my $dbname = '';
+	for my $i ($from_char .. $to_char)
+	{
+		next if $i == 7 || $i == 10 || $i == 13;    # skip BEL, LF, and CR
+		$dbname = $dbname . sprintf('%c', $i);
+	}
+	$node->run_log([ 'createdb', '--port', $node->port, $dbname ]);
+}
+
+my $startdir = getcwd();
+
+# From now on, the test of pg_upgrade consists in setting up an instance
+# on which regression tests are run.  This is the source instance used
+# for the upgrade. Then  a new, fresh, instance is created, and is used
+# as the target instance for the upgrade.  Before running an upgrade a
+# logical dump of the old instance is taken, and a second logical dump
+# of the new instance is taken after the upgrade.  The upgrade test
+# passes if there are no differences after running pg_upgrade.
+
+# Testing upgrades with an older instance of PostgreSQL requires
+# setting up two environment variables:
+# - "oldsrc", to point to the code source of the older version.
+# - "oldinstall", to point to the installation path of the older
+# version.
+
+# Default is the location of this source code for both nodes used with
+# the upgrade.
+my $newsrc = abs_path("../../..");
+my $oldsrc = $ENV{oldsrc} || $newsrc;
+$oldsrc = abs_path($oldsrc);
+
+if (   (defined($ENV{oldsrc}) && !defined($ENV{oldinstall}))
+	|| (!defined($ENV{oldsrc}) && defined($ENV{oldinstall})))
+{
+	# Not all variables are defined, so leave and die if test is
+	# done with an older installation.
+	die "oldsrc or oldinstall is undefined";
+}
+
+if (defined($ENV{oldsrc}) && $windows_os)
+{
+	# This configuration is not supported on Windows, as regress.so
+	# location diverges across the compilation methods used on this
+	# platform.
+	# XXX: this could likely be lifted.
+	die "No support for older version tests on Windows";
+}
+
+# Temporary location for the dumps taken
+my $tempdir = TestLib::tempdir;
+
+# Initialize node to upgrade
+my $oldnode = get_new_node('old_node', install_path => $ENV{oldinstall});
+
+$oldnode->init(extra => [ '--locale', 'C', '--encoding', 'LATIN1' ]);
+$oldnode->start;
+
+# Creating databases with names covering most ASCII bytes
+generate_db($oldnode, 1,  45);
+generate_db($oldnode, 46, 90);
+generate_db($oldnode, 91, 127);
+
+# Run core regression tests on the old instance.
+$oldnode->run_log([ "createdb", '--port', $oldnode->port, 'regression' ]);
+
+# This is more a trick than anything else, as pg_regress needs to be
+# from the old instance.  --dlpath is needed to be able to find the
+# location of regress.so, and is located normally in the same folder
+# as pg_regress itself.
+my $bindir = $oldnode->bin_dir;
+chdir "$oldsrc/src/test/regress/";
+$oldnode->command_ok(
+	[
+		$ENV{PG_REGRESS},    '--schedule',
+		'parallel_schedule', '--bindir',
+		$oldnode->bin_dir,   '--dlpath',
+		'.',                 '--use-existing',
+		'--port',            $oldnode->port
+	],
+	'regression test run on old instance');
+
+# Before dumping, get rid of objects not existing in later versions. This
+# depends on the version of the old server used, and matters only if the
+# old and new source paths
+my ($result, $oldpgversion, $stderr) =
+  $oldnode->psql('postgres', qq[SHOW server_version_num;]);
+my $fix_sql;
+if ($newsrc ne $oldsrc)
+{
+	if ($oldpgversion <= 80400)
+	{
+		$fix_sql =
+		  "DROP FUNCTION public.myfunc(integer); DROP FUNCTION public.oldstyle_length(integer, text);";
+	}
+	else
+	{
+		$fix_sql = "DROP FUNCTION public.oldstyle_length(integer, text);";
+	}
+	$oldnode->psql('postgres', $fix_sql);
+}
+
+# Initialize a new node for the upgrade.  This is done early so as it is
+# possible to know with which node's PATH the initial dump needs to be
+# taken.
+my $newnode = get_new_node('new_node');
+$newnode->init(extra => [ '--locale=C', '--encoding=LATIN1' ]);
+
+# Take a dump before performing the upgrade as a base comparison. Note
+# that we need to use pg_dumpall from the new node here.
+$newnode->command_ok(
+	[
+		'pg_dumpall', '--no-sync',
+		'-d',         $oldnode->connstr('postgres'),
+		'-f',         "$tempdir/dump1.sql"
+	],
+	'dump before running pg_upgrade');
+
+# After dumping, update references to the old source tree's regress.so and
+# such.
+if ($newsrc ne $oldsrc)
+{
+	if ($oldpgversion <= 80400)
+	{
+		$fix_sql =
+		  "UPDATE pg_proc SET probin = replace(probin::text, '$oldsrc', '$newsrc')::bytea WHERE probin LIKE '$oldsrc%';";
+	}
+	else
+	{
+		$fix_sql =
+		  "UPDATE pg_proc SET probin = replace(probin, '$oldsrc', '$newsrc') WHERE probin LIKE '$oldsrc%';";
+	}
+	$oldnode->psql('postgres', $fix_sql);
+
+	my $dump_data = slurp_file("$tempdir/dump1.sql");
+	$dump_data =~ s/$oldsrc/$newsrc/g;
+
+	open my $fh, ">", "$tempdir/dump1.sql" or die "could not open dump file";
+	print $fh $dump_data;
+	close $fh;
+}
+
+# Move back to current directory, all logs generated need to be located
+# at the origin.
+chdir $startdir;
+
+# Update the instance.
+$oldnode->stop;
+
+# Time for the real run.
+chdir "$newsrc/src/test/regress";
+my $newbindir = $newnode->bin_dir;
+$newnode->command_ok(
+	[
+		'pg_upgrade',       '-d', $oldnode->data_dir, '-D',
+		$newnode->data_dir, '-b', $oldnode->bin_dir,  '-B',
+		$newnode->bin_dir,  '-p', $oldnode->port,     '-P',
+		$newnode->port
+	],
+	'run of pg_upgrade for new instance');
+$newnode->start;
+
+# Take a second dump on the upgraded instance.
+$newnode->run_log(
+	[
+		'pg_dumpall', '--no-sync',
+		'-d',         $newnode->connstr('postgres'),
+		'-f',         "$tempdir/dump2.sql"
+	]);
+
+# Compare the two dumps, there should be no differences.
+command_ok([ 'diff', '-q', "$tempdir/dump1.sql", "$tempdir/dump2.sql" ],
+	'Old and new dump match after pg_upgrade');
diff --git a/src/bin/pg_upgrade/test.sh b/src/bin/pg_upgrade/test.sh
deleted file mode 100644
index 1ba326decd..0000000000
--- a/src/bin/pg_upgrade/test.sh
+++ /dev/null
@@ -1,272 +0,0 @@
-#!/bin/sh
-
-# src/bin/pg_upgrade/test.sh
-#
-# Test driver for pg_upgrade.  Initializes a new database cluster,
-# runs the regression tests (to put in some data), runs pg_dumpall,
-# runs pg_upgrade, runs pg_dumpall again, compares the dumps.
-#
-# Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
-# Portions Copyright (c) 1994, Regents of the University of California
-
-set -e
-
-: ${MAKE=make}
-
-# Guard against parallel make issues (see comments in pg_regress.c)
-unset MAKEFLAGS
-unset MAKELEVEL
-
-# Run a given "initdb" binary and overlay the regression testing
-# authentication configuration.
-standard_initdb() {
-	# To increase coverage of non-standard segment size and group access
-	# without increasing test runtime, run these tests with a custom setting.
-	# Also, specify "-A trust" explicitly to suppress initdb's warning.
-	"$1" -N --wal-segsize 1 -g -A trust
-	if [ -n "$TEMP_CONFIG" -a -r "$TEMP_CONFIG" ]
-	then
-		cat "$TEMP_CONFIG" >> "$PGDATA/postgresql.conf"
-	fi
-	../../test/regress/pg_regress --config-auth "$PGDATA"
-}
-
-# What flavor of host are we on?
-# Treat MINGW* (msys1) and MSYS* (msys2) the same.
-testhost=`uname -s | sed 's/^MSYS/MINGW/'`
-
-# Establish how the server will listen for connections
-case $testhost in
-	MINGW*)
-		LISTEN_ADDRESSES="localhost"
-		PG_REGRESS_SOCKET_DIR=""
-		PGHOST=localhost
-		;;
-	*)
-		LISTEN_ADDRESSES=""
-		# Select a socket directory.  The algorithm is from the "configure"
-		# script; the outcome mimics pg_regress.c:make_temp_sockdir().
-		if [ x"$PG_REGRESS_SOCKET_DIR" = x ]; then
-			set +e
-			dir=`(umask 077 &&
-				  mktemp -d /tmp/pg_upgrade_check-XXXXXX) 2>/dev/null`
-			if [ ! -d "$dir" ]; then
-				dir=/tmp/pg_upgrade_check-$$-$RANDOM
-				(umask 077 && mkdir "$dir")
-				if [ ! -d "$dir" ]; then
-					echo "could not create socket temporary directory in \"/tmp\""
-					exit 1
-				fi
-			fi
-			set -e
-			PG_REGRESS_SOCKET_DIR=$dir
-			trap 'rm -rf "$PG_REGRESS_SOCKET_DIR"' 0
-			trap 'exit 3' 1 2 13 15
-		fi
-		PGHOST=$PG_REGRESS_SOCKET_DIR
-		;;
-esac
-
-POSTMASTER_OPTS="-F -c listen_addresses=\"$LISTEN_ADDRESSES\" -k \"$PG_REGRESS_SOCKET_DIR\""
-export PGHOST
-
-# don't rely on $PWD here, as old shells don't set it
-temp_root=`pwd`/tmp_check
-rm -rf "$temp_root"
-mkdir "$temp_root"
-
-: ${oldbindir=$bindir}
-
-: ${oldsrc=../../..}
-oldsrc=`cd "$oldsrc" && pwd`
-newsrc=`cd ../../.. && pwd`
-
-# We need to make pg_regress use psql from the desired installation
-# (likely a temporary one), because otherwise the installcheck run
-# below would try to use psql from the proper installation directory
-# of the target version, which might be outdated or not exist. But
-# don't override anything else that's already in EXTRA_REGRESS_OPTS.
-EXTRA_REGRESS_OPTS="$EXTRA_REGRESS_OPTS --bindir='$oldbindir'"
-export EXTRA_REGRESS_OPTS
-
-# While in normal cases this will already be set up, adding bindir to
-# path allows test.sh to be invoked with different versions as
-# described in ./TESTING
-PATH=$bindir:$PATH
-export PATH
-
-BASE_PGDATA="$temp_root/data"
-PGDATA="${BASE_PGDATA}.old"
-export PGDATA
-
-# Send installcheck outputs to a private directory.  This avoids conflict when
-# check-world runs pg_upgrade check concurrently with src/test/regress check.
-# To retrieve interesting files after a run, use pattern tmp_check/*/*.diffs.
-outputdir="$temp_root/regress"
-EXTRA_REGRESS_OPTS="$EXTRA_REGRESS_OPTS --outputdir=$outputdir"
-export EXTRA_REGRESS_OPTS
-mkdir "$outputdir"
-
-logdir=`pwd`/log
-rm -rf "$logdir"
-mkdir "$logdir"
-
-# Clear out any environment vars that might cause libpq to connect to
-# the wrong postmaster (cf pg_regress.c)
-#
-# Some shells, such as NetBSD's, return non-zero from unset if the variable
-# is already unset. Since we are operating under 'set -e', this causes the
-# script to fail. To guard against this, set them all to an empty string first.
-PGDATABASE="";        unset PGDATABASE
-PGUSER="";            unset PGUSER
-PGSERVICE="";         unset PGSERVICE
-PGSSLMODE="";         unset PGSSLMODE
-PGREQUIRESSL="";      unset PGREQUIRESSL
-PGCONNECT_TIMEOUT=""; unset PGCONNECT_TIMEOUT
-PGHOSTADDR="";        unset PGHOSTADDR
-
-# Select a non-conflicting port number, similarly to pg_regress.c
-PG_VERSION_NUM=`grep '#define PG_VERSION_NUM' "$newsrc"/src/include/pg_config.h | awk '{print $3}'`
-PGPORT=`expr $PG_VERSION_NUM % 16384 + 49152`
-export PGPORT
-
-i=0
-while psql -X postgres </dev/null 2>/dev/null
-do
-	i=`expr $i + 1`
-	if [ $i -eq 16 ]
-	then
-		echo port $PGPORT apparently in use
-		exit 1
-	fi
-	PGPORT=`expr $PGPORT + 1`
-	export PGPORT
-done
-
-# buildfarm may try to override port via EXTRA_REGRESS_OPTS ...
-EXTRA_REGRESS_OPTS="$EXTRA_REGRESS_OPTS --port=$PGPORT"
-export EXTRA_REGRESS_OPTS
-
-standard_initdb "$oldbindir"/initdb
-"$oldbindir"/pg_ctl start -l "$logdir/postmaster1.log" -o "$POSTMASTER_OPTS" -w
-
-# Create databases with names covering the ASCII bytes other than NUL, BEL,
-# LF, or CR.  BEL would ring the terminal bell in the course of this test, and
-# it is not otherwise a special case.  PostgreSQL doesn't support the rest.
-dbname1=`awk 'BEGIN { for (i= 1; i < 46; i++)
-	if (i != 7 && i != 10 && i != 13) printf "%c", i }' </dev/null`
-# Exercise backslashes adjacent to double quotes, a Windows special case.
-dbname1='\"\'$dbname1'\\"\\\'
-dbname2=`awk 'BEGIN { for (i = 46; i <  91; i++) printf "%c", i }' </dev/null`
-dbname3=`awk 'BEGIN { for (i = 91; i < 128; i++) printf "%c", i }' </dev/null`
-createdb "regression$dbname1" || createdb_status=$?
-createdb "regression$dbname2" || createdb_status=$?
-createdb "regression$dbname3" || createdb_status=$?
-
-if "$MAKE" -C "$oldsrc" installcheck-parallel; then
-	oldpgversion=`psql -X -A -t -d regression -c "SHOW server_version_num"`
-
-	# before dumping, get rid of objects not feasible in later versions
-	if [ "$newsrc" != "$oldsrc" ]; then
-		fix_sql=""
-		case $oldpgversion in
-			804??)
-				fix_sql="DROP FUNCTION public.myfunc(integer);"
-				;;
-		esac
-		fix_sql="$fix_sql
-				 DROP FUNCTION IF EXISTS
-					public.oldstyle_length(integer, text);	-- last in 9.6
-				 DROP FUNCTION IF EXISTS
-					public.putenv(text);	-- last in v13
-				 DROP OPERATOR IF EXISTS	-- last in v13
-					public.#@# (pg_catalog.int8, NONE),
-					public.#%# (pg_catalog.int8, NONE),
-					public.!=- (pg_catalog.int8, NONE),
-					public.#@%# (pg_catalog.int8, NONE);"
-		psql -X -d regression -c "$fix_sql;" || psql_fix_sql_status=$?
-	fi
-
-	pg_dumpall --no-sync -f "$temp_root"/dump1.sql || pg_dumpall1_status=$?
-
-	if [ "$newsrc" != "$oldsrc" ]; then
-		# update references to old source tree's regress.so etc
-		fix_sql=""
-		case $oldpgversion in
-			804??)
-				fix_sql="UPDATE pg_proc SET probin = replace(probin::text, '$oldsrc', '$newsrc')::bytea WHERE probin LIKE '$oldsrc%';"
-				;;
-			*)
-				fix_sql="UPDATE pg_proc SET probin = replace(probin, '$oldsrc', '$newsrc') WHERE probin LIKE '$oldsrc%';"
-				;;
-		esac
-		psql -X -d regression -c "$fix_sql;" || psql_fix_sql_status=$?
-
-		mv "$temp_root"/dump1.sql "$temp_root"/dump1.sql.orig
-		sed "s;$oldsrc;$newsrc;g" "$temp_root"/dump1.sql.orig >"$temp_root"/dump1.sql
-	fi
-else
-	make_installcheck_status=$?
-fi
-"$oldbindir"/pg_ctl -m fast stop
-if [ -n "$createdb_status" ]; then
-	exit 1
-fi
-if [ -n "$make_installcheck_status" ]; then
-	exit 1
-fi
-if [ -n "$psql_fix_sql_status" ]; then
-	exit 1
-fi
-if [ -n "$pg_dumpall1_status" ]; then
-	echo "pg_dumpall of pre-upgrade database cluster failed"
-	exit 1
-fi
-
-PGDATA="$BASE_PGDATA"
-
-standard_initdb 'initdb'
-
-pg_upgrade $PG_UPGRADE_OPTS -d "${PGDATA}.old" -D "$PGDATA" -b "$oldbindir" -p "$PGPORT" -P "$PGPORT"
-
-# make sure all directories and files have group permissions, on Unix hosts
-# Windows hosts don't support Unix-y permissions.
-case $testhost in
-	MINGW*|CYGWIN*) ;;
-	*)	if [ `find "$PGDATA" -type f ! -perm 640 | wc -l` -ne 0 ]; then
-			echo "files in PGDATA with permission != 640";
-			exit 1;
-		fi ;;
-esac
-
-case $testhost in
-	MINGW*|CYGWIN*) ;;
-	*)	if [ `find "$PGDATA" -type d ! -perm 750 | wc -l` -ne 0 ]; then
-			echo "directories in PGDATA with permission != 750";
-			exit 1;
-		fi ;;
-esac
-
-pg_ctl start -l "$logdir/postmaster2.log" -o "$POSTMASTER_OPTS" -w
-
-pg_dumpall --no-sync -f "$temp_root"/dump2.sql || pg_dumpall2_status=$?
-pg_ctl -m fast stop
-
-if [ -n "$pg_dumpall2_status" ]; then
-	echo "pg_dumpall of post-upgrade database cluster failed"
-	exit 1
-fi
-
-case $testhost in
-	MINGW*)	MSYS2_ARG_CONV_EXCL=/c cmd /c delete_old_cluster.bat ;;
-	*)	    sh ./delete_old_cluster.sh ;;
-esac
-
-if diff "$temp_root"/dump1.sql "$temp_root"/dump2.sql >/dev/null; then
-	echo PASSED
-	exit 0
-else
-	echo "Files $temp_root/dump1.sql and $temp_root/dump2.sql differ"
-	echo "dumps were not identical"
-	exit 1
-fi
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index f7088667a4..45147b9c20 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -340,6 +340,31 @@ sub backup_dir
 
 =pod
 
+=item $node->bin_dir()
+
+The path to the binaries of the node.
+
+=cut
+
+sub bin_dir
+{
+	my ($self) = @_;
+	local %ENV = $self->_get_env();
+
+	# Find the information from pg_config --bindir
+	my ($stdout, $stderr);
+	my $result =
+	  IPC::Run::run [ $self->installed_command('pg_config'), '--bindir' ],
+	  '>', \$stdout, '2>', \$stderr
+	  or die "could not execute pg_config";
+	chomp($stdout);
+	$stdout =~ s/\r$//;
+
+	return $stdout;
+}
+
+=pod
+
 =item $node->info()
 
 Return a string containing human-readable diagnostic information (paths, etc)
diff --git a/doc/src/sgml/install-windows.sgml b/doc/src/sgml/install-windows.sgml
index 92087dba68..55e271897c 100644
--- a/doc/src/sgml/install-windows.sgml
+++ b/doc/src/sgml/install-windows.sgml
@@ -469,7 +469,6 @@ $ENV{CONFIG}="Debug";
 <userinput>vcregress isolationcheck</userinput>
 <userinput>vcregress bincheck</userinput>
 <userinput>vcregress recoverycheck</userinput>
-<userinput>vcregress upgradecheck</userinput>
 </screen>
 
    To change the schedule used (default is parallel), append it to the
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index 1852c34109..3a7e663e07 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -41,7 +41,7 @@ if (-e "src/tools/msvc/buildenv.pl")
 
 my $what = shift || "";
 if ($what =~
-	/^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck|bincheck|recoverycheck|taptest)$/i
+	/^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|bincheck|recoverycheck|taptest)$/i
   )
 {
 	$what = uc $what;
@@ -90,7 +90,6 @@ my %command = (
 	ISOLATIONCHECK => \&isolationcheck,
 	BINCHECK       => \&bincheck,
 	RECOVERYCHECK  => \&recoverycheck,
-	UPGRADECHECK   => \&upgradecheck,
 	TAPTEST        => \&taptest,);
 
 my $proc = $command{$what};
@@ -549,114 +548,6 @@ sub quote_system_arg
 	return "\"$arg\"";
 }
 
-# Generate a database with a name made of a range of ASCII characters, useful
-# for testing pg_upgrade.
-sub generate_db
-{
-	my ($prefix, $from_char, $to_char, $suffix) = @_;
-
-	my $dbname = $prefix;
-	for my $i ($from_char .. $to_char)
-	{
-		next if $i == 7 || $i == 10 || $i == 13;    # skip BEL, LF, and CR
-		$dbname = $dbname . sprintf('%c', $i);
-	}
-	$dbname .= $suffix;
-
-	system('createdb', quote_system_arg($dbname));
-	my $status = $? >> 8;
-	exit $status if $status;
-	return;
-}
-
-sub upgradecheck
-{
-	my $status;
-	my $cwd = getcwd();
-
-	# Much of this comes from the pg_upgrade test.sh script,
-	# but it only covers the --install case, and not the case
-	# where the old and new source or bin dirs are different.
-	# i.e. only this version to this version check. That's
-	# what pg_upgrade's "make check" does.
-
-	$ENV{PGHOST} = 'localhost';
-	$ENV{PGPORT} ||= 50432;
-	my $tmp_root = "$topdir/src/bin/pg_upgrade/tmp_check";
-	rmtree($tmp_root);
-	mkdir $tmp_root || die $!;
-	my $upg_tmp_install = "$tmp_root/install";    # unshared temp install
-	print "Setting up temp install\n\n";
-	Install($upg_tmp_install, "all", $config);
-
-	# Install does a chdir, so change back after that
-	chdir $cwd;
-	my ($bindir, $libdir, $oldsrc, $newsrc) =
-	  ("$upg_tmp_install/bin", "$upg_tmp_install/lib", $topdir, $topdir);
-	$ENV{PATH} = "$bindir;$ENV{PATH}";
-	my $data = "$tmp_root/data";
-	$ENV{PGDATA} = "$data.old";
-	my $outputdir          = "$tmp_root/regress";
-	my @EXTRA_REGRESS_OPTS = ("--outputdir=$outputdir");
-	mkdir "$outputdir" || die $!;
-
-	my $logdir = "$topdir/src/bin/pg_upgrade/log";
-	rmtree($logdir);
-	mkdir $logdir || die $!;
-	print "\nRunning initdb on old cluster\n\n";
-	standard_initdb() or exit 1;
-	print "\nStarting old cluster\n\n";
-	my @args = ('pg_ctl', 'start', '-l', "$logdir/postmaster1.log");
-	system(@args) == 0 or exit 1;
-
-	print "\nCreating databases with names covering most ASCII bytes\n\n";
-	generate_db("\\\"\\", 1,  45,  "\\\\\"\\\\\\");
-	generate_db('',       46, 90,  '');
-	generate_db('',       91, 127, '');
-
-	print "\nSetting up data for upgrading\n\n";
-	installcheck_internal('parallel', @EXTRA_REGRESS_OPTS);
-
-	# now we can chdir into the source dir
-	chdir "$topdir/src/bin/pg_upgrade";
-	print "\nDumping old cluster\n\n";
-	@args = ('pg_dumpall', '-f', "$tmp_root/dump1.sql");
-	system(@args) == 0 or exit 1;
-	print "\nStopping old cluster\n\n";
-	system("pg_ctl stop") == 0 or exit 1;
-	$ENV{PGDATA} = "$data";
-	print "\nSetting up new cluster\n\n";
-	standard_initdb() or exit 1;
-	print "\nRunning pg_upgrade\n\n";
-	@args = ('pg_upgrade', '-d', "$data.old", '-D', $data, '-b', $bindir);
-	system(@args) == 0 or exit 1;
-	print "\nStarting new cluster\n\n";
-	@args = ('pg_ctl', '-l', "$logdir/postmaster2.log", 'start');
-	system(@args) == 0 or exit 1;
-	print "\nDumping new cluster\n\n";
-	@args = ('pg_dumpall', '-f', "$tmp_root/dump2.sql");
-	system(@args) == 0 or exit 1;
-	print "\nStopping new cluster\n\n";
-	system("pg_ctl stop") == 0 or exit 1;
-	print "\nDeleting old cluster\n\n";
-	system(".\\delete_old_cluster.bat") == 0 or exit 1;
-	print "\nComparing old and new cluster dumps\n\n";
-
-	@args = ('diff', '-q', "$tmp_root/dump1.sql", "$tmp_root/dump2.sql");
-	system(@args);
-	$status = $?;
-	if (!$status)
-	{
-		print "PASSED\n";
-	}
-	else
-	{
-		print "dumps not identical!\n";
-		exit(1);
-	}
-	return;
-}
-
 sub fetchRegressOpts
 {
 	my $handle;
@@ -777,7 +668,6 @@ sub usage
 	  "  plcheck        run tests of PL languages\n",
 	  "  recoverycheck  run recovery test suite\n",
 	  "  taptest        run an arbitrary TAP test set\n",
-	  "  upgradecheck   run tests of pg_upgrade\n",
 	  "\nOptions for <arg>: (used by check and installcheck)\n",
 	  "  serial         serial mode\n",
 	  "  parallel       parallel mode\n",
-- 
2.31.1

---

From: Andrew Dunstan <[email-masked]>
Date: Sat, 15 May 2021 14:22:24 -0400

> here, would people here still support this switch?

PostgresNode is currently able to create nodes suitable for upgrade down
to release 10. If we add '-w' to the 'pg_ctl start' flags that can
extend down to release 9.5. (Just tested) I think we should do that
forthwith. '-w' is now the default, but having it there explicitly does
no harm.

If people are interested in what's incompatible on older versions, they
can look at
<https://gitlab.com/adunstan/postgresnodeng/-/blob/master/PostgresNode.pm>
starting at about line 2764.

I don't think this will work, though, unless there is enough data to
exercise pg_upgrade fairly thoroughly. The approach taken by both
test.sh and (somewhat more comprehensively) by the buildfarm cross
version upgrade module is to test a cluster where the regression tests
have been run. That might be more difficult when testing against older
versions, so I have published a snapshot of the dumps of each of the
versions we tests against in the buildfarm animal crake. These could be
loaded into PostgresNode instances and then an upgrade attempted. See
<https://gitlab.com/adunstan/pg-old-bin/-/tree/master/data>. The data
goes back to 9.2. These compressed dumps are a couple of megabytes each,
not huge.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

---

From: Michael Paquier <[email-masked]>
Date: Mon, 17 May 2021 10:55:57 +0900

> forthwith. '-w' is now the default, but having it there explicitly does

Agreed.  When testing manually, I have personally never worked on any
patches that required binaries older than 9.4, so I would be fine if
the TAP tests are able to work easily down to 9.5, even if pg_upgrade
is supported down to 8.4.

> starting at about line 2764.

We should really have adjust_conf() at some point in the in-core
module.

> have been run.

Yeah, that's what my patch is doing with pg_regress, FWIW.  This
requires regress.so from the old version.

> That might be more difficult when testing against older

I agree that this can be simpler in some cases.  In your experience,
how much of an issue is it when it becomes necessary to keep around
binaries that rely on libraries older than what a system can support?
It is easy to counter issues in this area with OpenSSL and
non-necessary things, but we had in the past also cases where we had
code that conflicted with the kernel, aka 3e68686.

At the end of this exercise, what I think we should achieve is to:
1) Reduce the diff between the buildfarm code and the in-core code.
2) Get rid of test.sh.
3) Be able to run easily upgrade tests across major versions for
developers.

As of now, 3) requires some extra facilities depending on if this is
done by the buildfarm or the in-core tests:
1) Path to the source code of the old version.  This is required once
it becomes necessary to find out src/test/regress/ for the schedule,
the tests to run and its regress.so.  There is no need to do that if
you have a dump of the old instance.
2) Path to a custom dump to replace the run with pg_regress from 1).
3) Location of the old binaries, for pg_upgrade.  When it comes to
PostgresNode, we require an install path, so we cannot use directly
the location of the binaries.

Looking at the code of the buildfarm, its code does something smarter
than what my patch or HEAD's test.sh does now, as these require the
path for the old source.  The buildfarm code first scans for the
probin's used in the regression database and then updates any
references.  What test.sh and my patch do is using the path to the old
source code and run a single UPDATE.  The approach taken by the
buildfarm code is more portable, as a path to the old source code
becomes necessary only if running pg_regress manually.  So, what about
doing the following thing?
1) Update the TAP test so as probin entries are updated in the same way
as the buildfarm.
2) Allow one to specify a path to a custom dump, or a path to the old
source code for pg_regress.

If we do that, then it should be possible to reduce the code footprint
in the buildfarm code, while still allowing people to test major
upgrades in the same old-fashioned way, right?  That's assuming that
PostgresNode is made compatible down to 9.2, of course, as a first
step, as that's the range of the dumps you are keeping around for the
buildfarm.

Thoughts?
--
Michael

---

From: Andrew Dunstan <[email-masked]>
Date: Mon, 17 May 2021 12:32:13 -0400

> We should really have adjust_conf() at some point in the in-core

Yes, I'm going to be proposing a series of smallish patches including
these when the tree is branched (which I hope will be in a few weeks).

> code that conflicted with the kernel, aka 3e68686.

That one at least isn't an issue. Old versions of postgres didn't have
pg_rewind.

> step, as that's the range of the dumps you are keeping around for the

I'm intending to add some older dumps. -) But for now 9.2 is a good target.

cheers

andrew

---

From: Michael Paquier <[email-masked]>
Date: Tue, 18 May 2021 10:49:39 +0900

> these when the tree is branched (which I hope will be in a few weeks).

Thanks!  That clearly needs to happen first.  I'll help reviewing
these.

> I'm intending to add some older dumps. -) But for now 9.2 is a good target.

Makes sense.  For now, I'll update this patch set so as it is possible
to use custom dumps, as an option in parallel of pg_regress when
specifying a different source code path.  I'll also decouple the
business with probin updates and stick with the approach used by the
buildfarm code.
--
Michael

---

From: Michael Paquier <[email-masked]>
Date: Thu, 20 May 2021 15:07:56 +0900

> business with probin updates and stick with the approach used by the

This has proved to not be that difficult.  With the updated version
attached, pg_upgrade has two modes to set up the old instance used for
the upgrade with older binaries:
- With oldsrc and oldinstall set, pg_regress gets used, same way as
HEAD.
- With olddump and oldinstall set, an old dump is loaded instead in
the old instance before launching the upgrade.

oldsrc and olddump are exclusive options.  Similarly to HEAD, the
dumps taken from the old and new instances generate diffs that can be
inspected manually.  The updates of probin are done without any
dependencies to the source path of the old instance, copying from the
buildfarm.

While on it, I have fixed a couple of things that exist in test.sh but
were not reflected in this new script:
- Handling of postfix operations with ~13 clusters.
- Handling oldstyle_length for ~9.6 clusters.
- Handling of EXTRA_REGRESS_OPT.

This stuff still needs to be expanded depending on how PostgresNode is
made backward-compatible, but I'll wait for that to happen before
going further down here.  I have also spent some time testing all that
with MSVC, and the installation paths used for pg_regress&co make the
script a tad more confusing, so I have dropped this part for now.

Thanks,
--
Michael
From bfb0086cfc35c063acc671896a7ffcb2c23dc2c2 Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email-masked]>
Date: Thu, 20 May 2021 15:04:55 +0900
Subject: [PATCH v2] Switch tests of pg_upgrade to use TAP

---
 src/bin/pg_upgrade/Makefile            |  17 +-
 src/bin/pg_upgrade/TESTING             |  29 ++-
 src/bin/pg_upgrade/t/001_basic.pl      |   9 +
 src/bin/pg_upgrade/t/002_pg_upgrade.pl | 266 ++++++++++++++++++++++++
 src/bin/pg_upgrade/test.sh             | 272 -------------------------
 src/test/perl/PostgresNode.pm          |  25 +++
 6 files changed, 323 insertions(+), 295 deletions(-)
 create mode 100644 src/bin/pg_upgrade/t/001_basic.pl
 create mode 100644 src/bin/pg_upgrade/t/002_pg_upgrade.pl
 delete mode 100644 src/bin/pg_upgrade/test.sh

 
-check: test.sh all temp-install
-	MAKE=$(NOTSUBMAKEMAKE) $(with_temp_install) bindir=$(abs_top_builddir)/tmp_install/$(bindir) EXTRA_REGRESS_OPTS="$(EXTRA_REGRESS_OPTS)" $(SHELL) $<
-
-# installcheck is not supported because there's no meaningful way to test
-# pg_upgrade against a single already-running server
+installcheck:
+	$(prove_installcheck)
diff --git a/src/bin/pg_upgrade/TESTING b/src/bin/pg_upgrade/TESTING
index e69874b42d..185943dd4b 100644
--- a/src/bin/pg_upgrade/TESTING
+++ b/src/bin/pg_upgrade/TESTING
@@ -4,19 +4,26 @@ THE SHORT VERSION
 On non-Windows machines, you can execute the testing process
 described below by running
 	make check
-in this directory.  This will run the shell script test.sh, performing
-an upgrade from the version in this source tree to a new instance of
-the same version.
+in this directory.  This will run the TAP tests to run pg_upgrade,
+performing an upgrade from the version in this source tree to a
+new instance of the same version.
 
-To test an upgrade from a different version, you must have a built
-source tree for the old version as well as this version, and you
-must have done "make install" for both versions.  Then do:
+To test an upgrade from a different version, there are two options
+available:
+
+1) You have a built source tree for the old version as well as this
+version's binaries.  Then set up the following variables:
 
 export oldsrc=...somewhere/postgresql	(old version's source tree)
-export oldbindir=...otherversion/bin	(old version's installed bin dir)
-export bindir=...thisversion/bin	(this version's installed bin dir)
-export libdir=...thisversion/lib	(this version's installed lib dir)
-sh test.sh
+export oldinstall=...otherversion/bin	(old version's install base path)
+
+2) You have a dump that can be used to set up the old version, as well
+as this version's binaries.  Then set up the following variables:
+export olddump=...somewhere/dump.sql	(old version's dump)
+export oldinstall=...otherversion/bin	(old version's install base path)
+
+Finally, the tests can be done by running
+	make check
 
@@ -87,3 +94,5 @@ steps:
 
 7)  Diff the regression database dump file with the regression dump
     file loaded into the old server.
+
+The generated dump may be reusable with "olddump", as defined above.
diff --git a/src/bin/pg_upgrade/t/001_basic.pl b/src/bin/pg_upgrade/t/001_basic.pl
new file mode 100644
index 0000000000..605a7f622f
--- /dev/null
+++ b/src/bin/pg_upgrade/t/001_basic.pl
@@ -0,0 +1,9 @@
+use strict;
+use warnings;
+
+use TestLib;
+use Test::More tests => 8;
+
+program_help_ok('pg_upgrade');
+program_version_ok('pg_upgrade');
+program_options_handling_ok('pg_upgrade');
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
new file mode 100644
index 0000000000..4d813ad5c3
--- /dev/null
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -0,0 +1,266 @@
+# Set of tests for pg_upgrade.
+use strict;
+use warnings;
+
+use Cwd qw(abs_path getcwd);
+use File::Basename qw(dirname);
+
+use PostgresNode;
+use TestLib;
+use Test::More tests => 4;
+
+# Generate a database with a name made of a range of ASCII characters.
+sub generate_db
+{
+	my ($node, $from_char, $to_char) = @_;
+
+	my $dbname = '';
+	for my $i ($from_char .. $to_char)
+	{
+		next if $i == 7 || $i == 10 || $i == 13;    # skip BEL, LF, and CR
+		$dbname = $dbname . sprintf('%c', $i);
+	}
+	$node->run_log([ 'createdb', '--port', $node->port, $dbname ]);
+}
+
+my $startdir = getcwd();
+
+# From now on, the test of pg_upgrade consists in setting up an instance.
+# This is the source instance used for the upgrade. Then a new and fresh
+# instance is created, and is used as the target instance for the
+# upgrade.  Before running an upgrade a logical dump of the old instance
+# is taken, and a second logical dump of the new instance is taken after
+# the upgrade.  The upgrade test passes if there are no differences after
+# running pg_upgrade.
+
+# Testing upgrades with an older instance of PostgreSQL requires
+# setting up two environment variables, among the following:
+# - "oldsrc", to point to the code source of the older version.
+#   This is required to set up the old instance with pg_upgrade.
+# - "olddump", to point to a dump file that will be used to set
+#   up the old instance to upgrade from.
+# - "oldinstall", to point to the installation path of the older
+# version.
+
+# "oldsrc" and "olddump" cannot be used together.  Setting up
+# "olddump" and "oldinstall" will use the dump pointed to to
+# set up the old instance.  If "oldsrc" is used instead of "olddump",
+# the full set of regression tests of the old instance is run
+# instead.
+
+if (defined($ENV{oldsrc}) && defined($ENV{olddump}))
+{
+	die "oldsrc and olddump are both defined";
+}
+elsif (defined($ENV{oldsrc}))
+{
+	if (   (defined($ENV{oldsrc}) && !defined($ENV{oldinstall}))
+		|| (!defined($ENV{oldsrc}) && defined($ENV{oldinstall})))
+	{
+		# Not all variables are defined, so leave and die if test is
+		# done with an older installation.
+		die "oldsrc or oldinstall is undefined";
+	}
+}
+elsif (defined($ENV{olddump}))
+{
+	if (   (defined($ENV{olddump}) && !defined($ENV{oldinstall}))
+		|| (!defined($ENV{olddump}) && defined($ENV{oldinstall})))
+	{
+		# Not all variables are defined, so leave and die if test is
+		# done with an older installation.
+		die "olddump or oldinstall is undefined";
+	}
+}
+
+if ((defined($ENV{oldsrc}) || defined($ENV{olddump})) && $windows_os)
+{
+	# This configuration is not supported on Windows, as regress.so
+	# location diverges across the compilation methods used on this
+	# platform.
+	die "No support for older version tests on Windows";
+}
+
+# Default is the location of this source code for both nodes used with
+# the upgrade.
+my $newsrc = abs_path("../../..");
+my $oldsrc = $ENV{oldsrc} || $newsrc;
+$oldsrc = abs_path($oldsrc);
+
+# Temporary location for the dumps taken
+my $tempdir = TestLib::tempdir;
+
+# Initialize node to upgrade
+my $oldnode = get_new_node('old_node', install_path => $ENV{oldinstall});
+
+$oldnode->init(extra => [ '--locale', 'C', '--encoding', 'LATIN1' ]);
+$oldnode->start;
+
+# Set up the data of the old instance with pg_regress or an old dump.
+if (defined($ENV{olddump}))
+{
+	# Use the dump specified.
+	my $olddumpfile = $ENV{olddump};
+	die "no dump file found!" unless -e $olddumpfile;
+
+	# Load the dump, and we are done here.
+	$oldnode->command_ok(
+		[ 'psql', '-f', $olddumpfile, '--port', $oldnode->port, 'postgres' ]);
+}
+else
+{
+	# Default is to just use pg_regress to setup the old instance
+	# Creating databases with names covering most ASCII bytes
+	generate_db($oldnode, 1,  45);
+	generate_db($oldnode, 46, 90);
+	generate_db($oldnode, 91, 127);
+
+	# Run core regression tests on the old instance.
+	$oldnode->run_log([ "createdb", '--port', $oldnode->port, 'regression' ]);
+
+	# This is more a trick than anything else, as pg_regress needs to be
+	# from the old instance.  --dlpath is needed to be able to find the
+	# location of regress.so, and it is located in the same folder as
+	# pg_regress itself.
+
+	# Grab any regression options that may be passed down by caller.
+	my $extra_opts_val = $ENV{EXTRA_REGRESS_OPT} || "";
+	my @extra_opts     = split(/\s+/, $extra_opts_val);
+
+	chdir "$oldsrc/src/test/regress/";
+	my @regress_command = [
+		$ENV{PG_REGRESS},                  '--schedule',
+		'parallel_schedule',               '--bindir',
+		$oldnode->config_data('--bindir'), '--make-testtablespace-dir',
+		'--dlpath',                        '.',
+		'--use-existing',                  '--port',
+		$oldnode->port
+	];
+	@regress_command = (@regress_command, @extra_opts);
+
+	$oldnode->command_ok(@regress_command,
+		'regression test run on old instance');
+
+	# Move back to the start path.
+	chdir $startdir;
+}
+
+# Before dumping, get rid of objects not existing or not supported in later
+# versions. This depends on the version of the old server used, and matters
+# only if different versions are used for the dump.
+my ($result, $oldpgversion, $stderr) =
+  $oldnode->psql('postgres', qq[SHOW server_version_num;]);
+my $fix_sql;
+
+if (defined($ENV{oldinstall}))
+{
+	# Changes for PostgreSQL ~13
+	if ($oldpgversion < 140000)
+	{
+		# Postfix operators are not supported anymore in 14.
+		$oldnode->psql(
+			'regression', "
+			DROP OPERATOR IF EXISTS #@# (bigint,NONE);
+			DROP OPERATOR IF EXISTS #%# (bigint,NONE);
+			DROP OPERATOR IF EXISTS !=- (bigint,NONE);
+			DROP OPERATOR IF EXISTS #@%# (bigint,NONE);");
+		# Last appeared in 13.
+		$oldnode->psql('regression', "DROP FUNCTION public.putenv(text);");
+	}
+
+	# Changes for PostgreSQL ~9.6
+	if ($oldpgversion < 100000)
+	{
+		# Last appeared in 9.6.
+		$oldnode->psql('regression',
+			"DROP FUNCTION public.oldstyle_length(integer, text);");
+	}
+
+	# Add here tweaks to objects to adapt to newer versions.
+}
+
+# Initialize a new node for the upgrade.  This is done early so as it is
+# possible to know with which node's PATH the initial dump needs to be
+# taken.
+my $newnode = get_new_node('new_node');
+$newnode->init(extra => [ '--locale=C', '--encoding=LATIN1' ]);
+my $newbindir = $newnode->config_data('--bindir');
+my $oldbindir = $oldnode->config_data('--bindir');
+
+# Take a dump before performing the upgrade as a base comparison. Note
+# that we need to use pg_dumpall from the new node here.
+$newnode->command_ok(
+	[
+		'pg_dumpall', '--no-sync',
+		'-d',         $oldnode->connstr('postgres'),
+		'-f',         "$tempdir/dump1.sql"
+	],
+	'dump before running pg_upgrade');
+
+# After dumping, update references to the old source tree's regress.so
+# to point to the new tree.
+if (defined($ENV{oldinstall}))
+{
+	# First, fetch all the references to libraries that are not part
+	# of the default path $libdir.
+	my $output = $oldnode->safe_psql('regression',
+		"SELECT probin::text from pg_proc where probin not like '\$libdir%';"
+	);
+	chomp($output);
+	my @libpaths = split("\n", $output);
+
+	my $dump_data = slurp_file("$tempdir/dump1.sql");
+
+	my $newregresssrc = "$newsrc/src/test/regress";
+	foreach (@libpaths)
+	{
+		my $libpath = $_;
+		$libpath = dirname($libpath);
+		$dump_data =~ s/$libpath/$newregresssrc/g;
+	}
+
+	open my $fh, ">", "$tempdir/dump1.sql" or die "could not open dump file";
+	print $fh $dump_data;
+	close $fh;
+
+	# This replaces any references to the old tree's regress.so
+	# the new tree's regress.so.  Any references that do *not*
+	# match $libdir are switched so as this request does not
+	# depend on the path of the old source tree.  This is useful
+	# when using an old dump.
+	$oldnode->safe_psql(
+		'regression', "UPDATE pg_proc SET probin =
+	  regexp_replace(probin, '.*/', '$newregresssrc/')
+	  WHERE probin NOT LIKE '\$libdir/%'");
+}
+
+# Move back to current directory, all logs generated need to be located
+# at the origin.
+chdir $startdir;
+
+# Update the instance.
+$oldnode->stop;
+
+# Time for the real run.
+chdir "$newsrc/src/test/regress";
+$newnode->command_ok(
+	[
+		'pg_upgrade',       '-d', $oldnode->data_dir, '-D',
+		$newnode->data_dir, '-b', $oldbindir,         '-B',
+		$newbindir,         '-p', $oldnode->port,     '-P',
+		$newnode->port
+	],
+	'run of pg_upgrade for new instance');
+$newnode->start;
+
+# Take a second dump on the upgraded instance.
+$newnode->run_log(
+	[
+		'pg_dumpall', '--no-sync',
+		'-d',         $newnode->connstr('postgres'),
+		'-f',         "$tempdir/dump2.sql"
+	]);
+
+# Compare the two dumps, there should be no differences.
+command_ok([ 'diff', '-q', "$tempdir/dump1.sql", "$tempdir/dump2.sql" ],
+	'Old and new dump match after pg_upgrade');
diff --git a/src/bin/pg_upgrade/test.sh b/src/bin/pg_upgrade/test.sh
deleted file mode 100644
index 1ba326decd..0000000000
--- a/src/bin/pg_upgrade/test.sh
+++ /dev/null
@@ -1,272 +0,0 @@
-#!/bin/sh
-
-# src/bin/pg_upgrade/test.sh
-#
-# Test driver for pg_upgrade.  Initializes a new database cluster,
-# runs the regression tests (to put in some data), runs pg_dumpall,
-# runs pg_upgrade, runs pg_dumpall again, compares the dumps.
-#
-# Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
-# Portions Copyright (c) 1994, Regents of the University of California
-
-set -e
-
-: ${MAKE=make}
-
-# Guard against parallel make issues (see comments in pg_regress.c)
-unset MAKEFLAGS
-unset MAKELEVEL
-
-# Run a given "initdb" binary and overlay the regression testing
-# authentication configuration.
-standard_initdb() {
-	# To increase coverage of non-standard segment size and group access
-	# without increasing test runtime, run these tests with a custom setting.
-	# Also, specify "-A trust" explicitly to suppress initdb's warning.
-	"$1" -N --wal-segsize 1 -g -A trust
-	if [ -n "$TEMP_CONFIG" -a -r "$TEMP_CONFIG" ]
-	then
-		cat "$TEMP_CONFIG" >> "$PGDATA/postgresql.conf"
-	fi
-	../../test/regress/pg_regress --config-auth "$PGDATA"
-}
-
-# What flavor of host are we on?
-# Treat MINGW* (msys1) and MSYS* (msys2) the same.
-testhost=`uname -s | sed 's/^MSYS/MINGW/'`
-
-# Establish how the server will listen for connections
-case $testhost in
-	MINGW*)
-		LISTEN_ADDRESSES="localhost"
-		PG_REGRESS_SOCKET_DIR=""
-		PGHOST=localhost
-		;;
-	*)
-		LISTEN_ADDRESSES=""
-		# Select a socket directory.  The algorithm is from the "configure"
-		# script; the outcome mimics pg_regress.c:make_temp_sockdir().
-		if [ x"$PG_REGRESS_SOCKET_DIR" = x ]; then
-			set +e
-			dir=`(umask 077 &&
-				  mktemp -d /tmp/pg_upgrade_check-XXXXXX) 2>/dev/null`
-			if [ ! -d "$dir" ]; then
-				dir=/tmp/pg_upgrade_check-$$-$RANDOM
-				(umask 077 && mkdir "$dir")
-				if [ ! -d "$dir" ]; then
-					echo "could not create socket temporary directory in \"/tmp\""
-					exit 1
-				fi
-			fi
-			set -e
-			PG_REGRESS_SOCKET_DIR=$dir
-			trap 'rm -rf "$PG_REGRESS_SOCKET_DIR"' 0
-			trap 'exit 3' 1 2 13 15
-		fi
-		PGHOST=$PG_REGRESS_SOCKET_DIR
-		;;
-esac
-
-POSTMASTER_OPTS="-F -c listen_addresses=\"$LISTEN_ADDRESSES\" -k \"$PG_REGRESS_SOCKET_DIR\""
-export PGHOST
-
-# don't rely on $PWD here, as old shells don't set it
-temp_root=`pwd`/tmp_check
-rm -rf "$temp_root"
-mkdir "$temp_root"
-
-: ${oldbindir=$bindir}
-
-: ${oldsrc=../../..}
-oldsrc=`cd "$oldsrc" && pwd`
-newsrc=`cd ../../.. && pwd`
-
-# We need to make pg_regress use psql from the desired installation
-# (likely a temporary one), because otherwise the installcheck run
-# below would try to use psql from the proper installation directory
-# of the target version, which might be outdated or not exist. But
-# don't override anything else that's already in EXTRA_REGRESS_OPTS.
-EXTRA_REGRESS_OPTS="$EXTRA_REGRESS_OPTS --bindir='$oldbindir'"
-export EXTRA_REGRESS_OPTS
-
-# While in normal cases this will already be set up, adding bindir to
-# path allows test.sh to be invoked with different versions as
-# described in ./TESTING
-PATH=$bindir:$PATH
-export PATH
-
-BASE_PGDATA="$temp_root/data"
-PGDATA="${BASE_PGDATA}.old"
-export PGDATA
-
-# Send installcheck outputs to a private directory.  This avoids conflict when
-# check-world runs pg_upgrade check concurrently with src/test/regress check.
-# To retrieve interesting files after a run, use pattern tmp_check/*/*.diffs.
-outputdir="$temp_root/regress"
-EXTRA_REGRESS_OPTS="$EXTRA_REGRESS_OPTS --outputdir=$outputdir"
-export EXTRA_REGRESS_OPTS
-mkdir "$outputdir"
-
-logdir=`pwd`/log
-rm -rf "$logdir"
-mkdir "$logdir"
-
-# Clear out any environment vars that might cause libpq to connect to
-# the wrong postmaster (cf pg_regress.c)
-#
-# Some shells, such as NetBSD's, return non-zero from unset if the variable
-# is already unset. Since we are operating under 'set -e', this causes the
-# script to fail. To guard against this, set them all to an empty string first.
-PGDATABASE="";        unset PGDATABASE
-PGUSER="";            unset PGUSER
-PGSERVICE="";         unset PGSERVICE
-PGSSLMODE="";         unset PGSSLMODE
-PGREQUIRESSL="";      unset PGREQUIRESSL
-PGCONNECT_TIMEOUT=""; unset PGCONNECT_TIMEOUT
-PGHOSTADDR="";        unset PGHOSTADDR
-
-# Select a non-conflicting port number, similarly to pg_regress.c
-PG_VERSION_NUM=`grep '#define PG_VERSION_NUM' "$newsrc"/src/include/pg_config.h | awk '{print $3}'`
-PGPORT=`expr $PG_VERSION_NUM % 16384 + 49152`
-export PGPORT
-
-i=0
-while psql -X postgres </dev/null 2>/dev/null
-do
-	i=`expr $i + 1`
-	if [ $i -eq 16 ]
-	then
-		echo port $PGPORT apparently in use
-		exit 1
-	fi
-	PGPORT=`expr $PGPORT + 1`
-	export PGPORT
-done
-
-# buildfarm may try to override port via EXTRA_REGRESS_OPTS ...
-EXTRA_REGRESS_OPTS="$EXTRA_REGRESS_OPTS --port=$PGPORT"
-export EXTRA_REGRESS_OPTS
-
-standard_initdb "$oldbindir"/initdb
-"$oldbindir"/pg_ctl start -l "$logdir/postmaster1.log" -o "$POSTMASTER_OPTS" -w
-
-# Create databases with names covering the ASCII bytes other than NUL, BEL,
-# LF, or CR.  BEL would ring the terminal bell in the course of this test, and
-# it is not otherwise a special case.  PostgreSQL doesn't support the rest.
-dbname1=`awk 'BEGIN { for (i= 1; i < 46; i++)
-	if (i != 7 && i != 10 && i != 13) printf "%c", i }' </dev/null`
-# Exercise backslashes adjacent to double quotes, a Windows special case.
-dbname1='\"\'$dbname1'\\"\\\'
-dbname2=`awk 'BEGIN { for (i = 46; i <  91; i++) printf "%c", i }' </dev/null`
-dbname3=`awk 'BEGIN { for (i = 91; i < 128; i++) printf "%c", i }' </dev/null`
-createdb "regression$dbname1" || createdb_status=$?
-createdb "regression$dbname2" || createdb_status=$?
-createdb "regression$dbname3" || createdb_status=$?
-
-if "$MAKE" -C "$oldsrc" installcheck-parallel; then
-	oldpgversion=`psql -X -A -t -d regression -c "SHOW server_version_num"`
-
-	# before dumping, get rid of objects not feasible in later versions
-	if [ "$newsrc" != "$oldsrc" ]; then
-		fix_sql=""
-		case $oldpgversion in
-			804??)
-				fix_sql="DROP FUNCTION public.myfunc(integer);"
-				;;
-		esac
-		fix_sql="$fix_sql
-				 DROP FUNCTION IF EXISTS
-					public.oldstyle_length(integer, text);	-- last in 9.6
-				 DROP FUNCTION IF EXISTS
-					public.putenv(text);	-- last in v13
-				 DROP OPERATOR IF EXISTS	-- last in v13
-					public.#@# (pg_catalog.int8, NONE),
-					public.#%# (pg_catalog.int8, NONE),
-					public.!=- (pg_catalog.int8, NONE),
-					public.#@%# (pg_catalog.int8, NONE);"
-		psql -X -d regression -c "$fix_sql;" || psql_fix_sql_status=$?
-	fi
-
-	pg_dumpall --no-sync -f "$temp_root"/dump1.sql || pg_dumpall1_status=$?
-
-	if [ "$newsrc" != "$oldsrc" ]; then
-		# update references to old source tree's regress.so etc
-		fix_sql=""
-		case $oldpgversion in
-			804??)
-				fix_sql="UPDATE pg_proc SET probin = replace(probin::text, '$oldsrc', '$newsrc')::bytea WHERE probin LIKE '$oldsrc%';"
-				;;
-			*)
-				fix_sql="UPDATE pg_proc SET probin = replace(probin, '$oldsrc', '$newsrc') WHERE probin LIKE '$oldsrc%';"
-				;;
-		esac
-		psql -X -d regression -c "$fix_sql;" || psql_fix_sql_status=$?
-
-		mv "$temp_root"/dump1.sql "$temp_root"/dump1.sql.orig
-		sed "s;$oldsrc;$newsrc;g" "$temp_root"/dump1.sql.orig >"$temp_root"/dump1.sql
-	fi
-else
-	make_installcheck_status=$?
-fi
-"$oldbindir"/pg_ctl -m fast stop
-if [ -n "$createdb_status" ]; then
-	exit 1
-fi
-if [ -n "$make_installcheck_status" ]; then
-	exit 1
-fi
-if [ -n "$psql_fix_sql_status" ]; then
-	exit 1
-fi
-if [ -n "$pg_dumpall1_status" ]; then
-	echo "pg_dumpall of pre-upgrade database cluster failed"
-	exit 1
-fi
-
-PGDATA="$BASE_PGDATA"
-
-standard_initdb 'initdb'
-
-pg_upgrade $PG_UPGRADE_OPTS -d "${PGDATA}.old" -D "$PGDATA" -b "$oldbindir" -p "$PGPORT" -P "$PGPORT"
-
-# make sure all directories and files have group permissions, on Unix hosts
-# Windows hosts don't support Unix-y permissions.
-case $testhost in
-	MINGW*|CYGWIN*) ;;
-	*)	if [ `find "$PGDATA" -type f ! -perm 640 | wc -l` -ne 0 ]; then
-			echo "files in PGDATA with permission != 640";
-			exit 1;
-		fi ;;
-esac
-
-case $testhost in
-	MINGW*|CYGWIN*) ;;
-	*)	if [ `find "$PGDATA" -type d ! -perm 750 | wc -l` -ne 0 ]; then
-			echo "directories in PGDATA with permission != 750";
-			exit 1;
-		fi ;;
-esac
-
-pg_ctl start -l "$logdir/postmaster2.log" -o "$POSTMASTER_OPTS" -w
-
-pg_dumpall --no-sync -f "$temp_root"/dump2.sql || pg_dumpall2_status=$?
-pg_ctl -m fast stop
-
-if [ -n "$pg_dumpall2_status" ]; then
-	echo "pg_dumpall of post-upgrade database cluster failed"
-	exit 1
-fi
-
-case $testhost in
-	MINGW*)	MSYS2_ARG_CONV_EXCL=/c cmd /c delete_old_cluster.bat ;;
-	*)	    sh ./delete_old_cluster.sh ;;
-esac
-
-if diff "$temp_root"/dump1.sql "$temp_root"/dump2.sql >/dev/null; then
-	echo PASSED
-	exit 0
-else
-	echo "Files $temp_root/dump1.sql and $temp_root/dump2.sql differ"
-	echo "dumps were not identical"
-	exit 1
-fi
diff --git a/src/test/perl/PostgresNode.pm b/src/test/perl/PostgresNode.pm
index f7088667a4..31ddbdfa4a 100644
--- a/src/test/perl/PostgresNode.pm
+++ b/src/test/perl/PostgresNode.pm
@@ -340,6 +340,31 @@ sub backup_dir
 
 =pod
 
+=item $node->config_data($option)
+
+Grab some data from pg_config, with $option being the switched
+used.
+
+=cut
+
+sub config_data
+{
+	my ($self, $option) = @_;
+	local %ENV = $self->_get_env();
+
+	my ($stdout, $stderr);
+	my $result =
+	  IPC::Run::run [ $self->installed_command('pg_config'), $option ],
+	  '>', \$stdout, '2>', \$stderr
+	  or die "could not execute pg_config";
+	chomp($stdout);
+	$stdout =~ s/\r$//;
+
+	return $stdout;
+}
+
+=pod
+
 =item $node->info()

---

From: Rachel Heaton <[email-masked]>
Date: Tue, 7 Sep 2021 14:43:15 -0700

Hello Michael,

This patch needs the update from 201a76183 -- the function `get_new_node`
no longer exists.

Running check tests in the pg_upgrade folder fails for this reason.

Thank you,
Rachel

---

From: Michael Paquier <[email-masked]>
Date: Wed, 8 Sep 2021 15:34:51 +0900

> Running check tests in the pg_upgrade folder fails for this reason.

Thanks, rebased as attached.  Andrew has posted another patch set that
completely reworks the shape of the modules by moving them into a
dedicated namespace, meaning that this is going to break again.  I'll
see about that when we reach this point.
--
Michael
From 6580f1b9bcf12a2b65e90f355ecf10250c121603 Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email-masked]>
Date: Wed, 8 Sep 2021 15:30:57 +0900
Subject: [PATCH v3] Switch tests of pg_upgrade to use TAP

---
 src/bin/pg_upgrade/Makefile            |  17 +-
 src/bin/pg_upgrade/TESTING             |  29 ++-
 src/bin/pg_upgrade/t/001_basic.pl      |   9 +
 src/bin/pg_upgrade/t/002_pg_upgrade.pl | 277 +++++++++++++++++++++++++
 src/bin/pg_upgrade/test.sh             | 272 ------------------------
 src/test/perl/PostgresNode.pm          |  25 +++
 6 files changed, 334 insertions(+), 295 deletions(-)
 create mode 100644 src/bin/pg_upgrade/t/001_basic.pl
 create mode 100644 src/bin/pg_upgrade/t/002_pg_upgrade.pl
 delete mode 100644 src/bin/pg_upgrade/test.sh

 

 
 
index 0000000000..0bdbe6dc26
@@ -0,0 +1,277 @@
+my $oldnode = PostgresNode->new('old_node', install_path => $ENV{oldinstall});
+my $newnode = PostgresNode->new('new_node');
+		"SELECT DISTINCT probin::text FROM pg_proc WHERE probin NOT LIKE '\$libdir%';"
+	# when using an old dump.  Do the operation on all the database
+	# that allow connections so as this includes the regression
+	# database and anything the user has set up.
+	$output = $oldnode->safe_psql('postgres',
+		"SELECT datname FROM pg_database WHERE datallowconn;"
+	my @datnames = split("\n", $output);
+	foreach (@datnames)
+		my $datname = $_;
+			$datname, "UPDATE pg_proc SET probin =
index c59da758c7..1033f5f614 100644
@@ -316,6 +316,31 @@ sub install_path
 
 =pod
 
+Grab some data from pg_config, with $option being the switch
 
2.33.0

---

From: Michael Paquier <[email-masked]>
Date: Fri, 1 Oct 2021 15:19:50 +0900

> script a tad more confusing, so I have dropped this part for now.

Andrew, as this is a bit tied to the buildfarm code and any
simplifications that could happen there, do you have any comments
and/or suggestions for this patch?

This still applies on HEAD and it holds all the properties of the
existing test by using PostgresNodes that point to older installations
for the business with binaries and libraries business.  There is one
part where pg_upgrade logs into src/test/regress/, which is not good,
but that should be easily fixable.
--
Michael

---

From: Andrew Dunstan <[email-masked]>
Date: Sat, 2 Oct 2021 16:58:47 -0400

> and/or suggestions for this patch?

I haven't looked at the patch closely yet, but from a buildfarm POV I
think the only thing that needs to be done is to inhibit the buildfarm
client module if the TAP tests are present. The buildfarm code that runs
TAP tests should automatically detect and run the new test.

I've just counted and there are 116 animals reporting check-pg_upgrade,
so we'd better put that out pronto. It's a little early but I'll try to
push out a release containing code for it on Monday or Tuesday (it's a
one line addition).

cheers

andrew

---

From: Tom Lane <[email-masked]>
Date: Sat, 02 Oct 2021 17:03:33 -0400

> push out a release containing code for it on Monday or Tuesday (it's a

IIUC, the only problem for a non-updated animal would be that it'd
run the test twice?  Or would it actually fail?  If the latter,
we'd need to sit on the patch rather longer.

			regards, tom lane

---

From: Andrew Dunstan <[email-masked]>
Date: Sat, 2 Oct 2021 23:32:36 -0400

> we'd need to sit on the patch rather longer.

The patch removes test.sh, so yes it would break.

cheers

andrew

---

From: Tom Lane <[email-masked]>
Date: Sat, 02 Oct 2021 23:34:38 -0400

> The patch removes test.sh, so yes it would break.

Maybe we could leave test.sh in place for awhile?  I'd rather
not cause a flag day for buildfarm owners.  (Also, how do we
see this working in the back branches?)

			regards, tom lane

---

From: Michael Paquier <[email-masked]>
Date: Sun, 3 Oct 2021 16:03:59 +0900

> see this working in the back branches?)

I would be fine with test.sh staying around for now.

If we do that, though, I think that we had better remove the support
for upgrades across different major versions in test.sh, and keep this
capability in the new script.  I am not sure that a lot of people use
that to begin with, but it would be weird to support that with a
different configuration layer for both at the same time (test.sh uses
a combination of bin/ and lib/ paths, while TAP uses just installation
path to accomodate with what PostgresNode.pm is able to do).  The
patch of this thread also adds support for the load of an old dump
instead of an installcheck run of the old instance, which is something
the buildfarm could use.

I also looked two days ago at a proposal to move all the
pg_upgrade-specific SQLs into a new, separate, file that makes use of
psql's \if to do the job encoded now in test.sh.  I think that it
would be strange to duplicate this logic in a the pg_upgrade TAP test
and test.sh if we finish by keeping both around for now.  So that's a
second item we had better deal with first, in my opinion:
https://www.postgresql.org/message-id/YVa/[email-masked]

Thoughts?
--
Michael

---

From: Andrew Dunstan <[email-masked]>
Date: Sun, 3 Oct 2021 08:22:57 -0400

> see this working in the back branches?)

Actually, I was wrong. The module just does "make check" for non-MSVC.
For MSVC it calls vcregress.pl, which the patch doesn't touch (it
should, I think).

cheers

andrew

---

From: Peter Eisentraut <[email-masked]>
Date: Sun, 10 Oct 2021 16:07:43 +0200

> I would be fine with test.sh staying around for now.

test.sh could be changed to invoke the TAP test.

---

From: Michael Paquier <[email-masked]>
Date: Mon, 11 Oct 2021 09:40:53 +0900

> test.sh could be changed to invoke the TAP test.

That would remove the possibility to run the tests of pg_upgrade with
--enable-tap-tests, which is the point I think Tom was making, because
TestUpgrade.pm in the buildfarm code just uses "make check" as of the
following:
$cmd = "cd $self->{pgsql}/src/bin/pg_upgrade && $make $instflags check";
--
Michael

---

From: Andrew Dunstan <[email-masked]>
Date: Mon, 11 Oct 2021 09:04:47 -0400

> test.sh could be changed to invoke the TAP test.

Keeping test.sh is not necessary - I mis-remembered what the test module
does.

cheers

andrew

---

From: Michael Paquier <[email-masked]>
Date: Tue, 12 Oct 2021 13:45:48 +0900

> For MSVC it calls vcregress.pl, which the patch doesn't touch (it

Yes, it should.  And I'd like to do things so as we replace all the
internals of upgradecheck() by a call to tap_check().  The patch does
not work yet properly with MSVC, and there were some problems in
getting the invocation of pg_regress right as far as I recall.  That's
why I have left this part for now.  I don't see why we could not do
the MSVC part as an independent step though, getting rid of test.sh is
appealing enough in itself.
--
Michael

---

From: Michael Paquier <[email-masked]>
Date: Tue, 12 Oct 2021 13:48:29 +0900

> Keeping test.sh is not necessary - I mis-remembered what the test module

So..  Are people fine to remove entirely test.sh at the end, requiring
the tests of pg_upgrade to have TAP installed?  I'd rather raise the
bar here, as it would keep the code simpler in the tree in the long
term.  Or am I misunderstanding something?
--
Michael

---

From: Andres Freund <[email-masked]>
Date: Mon, 13 Dec 2021 18:08:24 -0800

Hi,

> see this working in the back branches?)

Seems like we might get away with making make -C contrib/pg_upgrade check and
vcregress.pl upgradecheck do nothing?

For the common case of not testing cross-version stuff, pg_upgrade's tests
would just be invoked via run_build.pl:run_bin_tests(). And TestUpgrade.pm
should be fine with a test doing nothing.

We'd not loose coverage with non-updated BF animals unless they have tap tests
disabled. Just the cross-version test would need timely work by buildfarm
operators - but I think Andrew could deal with that.

Greetings,

Andres Freund

---

From: Michael Paquier <[email-masked]>
Date: Tue, 14 Dec 2021 14:31:24 +0900

> vcregress.pl upgradecheck do nothing?

You mean #contrib/#src/bin/# here, right?  I don't think that we have
any need to have "make -C" do nothing.  For vcregress.pl, we should
IMO just remove upgradecheck.

> should be fine with a test doing nothing.

Perhaps.  I am not sure what's the best picture here, TBH.  One
difference between the core stuff and the buldfarm is that in the case
of the buildfarm, we upgrade from a version that has not only the main
regression database, but everything from the contrib/ modules.

Speaking of which, I am going to send a patch for the buildfarm to be
able to use the SQL file from 0df9641, so as committers gain a bit
more control on the cross-version upgrade tests run by the buildfarm,
using the in-core code a maximum.
--
Michael

---

From: Andres Freund <[email-masked]>
Date: Mon, 13 Dec 2021 22:14:49 -0800

> IMO just remove upgradecheck.

Tom's point was that the buildfarm scripts do
	if ($self->{bfconf}->{using_msvc})
		@checklog = run_log("perl vcregress.pl upgradecheck");
	else
			  "cd $self->{pgsql}/src/bin/pg_upgrade && $make $instflags check";

if we don't want to break every buildfarm member that has TestUpgrade enabled
the moment this is committed, we need to have a backward compat path.

Greetings,

Andres Freund

---

From: Michael Paquier <[email-masked]>
Date: Wed, 15 Dec 2021 10:47:24 +0900

> the moment this is committed, we need to have a backward compat path.

Missed that, thanks!  I'll think about all that a bit more before
sending a long-overdue rebased version.
--
Michael

---

From: Michael Paquier <[email-masked]>
Date: Thu, 16 Dec 2021 11:51:55 +0900

> sending a long-overdue rebased version.

Okay, here is finally a rebase of this patch, where I have fixed a
couple of existing issues, and I have extended the patch to the point
where the support range is what I expect should be.  In short:
- Added support for MSVC for the TAP test.  I have considered making
upgradecheck silent, but after thinking about it I have just filtered
pg_upgrade from bincheck, and simplified upgradecheck to launch the
new test.  It is simple to switch from one approach to another.  This
shaves some code in vcregress.pl.
- Fixed a set of issues with various chdir commands in the previous
patches.  The command of pg_regress has been tweaked so as all results
are part of src/bin/pg_upgrade/.  Any logs generated by pg_upgrade
stay in this location, same way as HEAD.
- Adapted to the new modules of src/test/perl/.
- Support for cross-upgrades now uses upgrade_adapt.sql (I have sent a
patch for the buildfarm client about that yesterday actually), same
way as test.sh on HEAD.  Like HEAD, attempting to use the
cross-version HEAD causes diffs between the old and the new dumps.
But there is nothing new here.  This could be improved more but the
attached does already a lot. 
- Like the previous versions, this supports two modes when setting up
the to-be-upgraded cluster: setup things from an old dump or use
pg_regress.  The buildfarm does the former for upgrades down to 9.2.
The core code does the latter.

I may have missed one thing or two, but I think that's pretty much
what we should be looking for to do the switch to TAP in terms of
coverage.
--
Michael
From 47fcfac534f947595a880eacdbcfe4f8bd6af516 Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email-masked]>
Date: Thu, 16 Dec 2021 11:50:54 +0900
Subject: [PATCH v4] Switch tests of pg_upgrade to use TAP

---
 src/bin/pg_upgrade/.gitignore            |   5 +
 src/bin/pg_upgrade/Makefile              |  23 +-
 src/bin/pg_upgrade/TESTING               |  33 ++-
 src/bin/pg_upgrade/t/001_basic.pl        |   9 +
 src/bin/pg_upgrade/t/002_pg_upgrade.pl   | 275 ++++++++++++++++++++++
 src/bin/pg_upgrade/test.sh               | 279 -----------------------
 src/test/perl/PostgreSQL/Test/Cluster.pm |  25 ++
 src/tools/msvc/vcregress.pl              |  91 +-------
 8 files changed, 355 insertions(+), 385 deletions(-)
 create mode 100644 src/bin/pg_upgrade/t/001_basic.pl
 create mode 100644 src/bin/pg_upgrade/t/002_pg_upgrade.pl
 delete mode 100644 src/bin/pg_upgrade/test.sh

diff --git a/src/bin/pg_upgrade/.gitignore b/src/bin/pg_upgrade/.gitignore
index 2d3bfeaa50..3b64522ab6 100644
--- a/src/bin/pg_upgrade/.gitignore
+++ b/src/bin/pg_upgrade/.gitignore
@@ -7,3 +7,8 @@
 /loadable_libraries.txt
 /log/
 /tmp_check/
+
+# Generated by pg_regress
+/sql/
+/expected/
+/results/
diff --git a/src/bin/pg_upgrade/Makefile b/src/bin/pg_upgrade/Makefile
index 44d06be5a6..35b6c123a5 100644
--- a/src/bin/pg_upgrade/Makefile
+++ b/src/bin/pg_upgrade/Makefile
@@ -28,6 +28,12 @@ OBJS = \
 override CPPFLAGS := -DDLSUFFIX=\"$(DLSUFFIX)\" -I$(srcdir) -I$(libpq_srcdir) $(CPPFLAGS)
 LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils $(libpq_pgport)
 
+# required for 002_pg_upgrade.pl
+REGRESS_SHLIB=$(abs_top_builddir)/src/test/regress/regress$(DLSUFFIX)
+export REGRESS_SHLIB
+REGRESS_OUTPUTDIR=$(abs_top_builddir)/src/bin/pg_upgrade
+export REGRESS_OUTPUTDIR
+
 all: pg_upgrade
 
 pg_upgrade: $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
@@ -49,17 +55,8 @@ clean distclean maintainer-clean:
 	       pg_upgrade_dump_globals.sql \
 	       pg_upgrade_dump_*.custom pg_upgrade_*.log

-check: test.sh all temp-install
-	MAKE=$(NOTSUBMAKEMAKE) $(with_temp_install) bindir=$(abs_top_builddir)/tmp_install/$(bindir) EXTRA_REGRESS_OPTS="$(EXTRA_REGRESS_OPTS)" $(SHELL) $<
-
-# installcheck is not supported because there's no meaningful way to test
-# pg_upgrade against a single already-running server
+installcheck:
+	$(prove_installcheck)
diff --git a/src/bin/pg_upgrade/TESTING b/src/bin/pg_upgrade/TESTING
index e69874b42d..acf769386a 100644
--- a/src/bin/pg_upgrade/TESTING
+++ b/src/bin/pg_upgrade/TESTING
@@ -2,21 +2,30 @@ THE SHORT VERSION
 -----------------
 
 On non-Windows machines, you can execute the testing process
-described below by running
+described below by running the following command in this directory:
 	make check
-in this directory.  This will run the shell script test.sh, performing
-an upgrade from the version in this source tree to a new instance of
-the same version.
 
-To test an upgrade from a different version, you must have a built
-source tree for the old version as well as this version, and you
-must have done "make install" for both versions.  Then do:
+This will run the TAP tests to run pg_upgrade, performing an upgrade
+from the version in this source tree to a  new instance of the same
+version.
+
+To test an upgrade from a different version, there are two options
+available:
+
+1) You have a built source tree for the old version as well as this
+version's binaries.  Then set up the following variables before
+launching the test:
 
+export oldinstall=...otherversion/	(old version's install base path)
+export oldinstall=...otherversion/	(old version's install base path)
 
@@ -87,3 +96,5 @@ steps:
 
 7)  Diff the regression database dump file with the regression dump
     file loaded into the old server.
+
+The generated dump may be reusable with "olddump", as defined above.
diff --git a/src/bin/pg_upgrade/t/001_basic.pl b/src/bin/pg_upgrade/t/001_basic.pl
new file mode 100644
index 0000000000..75b0f98b08
--- /dev/null
+++ b/src/bin/pg_upgrade/t/001_basic.pl
@@ -0,0 +1,9 @@
+use strict;
+use warnings;
+
+use PostgreSQL::Test::Utils;
+use Test::More tests => 8;
+
+program_help_ok('pg_upgrade');
+program_version_ok('pg_upgrade');
+program_options_handling_ok('pg_upgrade');
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
new file mode 100644
index 0000000000..721741d0f8
--- /dev/null
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -0,0 +1,275 @@
+# Set of tests for pg_upgrade, including cross-version checks.
+use strict;
+use warnings;
+
+use Cwd qw(abs_path getcwd);
+use File::Basename qw(dirname);
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More tests => 4;
+
+# Generate a database with a name made of a range of ASCII characters.
+sub generate_db
+{
+	my ($node, $from_char, $to_char) = @_;
+
+	my $dbname = '';
+	for my $i ($from_char .. $to_char)
+	{
+		next if $i == 7 || $i == 10 || $i == 13;    # skip BEL, LF, and CR
+		$dbname = $dbname . sprintf('%c', $i);
+	}
+	$node->run_log([ 'createdb', '--port', $node->port, $dbname ]);
+}
+
+# From now on, the test of pg_upgrade consists in setting up an instance.
+# This is the source instance used for the upgrade. Then a new and fresh
+# instance is created, and is used as the target instance for the
+# upgrade.  Before running an upgrade a logical dump of the old instance
+# is taken, and a second logical dump of the new instance is taken after
+# the upgrade.  The upgrade test passes if there are no differences after
+# running pg_upgrade.
+
+# Testing upgrades with an older instance of PostgreSQL requires
+# setting up two environment variables, among the following:
+# - "oldsrc", to point to the code source of the older version.
+#   This is required to set up the old instance with pg_upgrade.
+# - "olddump", to point to a dump file that will be used to set
+#   up the old instance to upgrade from.
+# - "oldinstall", to point to the installation path of the older
+# version.
+
+# "oldsrc" and "olddump" cannot be used together.  Setting up
+# "olddump" and "oldinstall" will use the dump pointed to to
+# set up the old instance.  If "oldsrc" is used instead of "olddump",
+# the full set of regression tests of the old instance is run
+# instead.
+
+if (defined($ENV{oldsrc}) && defined($ENV{olddump}))
+{
+	die "oldsrc and olddump are both defined";
+}
+elsif (defined($ENV{oldsrc}))
+{
+	if (   (defined($ENV{oldsrc}) && !defined($ENV{oldinstall}))
+		|| (!defined($ENV{oldsrc}) && defined($ENV{oldinstall})))
+	{
+		# Not all variables are defined, so leave and die if test is
+		# done with an older installation.
+		die "oldsrc or oldinstall is undefined";
+	}
+}
+elsif (defined($ENV{olddump}))
+{
+	if (   (defined($ENV{olddump}) && !defined($ENV{oldinstall}))
+		|| (!defined($ENV{olddump}) && defined($ENV{oldinstall})))
+	{
+		# Not all variables are defined, so leave and die if test is
+		# done with an older installation.
+		die "olddump or oldinstall is undefined";
+	}
+}
+
+if ((defined($ENV{oldsrc}) || defined($ENV{olddump})) && $windows_os)
+{
+	# This configuration is not supported on Windows, as regress.so
+	# location diverges across the compilation methods used on this
+	# platform.
+	die "No support for older version tests on Windows";
+}
+
+# Default is the location of this source code for both nodes used with
+# the upgrade.
+my $newsrc = abs_path("../../..");
+my $oldsrc = $ENV{oldsrc} || $newsrc;
+$oldsrc = abs_path($oldsrc);
+
+# Temporary location for the dumps taken
+my $tempdir = PostgreSQL::Test::Utils::tempdir;
+
+# Initialize node to upgrade
+my $oldnode = PostgreSQL::Test::Cluster->new('old_node',
+	install_path => $ENV{oldinstall});
+
+$oldnode->init(extra => [ '--locale', 'C', '--encoding', 'LATIN1' ]);
+$oldnode->start;
+
+# Set up the data of the old instance with pg_regress or an old dump.
+if (defined($ENV{olddump}))
+{
+	# Use the dump specified.
+	my $olddumpfile = $ENV{olddump};
+	die "no dump file found!" unless -e $olddumpfile;
+
+	# Load the dump, and we are done here.
+	$oldnode->command_ok(
+		[
+			'psql',   '-X',           '-f', $olddumpfile,
+			'--port', $oldnode->port, 'regression'
+		]);
+}
+else
+{
+	# Default is to just use pg_regress to setup the old instance
+	# Creating databases with names covering most ASCII bytes
+	generate_db($oldnode, 1,  45);
+	generate_db($oldnode, 46, 90);
+	generate_db($oldnode, 91, 127);
+
+	# Run core regression tests on the old instance.
+	$oldnode->run_log([ "createdb", '--port', $oldnode->port, 'regression' ]);
+
+	# Grab any regression options that may be passed down by caller.
+	my $extra_opts_val = $ENV{EXTRA_REGRESS_OPT} || "";
+	my @extra_opts     = split(/\s+/, $extra_opts_val);
+
+	# --dlpath is needed to be able to find the location of regress.so and
+	# any libraries the regression tests required.  This needs to point to
+	# in the old instance when using it.  In the default case, fallback to
+	# what the caller provided for REGRESS_SHLIB.
+	my $dlpath;
+	if (defined($ENV{oldinstall}))
+	{
+		$dlpath = "$oldsrc/src/test/regress";
+	}
+	else
+	{
+		$dlpath = dirname($ENV{REGRESS_SHLIB});
+	}
+	$dlpath = PostgreSQL::Test::Utils::perl2host($dlpath);
+
+	# --outputdir points to the path where to place the output files, which
+	# had better be this directory.
+	my $outputdir =
+	  PostgreSQL::Test::Utils::perl2host($ENV{REGRESS_OUTPUTDIR});
+
+	# --inputdir needs to point to the location of the input files, from the
+	# cluster to-be-upgraded.
+	my $inputdir =
+	  PostgreSQL::Test::Utils::perl2host("$oldsrc/src/test/regress");
+
+	my @regress_command = [
+		$ENV{PG_REGRESS}, '--make-testtablespace-dir',
+		'--schedule',     "$oldsrc/src/test/regress/parallel_schedule",
+		'--bindir',       $oldnode->config_data('--bindir'),
+		'--dlpath',       $dlpath,
+		'--port',         $oldnode->port,
+		'--outputdir',    $outputdir,
+		'--inputdir',     $inputdir,
+		'--use-existing'
+	];
+	@regress_command = (@regress_command, @extra_opts);
+
+	$oldnode->command_ok(@regress_command,
+		'regression test run on old instance');
+}
+
+# Before dumping, get rid of objects not existing or not supported in later
+# versions. This depends on the version of the old server used, and matters
+# only if different versions are used for the dump.
+if (defined($ENV{oldinstall}))
+{
+	# Note that upgrade_adapt.sql from the new version is used, to
+	# cope with an upgrade to this version.
+	$oldnode->run_log(
+		[
+			'psql', '-X', '-f',
+			PostgreSQL::Test::Utils::perl2host(
+				"$newsrc/src/bin/pg_upgrade/upgrade_adapt.sql"),
+			'--port',
+			$oldnode->port,
+			'regression'
+		]);
+}
+
+# Initialize a new node for the upgrade.  This is done early so as it is
+# possible to know with which node's PATH the initial dump needs to be
+# taken.
+my $newnode = PostgreSQL::Test::Cluster->new('new_node');
+$newnode->init(extra => [ '--locale=C', '--encoding=LATIN1' ]);
+my $newbindir = $newnode->config_data('--bindir');
+my $oldbindir = $oldnode->config_data('--bindir');
+
+# Take a dump before performing the upgrade as a base comparison. Note
+# that we need to use pg_dumpall from the new node here.
+$newnode->command_ok(
+	[
+		'pg_dumpall', '--no-sync',
+		'-d',         $oldnode->connstr('postgres'),
+		'-f',         "$tempdir/dump1.sql"
+	],
+	'dump before running pg_upgrade');
+
+# After dumping, update references to the old source tree's regress.so
+# to point to the new tree.
+if (defined($ENV{oldinstall}))
+{
+	# First, fetch all the references to libraries that are not part
+	# of the default path $libdir.
+	my $output = $oldnode->safe_psql('regression',
+		"SELECT DISTINCT probin::text FROM pg_proc WHERE probin NOT LIKE '\$libdir%';"
+	);
+	chomp($output);
+	my @libpaths = split("\n", $output);
+
+	my $dump_data = slurp_file("$tempdir/dump1.sql");
+
+	my $newregresssrc = "$newsrc/src/test/regress";
+	foreach (@libpaths)
+	{
+		my $libpath = $_;
+		$libpath = dirname($libpath);
+		$dump_data =~ s/$libpath/$newregresssrc/g;
+	}
+
+	open my $fh, ">", "$tempdir/dump1.sql" or die "could not open dump file";
+	print $fh $dump_data;
+	close $fh;
+
+	# This replaces any references to the old tree's regress.so
+	# the new tree's regress.so.  Any references that do *not*
+	# match $libdir are switched so as this request does not
+	# depend on the path of the old source tree.  This is useful
+	# when using an old dump.  Do the operation on all the database
+	# that allow connections so as this includes the regression
+	# database and anything the user has set up.
+	$output = $oldnode->safe_psql('postgres',
+		"SELECT datname FROM pg_database WHERE datallowconn;");
+	chomp($output);
+	my @datnames = split("\n", $output);
+	foreach (@datnames)
+	{
+		my $datname = $_;
+		$oldnode->safe_psql(
+			$datname, "UPDATE pg_proc SET probin =
+		  regexp_replace(probin, '.*/', '$newregresssrc/')
+		  WHERE probin NOT LIKE '\$libdir/%'");
+	}
+}
+
+# Update the instance.
+$oldnode->stop;
+
+# Time for the real run.
+$newnode->command_ok(
+	[
+		'pg_upgrade',       '-d', $oldnode->data_dir, '-D',
+		$newnode->data_dir, '-b', $oldbindir,         '-B',
+		$newbindir,         '-p', $oldnode->port,     '-P',
+		$newnode->port
+	],
+	'run of pg_upgrade for new instance');
+$newnode->start;
+
+# Take a second dump on the upgraded instance.
+$newnode->run_log(
+	[
+		'pg_dumpall', '--no-sync',
+		'-d',         $newnode->connstr('postgres'),
+		'-f',         "$tempdir/dump2.sql"
+	]);
+
+# Compare the two dumps, there should be no differences.
+command_ok([ 'diff', '-q', "$tempdir/dump1.sql", "$tempdir/dump2.sql" ],
+	'old and new dump match after pg_upgrade');
diff --git a/src/bin/pg_upgrade/test.sh b/src/bin/pg_upgrade/test.sh
deleted file mode 100644
index 32d186d897..0000000000
--- a/src/bin/pg_upgrade/test.sh
+++ /dev/null
@@ -1,279 +0,0 @@
-#!/bin/sh
-
-# src/bin/pg_upgrade/test.sh
-#
-# Test driver for pg_upgrade.  Initializes a new database cluster,
-# runs the regression tests (to put in some data), runs pg_dumpall,
-# runs pg_upgrade, runs pg_dumpall again, compares the dumps.
-#
-# Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
-# Portions Copyright (c) 1994, Regents of the University of California
-
-set -e
-
-: ${MAKE=make}
-
-# Guard against parallel make issues (see comments in pg_regress.c)
-unset MAKEFLAGS
-unset MAKELEVEL
-
-# Run a given "initdb" binary and overlay the regression testing
-# authentication configuration.
-standard_initdb() {
-	# To increase coverage of non-standard segment size and group access
-	# without increasing test runtime, run these tests with a custom setting.
-	# Also, specify "-A trust" explicitly to suppress initdb's warning.
-	# --allow-group-access and --wal-segsize have been added in v11.
-	"$1" -N --wal-segsize 1 --allow-group-access -A trust
-	if [ -n "$TEMP_CONFIG" -a -r "$TEMP_CONFIG" ]
-	then
-		cat "$TEMP_CONFIG" >> "$PGDATA/postgresql.conf"
-	fi
-	../../test/regress/pg_regress --config-auth "$PGDATA"
-}
-
-# What flavor of host are we on?
-# Treat MINGW* (msys1) and MSYS* (msys2) the same.
-testhost=`uname -s | sed 's/^MSYS/MINGW/'`
-
-# Establish how the server will listen for connections
-case $testhost in
-	MINGW*)
-		LISTEN_ADDRESSES="localhost"
-		PG_REGRESS_SOCKET_DIR=""
-		PGHOST=localhost
-		;;
-	*)
-		LISTEN_ADDRESSES=""
-		# Select a socket directory.  The algorithm is from the "configure"
-		# script; the outcome mimics pg_regress.c:make_temp_sockdir().
-		if [ x"$PG_REGRESS_SOCKET_DIR" = x ]; then
-			set +e
-			dir=`(umask 077 &&
-				  mktemp -d /tmp/pg_upgrade_check-XXXXXX) 2>/dev/null`
-			if [ ! -d "$dir" ]; then
-				dir=/tmp/pg_upgrade_check-$$-$RANDOM
-				(umask 077 && mkdir "$dir")
-				if [ ! -d "$dir" ]; then
-					echo "could not create socket temporary directory in \"/tmp\""
-					exit 1
-				fi
-			fi
-			set -e
-			PG_REGRESS_SOCKET_DIR=$dir
-			trap 'rm -rf "$PG_REGRESS_SOCKET_DIR"' 0
-			trap 'exit 3' 1 2 13 15
-		fi
-		PGHOST=$PG_REGRESS_SOCKET_DIR
-		;;
-esac
-
-POSTMASTER_OPTS="-F -c listen_addresses=\"$LISTEN_ADDRESSES\" -k \"$PG_REGRESS_SOCKET_DIR\""
-export PGHOST
-
-# don't rely on $PWD here, as old shells don't set it
-temp_root=`pwd`/tmp_check
-rm -rf "$temp_root"
-mkdir "$temp_root"
-
-: ${oldbindir=$bindir}
-
-: ${oldsrc=../../..}
-oldsrc=`cd "$oldsrc" && pwd`
-newsrc=`cd ../../.. && pwd`
-
-# We need to make pg_regress use psql from the desired installation
-# (likely a temporary one), because otherwise the installcheck run
-# below would try to use psql from the proper installation directory
-# of the target version, which might be outdated or not exist. But
-# don't override anything else that's already in EXTRA_REGRESS_OPTS.
-EXTRA_REGRESS_OPTS="$EXTRA_REGRESS_OPTS --bindir='$oldbindir'"
-export EXTRA_REGRESS_OPTS
-
-# While in normal cases this will already be set up, adding bindir to
-# path allows test.sh to be invoked with different versions as
-# described in ./TESTING
-PATH=$bindir:$PATH
-export PATH
-
-BASE_PGDATA="$temp_root/data"
-PGDATA="${BASE_PGDATA}.old"
-export PGDATA
-
-# Send installcheck outputs to a private directory.  This avoids conflict when
-# check-world runs pg_upgrade check concurrently with src/test/regress check.
-# To retrieve interesting files after a run, use pattern tmp_check/*/*.diffs.
-outputdir="$temp_root/regress"
-EXTRA_REGRESS_OPTS="$EXTRA_REGRESS_OPTS --outputdir=$outputdir"
-export EXTRA_REGRESS_OPTS
-mkdir "$outputdir"
-
-# pg_regress --make-tablespacedir would take care of that in 14~, but this is
-# still required for older versions where this option is not supported.
-if [ "$newsrc" != "$oldsrc" ]; then
-	mkdir "$outputdir"/testtablespace
-	mkdir "$outputdir"/sql
-	mkdir "$outputdir"/expected
-fi
-
-logdir=`pwd`/log
-rm -rf "$logdir"
-mkdir "$logdir"
-
-# Clear out any environment vars that might cause libpq to connect to
-# the wrong postmaster (cf pg_regress.c)
-#
-# Some shells, such as NetBSD's, return non-zero from unset if the variable
-# is already unset. Since we are operating under 'set -e', this causes the
-# script to fail. To guard against this, set them all to an empty string first.
-PGDATABASE="";        unset PGDATABASE
-PGUSER="";            unset PGUSER
-PGSERVICE="";         unset PGSERVICE
-PGSSLMODE="";         unset PGSSLMODE
-PGREQUIRESSL="";      unset PGREQUIRESSL
-PGCONNECT_TIMEOUT=""; unset PGCONNECT_TIMEOUT
-PGHOSTADDR="";        unset PGHOSTADDR
-
-# Select a non-conflicting port number, similarly to pg_regress.c
-PG_VERSION_NUM=`grep '#define PG_VERSION_NUM' "$newsrc"/src/include/pg_config.h | awk '{print $3}'`
-PGPORT=`expr $PG_VERSION_NUM % 16384 + 49152`
-export PGPORT
-
-i=0
-while psql -X postgres </dev/null 2>/dev/null
-do
-	i=`expr $i + 1`
-	if [ $i -eq 16 ]
-	then
-		echo port $PGPORT apparently in use
-		exit 1
-	fi
-	PGPORT=`expr $PGPORT + 1`
-	export PGPORT
-done
-
-# buildfarm may try to override port via EXTRA_REGRESS_OPTS ...
-EXTRA_REGRESS_OPTS="$EXTRA_REGRESS_OPTS --port=$PGPORT"
-export EXTRA_REGRESS_OPTS
-
-standard_initdb "$oldbindir"/initdb
-"$oldbindir"/pg_ctl start -l "$logdir/postmaster1.log" -o "$POSTMASTER_OPTS" -w
-
-# Create databases with names covering the ASCII bytes other than NUL, BEL,
-# LF, or CR.  BEL would ring the terminal bell in the course of this test, and
-# it is not otherwise a special case.  PostgreSQL doesn't support the rest.
-dbname1=`awk 'BEGIN { for (i= 1; i < 46; i++)
-	if (i != 7 && i != 10 && i != 13) printf "%c", i }' </dev/null`
-# Exercise backslashes adjacent to double quotes, a Windows special case.
-dbname1='\"\'$dbname1'\\"\\\'
-dbname2=`awk 'BEGIN { for (i = 46; i <  91; i++) printf "%c", i }' </dev/null`
-dbname3=`awk 'BEGIN { for (i = 91; i < 128; i++) printf "%c", i }' </dev/null`
-createdb "regression$dbname1" || createdb_status=$?
-createdb "regression$dbname2" || createdb_status=$?
-createdb "regression$dbname3" || createdb_status=$?
-
-# Extra options to apply to the dump.  This may be changed later.
-extra_dump_options=""
-
-if "$MAKE" -C "$oldsrc" installcheck-parallel; then
-	oldpgversion=`psql -X -A -t -d regression -c "SHOW server_version_num"`
-
-	# Before dumping, tweak the database of the old instance depending
-	# on its version.
-	if [ "$newsrc" != "$oldsrc" ]; then
-		# This SQL script has its own idea of the cleanup that needs to be
-		# done on the cluster to-be-upgraded, and includes version checks.
-		# Note that this uses the script stored on the new branch.
-		psql -X -d regression -f "$newsrc/src/bin/pg_upgrade/upgrade_adapt.sql" \
-			|| psql_fix_sql_status=$?
-
-		# Handling of --extra-float-digits gets messy after v12.
-		# Note that this changes the dumps from the old and new
-		# instances if involving an old cluster of v11 or older.
-		if [ $oldpgversion -lt 120000 ]; then
-			extra_dump_options="--extra-float-digits=0"
-		fi
-	fi
-
-	pg_dumpall $extra_dump_options --no-sync \
-		-f "$temp_root"/dump1.sql || pg_dumpall1_status=$?
-
-	if [ "$newsrc" != "$oldsrc" ]; then
-		# update references to old source tree's regress.so etc
-		fix_sql=""
-		case $oldpgversion in
-			*)
-				fix_sql="UPDATE pg_proc SET probin = replace(probin, '$oldsrc', '$newsrc') WHERE probin LIKE '$oldsrc%';"
-				;;
-		esac
-		psql -X -d regression -c "$fix_sql;" || psql_fix_sql_status=$?
-
-		mv "$temp_root"/dump1.sql "$temp_root"/dump1.sql.orig
-		sed "s;$oldsrc;$newsrc;g" "$temp_root"/dump1.sql.orig >"$temp_root"/dump1.sql
-	fi
-else
-	make_installcheck_status=$?
-fi
-"$oldbindir"/pg_ctl -m fast stop
-if [ -n "$createdb_status" ]; then
-	exit 1
-fi
-if [ -n "$make_installcheck_status" ]; then
-	exit 1
-fi
-if [ -n "$psql_fix_sql_status" ]; then
-	exit 1
-fi
-if [ -n "$pg_dumpall1_status" ]; then
-	echo "pg_dumpall of pre-upgrade database cluster failed"
-	exit 1
-fi
-
-PGDATA="$BASE_PGDATA"
-
-standard_initdb 'initdb'
-
-pg_upgrade $PG_UPGRADE_OPTS -d "${PGDATA}.old" -D "$PGDATA" -b "$oldbindir" -p "$PGPORT" -P "$PGPORT"
-
-# make sure all directories and files have group permissions, on Unix hosts
-# Windows hosts don't support Unix-y permissions.
-case $testhost in
-	MINGW*|CYGWIN*) ;;
-	*)	if [ `find "$PGDATA" -type f ! -perm 640 | wc -l` -ne 0 ]; then
-			echo "files in PGDATA with permission != 640";
-			exit 1;
-		fi ;;
-esac
-
-case $testhost in
-	MINGW*|CYGWIN*) ;;
-	*)	if [ `find "$PGDATA" -type d ! -perm 750 | wc -l` -ne 0 ]; then
-			echo "directories in PGDATA with permission != 750";
-			exit 1;
-		fi ;;
-esac
-
-pg_ctl start -l "$logdir/postmaster2.log" -o "$POSTMASTER_OPTS" -w
-
-pg_dumpall $extra_dump_options --no-sync \
-	-f "$temp_root"/dump2.sql || pg_dumpall2_status=$?
-pg_ctl -m fast stop
-
-if [ -n "$pg_dumpall2_status" ]; then
-	echo "pg_dumpall of post-upgrade database cluster failed"
-	exit 1
-fi
-
-case $testhost in
-	MINGW*)	MSYS2_ARG_CONV_EXCL=/c cmd /c delete_old_cluster.bat ;;
-	*)	    sh ./delete_old_cluster.sh ;;
-esac
-
-if diff "$temp_root"/dump1.sql "$temp_root"/dump2.sql >/dev/null; then
-	echo PASSED
-	exit 0
-else
-	echo "Files $temp_root/dump1.sql and $temp_root/dump2.sql differ"
-	echo "dumps were not identical"
-	exit 1
-fi
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index c061e850fb..2bf2fcb69d 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -327,6 +327,31 @@ sub install_path
 
 =pod

 Return a string containing human-readable diagnostic information (paths, etc)
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index dc5b2df7d7..d62a15883e 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -287,6 +287,10 @@ sub bincheck
 	foreach my $dir (@bin_dirs)
 	{
 		next unless -d "$dir/t";
+		# Do not consider pg_upgrade, as it is handled by
+		# upgradecheck.
+		next if ($dir =~ "/pg_upgrade/");
+
 		my $status = tap_check($dir);
 		$mstat ||= $status;
 	}
@@ -591,89 +595,12 @@ sub generate_db
 
 sub upgradecheck
 {
-	my $status;
-	my $cwd = getcwd();
+	InstallTemp();
+	# Tweak environment for the upgrade tests
+	$ENV{REGRESS_OUTPUTDIR} = "$topdir/src/bin/pg_upgrade";
 
+	my $mstat = tap_check("$topdir/src/bin/pg_upgrade");
+	exit $mstat if $mstat;
 	return;
 }
 
-- 
2.34.1