Date: Wed, 5 Apr 2017 10:40:41 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Michael Paquier <michael.paquier@gmail.com>,
	Peter Eisentraut <peter.eisentraut@2ndquadrant.com>,
	Andres Freund <andres@anarazel.de>,
	PostgreSQL mailing lists <pgsql-hackers@postgresql.org>,
	Noah Misch <noah@leadboat.com>
Subject: Re: Rewriting the test of pg_upgrade as a TAP test
Message-ID: <20170405144041.GC9812@tamriel.snowman.net>
References: 
 <CAB7nPqRdaN1A1YNjxNL9T1jUEWct8ttqq29dNv8W_o37+e8wfA@mail.gmail.com>
 <62a4e16e-5ea3-89c9-c94a-ee3be953b94a@2ndquadrant.com>
 <20170403153200.jqntjo7zat6x5rhh@alap3.anarazel.de>
 <42d412a2-9cd7-7ee1-0c1a-e8d230d568b0@2ndquadrant.com>
 <CAB7nPqR9JWCoU8z_CXovL2UkTxahcEx3TzyBYzk3qyXh+MTXMg@mail.gmail.com>
 <20170405135211.GZ9812@tamriel.snowman.net>
 <6679.1491401713@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="2MV3VyHqaIgwH6pq"
Content-Disposition: inline
In-Reply-To: <6679.1491401713@sss.pgh.pa.us>
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org


--2MV3VyHqaIgwH6pq
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Tom,

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > I believe that what Peter was getting at is that the pg_dump TAP tests
> > create a whole slew of objects in just a few seconds and are able to
> > then exercise those code-paths in pg_dump, without needing to run the
> > entire serial regression test run.
>=20
> Right.  But there's a certain amount of serendipity involved in using the
> core regression tests' final results.  For example, I don't know how long
> it would've taken us to understand the problems around dumping and
> reloading child tables with inconsistent column orders, had there not been
> examples of that in the regression tests.  I worry that creating a sterile
> set of objects for testing pg_dump will leave blind spots, because it will
> mean that we only test cases that we explicitly created test cases for.

We don't need to only create sterile sets of objects in the pg_dump TAP
tests.  I don't believe we need to populate GIN indexes or vacuum them
to test pg_dump/pg_upgrade either (at least, not if we're going to stick
to the pg_upgrade test basically being if pg_dump returns the same
results before-and-after).

I'm all for adding tests into pg_dump which do things like drop columns
and change column names and other cases which could impact if the
pg_dump is correct or not, and there's nothing preventing those tests
=66rom being added in the existing structure.  Certainly, before we remove
the coverage provided by running the serial test suite and then using
pg_upgrade, we should analyze what is being tested and ensure that we're
providing that same set of testing in the pg_dump TAP tests.

> > I'm still not completely convinced that we actually need to
> > independently test pg_upgrade by creating all the objects which the
> > pg_dump TAP tests do, given that pg_upgrade just runs pg_dump
> > underneath.  If we really want to do that, however, what we should do is
> > abstract out the pg_dump set of tests into a place that both the pg_dump
> > and pg_upgrade TAP tests could use them to create all the types of
> > objects which are supported to perform their tests.
>=20
> I think it's largely pointless to test pg_dump --binary-upgrade except
> as a part of pg_upgrade.

That's how I discovered that comments and security labels weren't being
pulled through to the new cluster for blobs, so I would have to disagree
with this.  Frankly, it's also much more straight-forward to run
pg_dump --binary-upgrade than it is to get pg_upgrade to do the same.

Still, I'm not actually against centralizing the tests done with pg_dump
such that they could be used by pg_upgrade also.  Creating all those
objects takes less than a second, at least on my system, so it would
still be quite a bit faster than running the serial regression suite.

We might also consider if there's a way to change the format for those
tests to make them a bit less impenetrable for non-Perl folks to work
with and to make it simpler to add new tests as new features are added.

Thanks!

Stephen

--2MV3VyHqaIgwH6pq
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJY5QHpAAoJEO1sijiDR2RVmyYQAMJq2XDqdkcng4t+fITrzYag
t3n1OYB1TwgtJ/oiKqwcMvrt/+/P8QpM12KGjyhnjJSHo7tFm1v31vl3eTkWGtQv
3icmqhoC5GP1h83Fy5jejAg5TnF1ITx0NIlk2Ewax4dVBpg1oyCoYwDNaiFae/y+
Gi5LlscjV/zutDxBe5uHGxr7VcAjbUJ98VI+ME/2A64QdtFugpVD3wybeRnekQ+3
V2w4NwNe7cbqers3LRARlFmiG1u5IzUeKdcJ5Vvqu6WpLdlntFN89PvMrM939s8K
5bSgXxp3+0oDvOO77d3QtMGnov9ycNgW2mqQ11BuKMKx9nf2eD68T3jlJFBqLvQB
AaOBnu7GRvBABUBLx+ImM94YR6ySqlj08ojrU6VvbLselYlOgaLasu6/0/Mdbleo
1OpnSycAkTNABL7oUFJJVTRyuwg9dbF4MbRYhAOspclmpfSXhAIf7rv6inG2c0N6
car4njK1DnxXLJx0NijnryVdJMmtRJF5Fg+zM2wzKVovUxQ4BO7JSjRWSbCvyXTz
cwy7xXBGtLE5T5ClBJ1NNZGGquWlAQnUR7nyUXbQ59a3GjqklxjqBKCWZsz4fI8D
6F0HanjrHyWXVL3xaPWKwCgss5VGmhapoVJ9pcE86ULD074dH1F9QHFjlyIzUIPd
ZSXV7j5YZCi3OfxaBao+
=QTJe
-----END PGP SIGNATURE-----

--2MV3VyHqaIgwH6pq--