Date: Wed, 5 Apr 2017 10:50:19 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Andres Freund <andres@anarazel.de>
Cc: Tom Lane <tgl@sss.pgh.pa.us>,
	Michael Paquier <michael.paquier@gmail.com>,
	Peter Eisentraut <peter.eisentraut@2ndquadrant.com>,
	PostgreSQL mailing lists <pgsql-hackers@postgresql.org>,
	Noah Misch <noah@leadboat.com>
Subject: Re: Rewriting the test of pg_upgrade as a TAP test
Message-ID: <20170405145019.GD9812@tamriel.snowman.net>
References: 
 <CAB7nPqRdaN1A1YNjxNL9T1jUEWct8ttqq29dNv8W_o37+e8wfA@mail.gmail.com>
 <62a4e16e-5ea3-89c9-c94a-ee3be953b94a@2ndquadrant.com>
 <20170403153200.jqntjo7zat6x5rhh@alap3.anarazel.de>
 <42d412a2-9cd7-7ee1-0c1a-e8d230d568b0@2ndquadrant.com>
 <CAB7nPqR9JWCoU8z_CXovL2UkTxahcEx3TzyBYzk3qyXh+MTXMg@mail.gmail.com>
 <20170405135211.GZ9812@tamriel.snowman.net>
 <6679.1491401713@sss.pgh.pa.us>
 <20170405144041.GC9812@tamriel.snowman.net>
 <20170405144521.7guijbgfxpstfbq2@alap3.anarazel.de>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="Cyf1dbM4CABc4Tyb"
Content-Disposition: inline
In-Reply-To: <20170405144521.7guijbgfxpstfbq2@alap3.anarazel.de>
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org


--Cyf1dbM4CABc4Tyb
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Andres,

* Andres Freund (andres@anarazel.de) wrote:
> On 2017-04-05 10:40:41 -0400, Stephen Frost wrote:
> > * Tom Lane (tgl@sss.pgh.pa.us) wrote:
> > > Stephen Frost <sfrost@snowman.net> writes:
> > > > I believe that what Peter was getting at is that the pg_dump TAP te=
sts
> > > > create a whole slew of objects in just a few seconds and are able to
> > > > then exercise those code-paths in pg_dump, without needing to run t=
he
> > > > entire serial regression test run.
> > >=20
> > > Right.  But there's a certain amount of serendipity involved in using=
 the
> > > core regression tests' final results.  For example, I don't know how =
long
> > > it would've taken us to understand the problems around dumping and
> > > reloading child tables with inconsistent column orders, had there not=
 been
> > > examples of that in the regression tests.  I worry that creating a st=
erile
> > > set of objects for testing pg_dump will leave blind spots, because it=
 will
> > > mean that we only test cases that we explicitly created test cases fo=
r.
> >=20
> > We don't need to only create sterile sets of objects in the pg_dump TAP
> > tests.
>=20
> I really, really don't understand why we're conflating making pg_upgrade
> tests less fragile / duplicative with changing what we use to test it.
> This seems to have the sole result that we're not going to get anywhere.

Probably because the point was brought up that the regression tests for
pg_upgrade spend a bunch of time doing something which, ultimately,
don't actually add any real value.  Yes, there are bits of the core
regression tests that currently add value over what we have through
other approaches, but that's not where the bulk of running those tests
go.

> > I don't believe we need to populate GIN indexes or vacuum them
> > to test pg_dump/pg_upgrade either (at least, not if we're going to stick
> > to the pg_upgrade test basically being if pg_dump returns the same
> > results before-and-after).
>=20
> I think we *should* have populated GIN indexes. Yes, the coverage isn't
> perfect, but the VACUUM definitely gives a decent amount of coverage
> whether the gin index looks halfway sane after the upgrade.

We don't look at the gin index after the upgrade in the current
pg_upgrade testing, so I don't see why you feel it's at all valuable.
If we *did* do that (and I'm all for adding such tests), then perhaps
this argument would make sense, but we don't today and I haven't seen
anyone propose changing that.

Thanks!

Stephen

--Cyf1dbM4CABc4Tyb
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJY5QQrAAoJEO1sijiDR2RVqnIQAIZrCrS+T/uhYcRvp107/Tty
Wi5db76yxMv4DkEV8Tcg844MK7qRLNXa7iIOpSw8YVuy1Ob/MDz5UDdcZGSMwjO+
14ZwqK0rOafwn7QxNYT3O1mRLu+UIsqxAeLpP9UyMklOl6iNEXf6iOtz2UCjCkE0
rbDIrTEnzHddrJZuYboFfaRKvY6A1o9k/VUFayPOX7G2rgfulU8MO9G7v9f8iEf3
6SwTrvOnjlyVjzMz4EBbZcckHdUFqdMBs1DmWkZm5+Ou6dUDxGq8LhNvu09YMTtz
UQ03mwDXr7S2SvOLtQNCDXFNvhWDvNMJ00URcF0y79F55vLHKAbuptY31ZwNkpWH
TiV+sljrAIZ46pLNb9EJZfddQNp90lkKkd8Yz/OqH5LyIuqG5EcRSjpTnJblB2SV
a2NavXgcBC2nYOp2wMRo/VeCekT5V6VYzEG+i0v8KUfX03aVfEJ6GxuZUAiwyBir
n1Nzw+QYch4uCo0KXqz2lGdqAIk8xRRnL+BlypLvYgSWWotIHqbl4ftbfLOI8R48
84v65SI27jDF2TWtW8oHa0Vl7QGps0USBXgbUgpD0e8EMObpYPllF/SIKFy84vZ8
C6GRnY3BZYBSa/kWShljUqW/6IZ8hHGWVZRhyK2+y1IYMCltxEWQBEsTaUp5yoTq
zYRw2fY6FmtJwtx9cUx1
=xIvH
-----END PGP SIGNATURE-----

--Cyf1dbM4CABc4Tyb--