Date: Wed, 5 Apr 2017 21:30:38 -0400
From: Noah Misch <noah@leadboat.com>
To: Stephen Frost <sfrost@snowman.net>
Cc: Tom Lane <tgl@sss.pgh.pa.us>,
	Michael Paquier <michael.paquier@gmail.com>,
	Peter Eisentraut <peter.eisentraut@2ndquadrant.com>,
	Andres Freund <andres@anarazel.de>,
	PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
Subject: Re: Rewriting the test of pg_upgrade as a TAP test
Message-ID: <20170406013038.GA2731217@tornado.leadboat.com>
References: 
 <CAB7nPqRdaN1A1YNjxNL9T1jUEWct8ttqq29dNv8W_o37+e8wfA@mail.gmail.com>
 <62a4e16e-5ea3-89c9-c94a-ee3be953b94a@2ndquadrant.com>
 <20170403153200.jqntjo7zat6x5rhh@alap3.anarazel.de>
 <42d412a2-9cd7-7ee1-0c1a-e8d230d568b0@2ndquadrant.com>
 <CAB7nPqR9JWCoU8z_CXovL2UkTxahcEx3TzyBYzk3qyXh+MTXMg@mail.gmail.com>
 <20170405135211.GZ9812@tamriel.snowman.net>
 <6679.1491401713@sss.pgh.pa.us>
 <20170405144041.GC9812@tamriel.snowman.net>
 <8402.1491404529@sss.pgh.pa.us>
 <20170405151333.GF9812@tamriel.snowman.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20170405151333.GF9812@tamriel.snowman.net>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

On Wed, Apr 05, 2017 at 11:13:33AM -0400, Stephen Frost wrote:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
> > Stephen Frost <sfrost@snowman.net> writes:
> > > I'm all for adding tests into pg_dump which do things like drop columns
> > > and change column names and other cases which could impact if the
> > > pg_dump is correct or not, and there's nothing preventing those tests
> > > from being added in the existing structure.  Certainly, before we remove
> > > the coverage provided by running the serial test suite and then using
> > > pg_upgrade, we should analyze what is being tested and ensure that we're
> > > providing that same set of testing in the pg_dump TAP tests.
> > 
> > I don't think you grasped my basic point, which is that I'm worried about
> > emergent cases that we don't foresee needing to test (and that no amount
> > of code coverage checking would have shown up as being overlooked).
> > Admittedly, relying on the core regression tests to trigger such cases is
> > a pretty haphazard strategy, but it's way better than no strategy at all.
> 
> The tests that were added to the core regression suite were done so for
> a reason and hopefully we can identify cases where it'd make sense to
> also run those tests for pg_upgrade/pg_dump testing.

I think you _are_ missing Tom's point.  We've caught pg_dump and pg_upgrade
bugs thanks to regression database objects created for purposes unrelated to
pg_dump.  It's true that there exist other test strategies that are more
efficient or catch more bugs overall.  None of them substitute 100% for the
serendipity seen in testing dump/restore on the regression database.

> More-or-less
> anything that materially changes the catalog should be included, I would
> think.  Things that are only really only working with the heap/index
> files don't really need to be performed because the pg_upgrade process
> doesn't change those.

That is formally true.


Also, I agree with Andres that this is not a thread for discussing test
changes beyond mechanical translation of the pg_upgrade test suite.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers