Date: Mon, 3 Apr 2017 10:43:54 -0700
From: Andres Freund <andres@anarazel.de>
To: Stephen Frost <sfrost@snowman.net>
Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>,
	Michael Paquier <michael.paquier@gmail.com>,
	PostgreSQL mailing lists <pgsql-hackers@postgresql.org>,
	Noah Misch <noah@leadboat.com>
Subject: Re: Rewriting the test of pg_upgrade as a TAP test
Message-ID: <20170403174354.45oqokiy2ellfhbd@alap3.anarazel.de>
References: 
 <CAB7nPqRdaN1A1YNjxNL9T1jUEWct8ttqq29dNv8W_o37+e8wfA@mail.gmail.com>
 <62a4e16e-5ea3-89c9-c94a-ee3be953b94a@2ndquadrant.com>
 <20170403153452.GL9812@tamriel.snowman.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20170403153452.GL9812@tamriel.snowman.net>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

On 2017-04-03 11:34:52 -0400, Stephen Frost wrote:
> Peter,
> 
> * Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:
> > On 4/3/17 09:07, Michael Paquier wrote:
> > > I had for some time a WIP patch on which dust has accumulated, so
> > > attached is a more polished version. In more details, here is what
> > > happens:
> > > - test.sh is removed.
> > > - vcregress.pl loses upgradecheck.
> > > - The new test is added. In the case of MSVC this is now part of bincheck.
> > > Patch has been tested on macos and Windows.
> > 
> > This is a useful start.  What I'd really like to see is that instead of
> > running the full serial tests to populate the pre-upgrade database, we
> > determine a useful subset of what that ends up generating and just
> > populate with that.
> 
> In the past, we've had the notion that the regression tests are intended
> to also cover pg_upgrade/pg_dump by "leaving things around".  What I
> found in my efforts to provide better coverage in pg_dump is that there
> was quite a bit of coverage missing using that approach.
> 
> Perhaps that could be fixed, but I tend to think it's a better approach
> to have a complete set of pg_upgrade/pg_dump tests in one place that
> doesn't also have a bunch of other tests mixed in (and would also mean
> that the regular regression tests could be 'clean').
> 
> I could also see us defining one set of commands to run which create
> every type of object in the system that pg_dump understands and then
> using that to perform the pg_dump and pg_upgrade tests.  Those commands
> would have to be annotated with minimum major version and maximum major
> version, assuming we're going to use them cross-version, but that should
> be reasonably straight-forward to do.
> 
> Another question is how much sense it makes to test this logic,
> essentially, twice.  The testing of pg_dump covers the pg_dump code,
> which is what pg_upgrade uses anyway.  The pg_upgrade tests really need
> to cover the non-pg_dump-related parts, assuming we have appropriate
> coverage in the pg_dump tests for the --binary-upgrade mode.  Of course,
> if we don't, then we should go about fixing that.  There are certainly
> some tests now but perhaps we need more or need to have improvmenets
> made there.

I don't fundamentally disagree with anything here, but I think it'd be a
serious mistake to link this to the conversion of the pg_upgrade tests
to tap tests.

Greetings,

Andres Freund


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers