Subject: Re: Reviewing freeze map code
To: Andres Freund <andres@anarazel.de>
References: 
 <CA+TgmobF=86sa=6dF30jaiEFOhZnO+asOB5j69PynPCqtaz9xA@mail.gmail.com>
 <572D0309.4090402@commandprompt.com>
 <20160506205012.ipuhvctss2ozzs6m@alap3.anarazel.de>
 <572D0471.3040009@commandprompt.com>
 <20160506205824.ysst7mk3k26ztok2@alap3.anarazel.de>
 <WM!3c465a785ef366521ed338bb93585bc99a43cc6db976bb0e71111a522ee7f39a31942de6526cb0bccd5f2b7d1678818f!@asav-3.01.com>
 <572D062E.5030308@agliodbs.com> <572D068F.3050608@commandprompt.com>
 <20160506210806.f6pnk6ueirjqrlmu@alap3.anarazel.de>
 <572D09D9.5080407@commandprompt.com>
 <20160506212958.wwwc22hfesvj76af@alap3.anarazel.de>
Cc: Josh berkus <josh@agliodbs.com>, Robert Haas <robertmhaas@gmail.com>,
 Masahiko Sawada <sawada.mshk@gmail.com>,
 "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
From: "Joshua D. Drake" <jd@commandprompt.com>
Organization: Command Prompt, Inc.
Message-ID: <572D0F2D.7050207@commandprompt.com>
Date: Fri, 6 May 2016 14:39:57 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.7.2
MIME-Version: 1.0
In-Reply-To: <20160506212958.wwwc22hfesvj76af@alap3.anarazel.de>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

On 05/06/2016 02:29 PM, Andres Freund wrote:
> Hi,
>
>
> On 2016-05-06 14:17:13 -0700, Joshua D. Drake wrote:
>> How do I test?
>>
>> Is there a script I can run?
>
> Unfortunately there's few interesting things to test with pre-made
> scripts. There's no relevant OS dependency here, so each already
> existing test doesn't really lead to significantly increased coverage
> being run by other people.  Generally, when testing for correctness
> issues, it's often of limited benefit to run tests written by the author
> of reviewer - such scripts will usually just test things either has
> thought of.  The dangerous areas are the ones neither author or reviewer
> has considered.

I can't argue with that.

>
>
>> Are there specific things I can do to try and break it?
>
> Upgrade clusters using pg_upgrade and make sure things like index only
> scans still work and yield correct data.  Set up workloads that involve
> freezing, and check that less WAL (and not more!) is generated with 9.6
> than with 9.5.  Make sure queries still work.
>
>
>> What are we looking for exactly?
>
> Data corruption, efficiency problems.
>

I am really not trying to be difficult here but Data Corruption is an 
easy one... what is the metric we accept as an efficiency problem?

>
>> A lot of -hackers seem to forget that although we have 100 -hackers, we have
>> 10000 "consultant/practitioners". Could I read the code and with a weekend
>> of WTF and -hackers questions figure out what is going on, yes but a lot of
>> people couldn't and I don't have the time.
>
> I think tests without reading the code are quite sensible and
> important. And it perfectly makes sense to ask for information about
> what to test.  But fundamentally testing is a lot of work, as is writing
> and reviewing code; unless you're really really good at destructive
> testing, you won't find much in a 15 minute break.
>

Yes, this is true but with a proper testing framework, I don't need a 15 
minute break. I need 1 hour to configure, the rest just "happens" and 
reports back.

I have cycles to test, I have team members to help test (as do *lots* of 
other people) but sometimes we just get lost in how to help.

>
>> You want me (or people like me) to test more? Give us an easy way to
>> do it.
>
> Useful additional testing and easy just don't go well together. By the
> time I have made it easy I've done the testing that's needed.

I don't know that I can agree with this. A proper harness allows you to 
execute: go.sh and boom... 2, 4, even 8 hours later you get a report. I 
will not argue that it isn't easy to implement but I know it can be done.

>> Otherwise, we do what we can, which is try and interface on the things that
>> will directly and immediately affect us (like keywords and syntax).
>
> The amount of bikeshedding on -hackers steals energy and time for
> actually working on stuff, including testing. So I have little sympathy
> for the amount of bike shedding done.

Insuring a reasonable and thought out interface for users is not bike 
shedding, it is at least as important and possibly more important than 
any feature we add.

Sincerely,

JD

>
> Greetings,
>
> Andres Freund
>


-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers