Date: Wed, 11 May 2016 01:36:48 -0400
From: Noah Misch <noah@leadboat.com>
To: Kevin Grittner <kgrittn@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>,
	Tomas Vondra <tomas.vondra@2ndquadrant.com>,
	Andres Freund <andres@anarazel.de>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: what to revert
Message-ID: <20160511053648.GA89493@tornado.leadboat.com>
References: 
 <CACjxUsPb63TqU1uBc6zW5fsFynUB=BZ8NbFp6E=jK9qA7twibg@mail.gmail.com>
 <20160503174104.tk7oz4akwbeqsmuq@alap3.anarazel.de>
 <c7ad0f06-52c6-6284-c3a0-251807be9d3f@2ndquadrant.com>
 <20160503224241.rwtxrhgmprzqxhlv@alap3.anarazel.de>
 <eb3d6105-7165-fe71-f753-0a95281c3c19@2ndquadrant.com>
 <e17a7535-09b8-2273-0f7c-2d368390b0ed@2ndquadrant.com>
 <CACjxUsPgmm+LLG1+3d56EhCD8yEKP_b14zHGFOUpJp0Qx-J2pw@mail.gmail.com>
 <CACjxUsOL0s2DWRT5kNgc_498rsrrGXVNWHGiqBiAA2BGv9kxZw@mail.gmail.com>
 <2289.1462888970@sss.pgh.pa.us>
 <CACjxUsO+KT6NHc3AgLWfLpGFYpbiSeazWe2d9wiFe-mRwsXD9g@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CACjxUsO+KT6NHc3AgLWfLpGFYpbiSeazWe2d9wiFe-mRwsXD9g@mail.gmail.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

On Tue, May 10, 2016 at 10:05:13AM -0500, Kevin Grittner wrote:
> On Tue, May 10, 2016 at 9:02 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Kevin Grittner <kgrittn@gmail.com> writes:
> >> There were 75 samples each of "disabled" and "reverted" in the
> >> spreadsheet.  Averaging them all, I see this:
> >
> >> reverted:  290,660 TPS
> >> disabled:  292,014 TPS
> >
> >> That's a 0.46% overall increase in performance with the patch,
> >> disabled, compared to reverting it.  I'm surprised that you
> >> consider that to be a "clearly measurable difference".  I mean, it
> >> was measured and it is a difference, but it seems to be well within
> >> the noise.  Even though it is based on 150 samples, I'm not sure we
> >> should consider it statistically significant.
> >
> > You don't have to guess about that --- compare it to the standard
> > deviation within each group.
> 
> My statistics skills are rusty, but I thought that just gives you
> an effect size, not any idea of whether the effect is statistically
> significant.

I discourage focusing on the statistical significance, because the hypothesis
in question ("Applying revert.patch to 4bbc1a7e decreases 'pgbench -S -M
prepared -j N -c N' tps by 0.46%.") is already an unreliable proxy for
anything we care about.  PostgreSQL performance variation due to incidental,
ephemeral binary layout motion is roughly +/-5%.  Assuming perfect confidence
that 4bbc1a7e+revert.patch is 0.46% slower than 4bbc1a7e, the long-term effect
of revert.patch could be anywhere from -5% to +4%.

If one wishes to make benchmark-driven decisions about single-digit
performance changes, one must control for binary layout effects:
http://www.postgresql.org/message-id/87vbitb2zp.fsf@news-spur.riddles.org.uk
http://www.postgresql.org/message-id/20160416204452.GA1910190@tornado.leadboat.com

nm


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers