pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Noah Misch <noah@leadboat.com>
To: Kevin Grittner <kgrittn@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Andres Freund <andres@anarazel.de>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: what to revert
Date: Wed, 11 May 2016 01:36:48 -0400
Message-ID: <20160511053648.GA89493@tornado.leadboat.com> (raw)
In-Reply-To: <CACjxUsO+KT6NHc3AgLWfLpGFYpbiSeazWe2d9wiFe-mRwsXD9g@mail.gmail.com>
References: <CACjxUsPb63TqU1uBc6zW5fsFynUB=BZ8NbFp6E=jK9qA7twibg@mail.gmail.com>
	<20160503174104.tk7oz4akwbeqsmuq@alap3.anarazel.de>
	<c7ad0f06-52c6-6284-c3a0-251807be9d3f@2ndquadrant.com>
	<20160503224241.rwtxrhgmprzqxhlv@alap3.anarazel.de>
	<eb3d6105-7165-fe71-f753-0a95281c3c19@2ndquadrant.com>
	<e17a7535-09b8-2273-0f7c-2d368390b0ed@2ndquadrant.com>
	<CACjxUsPgmm+LLG1+3d56EhCD8yEKP_b14zHGFOUpJp0Qx-J2pw@mail.gmail.com>
	<CACjxUsOL0s2DWRT5kNgc_498rsrrGXVNWHGiqBiAA2BGv9kxZw@mail.gmail.com>
	<2289.1462888970@sss.pgh.pa.us>
	<CACjxUsO+KT6NHc3AgLWfLpGFYpbiSeazWe2d9wiFe-mRwsXD9g@mail.gmail.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>

On Tue, May 10, 2016 at 10:05:13AM -0500, Kevin Grittner wrote:
> On Tue, May 10, 2016 at 9:02 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > Kevin Grittner <kgrittn@gmail.com> writes:
> >> There were 75 samples each of "disabled" and "reverted" in the
> >> spreadsheet.  Averaging them all, I see this:
> >
> >> reverted:  290,660 TPS
> >> disabled:  292,014 TPS
> >
> >> That's a 0.46% overall increase in performance with the patch,
> >> disabled, compared to reverting it.  I'm surprised that you
> >> consider that to be a "clearly measurable difference".  I mean, it
> >> was measured and it is a difference, but it seems to be well within
> >> the noise.  Even though it is based on 150 samples, I'm not sure we
> >> should consider it statistically significant.
> >
> > You don't have to guess about that --- compare it to the standard
> > deviation within each group.
> 
> My statistics skills are rusty, but I thought that just gives you
> an effect size, not any idea of whether the effect is statistically
> significant.

I discourage focusing on the statistical significance, because the hypothesis
in question ("Applying revert.patch to 4bbc1a7e decreases 'pgbench -S -M
prepared -j N -c N' tps by 0.46%.") is already an unreliable proxy for
anything we care about.  PostgreSQL performance variation due to incidental,
ephemeral binary layout motion is roughly +/-5%.  Assuming perfect confidence
that 4bbc1a7e+revert.patch is 0.46% slower than 4bbc1a7e, the long-term effect
of revert.patch could be anywhere from -5% to +4%.

If one wishes to make benchmark-driven decisions about single-digit
performance changes, one must control for binary layout effects:
http://www.postgresql.org/message-id/87vbitb2zp.fsf@news-spur.riddles.org.uk
http://www.postgresql.org/message-id/20160416204452.GA1910190@tornado.leadboat.com

nm


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers



view thread (56+ messages)  latest in thread

Message-ID: <20160511053648.GA89493@tornado.leadboat.com>
Permalink:  ../20160511053648.GA89493@tornado.leadboat.com/
Also on:    postgresql.org/message-id/20160511053648.GA89493@tornado.leadboat.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: noah@leadboat.com, kgrittn@gmail.com, tgl@sss.pgh.pa.us, tomas.vondra@2ndquadrant.com, andres@anarazel.de
  Subject: Re: what to revert
  In-Reply-To: <20160511053648.GA89493@tornado.leadboat.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox