From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Kevin Grittner <kgrittn@gmail.com>
Cc: Andres Freund <andres@anarazel.de>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: what to revert
Date: Tue, 10 May 2016 18:31:49 +0200
Message-ID: <a1338386-752a-485b-fa8c-5b113f10a4e6@2ndquadrant.com> (raw)
In-Reply-To: <CACjxUsOL0s2DWRT5kNgc_498rsrrGXVNWHGiqBiAA2BGv9kxZw@mail.gmail.com>
References: <CA+TgmoYOWTtBQEL+Bv=w93bvUjbXSUw3uGnp+R29dduZ==8K0Q@mail.gmail.com>
<CACjxUsOuf_a6KMsFBNtt8Tn8y7qNLE2Wcb9BQeRFdHD6oAKp3g@mail.gmail.com>
<20160503162209.ufuezlkovbrdrayp@alap3.anarazel.de>
<CACjxUsPb63TqU1uBc6zW5fsFynUB=BZ8NbFp6E=jK9qA7twibg@mail.gmail.com>
<20160503174104.tk7oz4akwbeqsmuq@alap3.anarazel.de>
<c7ad0f06-52c6-6284-c3a0-251807be9d3f@2ndquadrant.com>
<20160503224241.rwtxrhgmprzqxhlv@alap3.anarazel.de>
<eb3d6105-7165-fe71-f753-0a95281c3c19@2ndquadrant.com>
<e17a7535-09b8-2273-0f7c-2d368390b0ed@2ndquadrant.com>
<CACjxUsPgmm+LLG1+3d56EhCD8yEKP_b14zHGFOUpJp0Qx-J2pw@mail.gmail.com>
<CACjxUsOL0s2DWRT5kNgc_498rsrrGXVNWHGiqBiAA2BGv9kxZw@mail.gmail.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
On 05/10/2016 03:04 PM, Kevin Grittner wrote:
> On Tue, May 10, 2016 at 3:29 AM, Kevin Grittner <kgrittn@gmail.com
> <mailto:kgrittn@gmail.com>> wrote:
>
>>> * The results are a bit noisy, but I think in general this shows
>>> that for certain cases there's a clearly measurable difference
>>> (up to 5%) between the "disabled" and "reverted" cases. This is
>>> particularly visible on the smallest data set.
>>
>> In some cases, the differences are in favor of disabled over
>> reverted.
>
> There were 75 samples each of "disabled" and "reverted" in the
> spreadsheet. Averaging them all, I see this:
>
> reverted: 290,660 TPS
> disabled: 292,014 TPS
Well, that kinda assumes it's one large group. I was wondering whether
the difference depends on some of the other factors (scale factor,
number of clients), which is why I mentioned "for certain cases".
The other problem is averaging the difference like this overweights the
results for large client counts. Also, it mixes results for different
scales, which I think is pretty important.
The following table shows the differences between the disabled and
reverted cases like this:
sum('reverted' results with N clients)
---------------------------------------- - 1.0
sum('disabled' results with N clients)
for each scale/client count combination. So for example 4.83% means with
a single client on the smallest data set, the sum of the 5 runs for
reverted was about 1.0483x than for disabled.
scale 1 16 32 64 128
100 4.83% 2.84% 1.21% 1.16% 3.85%
3000 1.97% 0.83% 1.78% 0.09% 7.70%
10000 -6.94% -5.24% -12.98% -3.02% -8.78%
So in average for each scale;
scale revert/disable
100 2.78%
3000 2.47%
10000 -7.39%
Of course, it still might be due to noise. But looking at the two tables
that seems rather unlikely, I guess.
>
> That's a 0.46% overall increase in performance with the patch,
> disabled, compared to reverting it. I'm surprised that you
> consider that to be a "clearly measurable difference". I mean, it
> was measured and it is a difference, but it seems to be well within
> the noise. Even though it is based on 150 samples, I'm not sure we
> should consider it statistically significant.
Well, luckily we're in the position that we can collect more data.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@2ndquadrant.com, kgrittn@gmail.com, andres@anarazel.de
Subject: Re: what to revert
In-Reply-To: <a1338386-752a-485b-fa8c-5b113f10a4e6@2ndquadrant.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox