Subject: Re: what to revert
To: Kevin Grittner <kgrittn@gmail.com>
References: 
 <CA+TgmoYOWTtBQEL+Bv=w93bvUjbXSUw3uGnp+R29dduZ==8K0Q@mail.gmail.com>
 <CACjxUsOuf_a6KMsFBNtt8Tn8y7qNLE2Wcb9BQeRFdHD6oAKp3g@mail.gmail.com>
 <20160503162209.ufuezlkovbrdrayp@alap3.anarazel.de>
 <CACjxUsPb63TqU1uBc6zW5fsFynUB=BZ8NbFp6E=jK9qA7twibg@mail.gmail.com>
 <20160503174104.tk7oz4akwbeqsmuq@alap3.anarazel.de>
 <c7ad0f06-52c6-6284-c3a0-251807be9d3f@2ndquadrant.com>
 <20160503224241.rwtxrhgmprzqxhlv@alap3.anarazel.de>
 <eb3d6105-7165-fe71-f753-0a95281c3c19@2ndquadrant.com>
 <e17a7535-09b8-2273-0f7c-2d368390b0ed@2ndquadrant.com>
 <CACjxUsPgmm+LLG1+3d56EhCD8yEKP_b14zHGFOUpJp0Qx-J2pw@mail.gmail.com>
 <CACjxUsOL0s2DWRT5kNgc_498rsrrGXVNWHGiqBiAA2BGv9kxZw@mail.gmail.com>
Cc: Andres Freund <andres@anarazel.de>,
 "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Message-ID: <a1338386-752a-485b-fa8c-5b113f10a4e6@2ndquadrant.com>
Date: Tue, 10 May 2016 18:31:49 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101
 Thunderbird/46.0a2
MIME-Version: 1.0
In-Reply-To: 
 <CACjxUsOL0s2DWRT5kNgc_498rsrrGXVNWHGiqBiAA2BGv9kxZw@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org


On 05/10/2016 03:04 PM, Kevin Grittner wrote:
> On Tue, May 10, 2016 at 3:29 AM, Kevin Grittner <kgrittn@gmail.com
> <mailto:kgrittn@gmail.com>> wrote:
>
>>> * The results are a bit noisy, but I think in general this shows
>>> that for certain cases there's a clearly measurable difference
>>> (up to 5%) between the "disabled" and "reverted" cases. This is
>>> particularly visible on the smallest data set.
>>
>> In some cases, the differences are in favor of disabled over
>> reverted.
>
> There were 75 samples each of "disabled" and "reverted" in the
> spreadsheet.  Averaging them all, I see this:
>
> reverted:  290,660 TPS
> disabled:  292,014 TPS

Well, that kinda assumes it's one large group. I was wondering whether 
the difference depends on some of the other factors (scale factor, 
number of clients), which is why I mentioned "for certain cases".

The other problem is averaging the difference like this overweights the 
results for large client counts. Also, it mixes results for different 
scales, which I think is pretty important.

The following table shows the differences between the disabled and 
reverted cases like this:

     sum('reverted' results with N clients)
    ---------------------------------------- - 1.0
     sum('disabled' results with N clients)

for each scale/client count combination. So for example 4.83% means with 
a single client on the smallest data set, the sum of the 5 runs for 
reverted was about 1.0483x than for disabled.

     scale        1       16       32      64      128
     100      4.83%    2.84%    1.21%   1.16%    3.85%
     3000     1.97%    0.83%    1.78%   0.09%    7.70%
     10000   -6.94%   -5.24%  -12.98%  -3.02%   -8.78%

So in average for each scale;

     scale    revert/disable
     100               2.78%
     3000              2.47%
     10000            -7.39%

Of course, it still might be due to noise. But looking at the two tables 
that seems rather unlikely, I guess.

>
> That's a 0.46% overall increase in performance with the patch,
> disabled, compared to reverting it.  I'm surprised that you
> consider that to be a "clearly measurable difference".  I mean, it
> was measured and it is a difference, but it seems to be well within
> the noise.  Even though it is based on 150 samples, I'm not sure we
> should consider it statistically significant.

Well, luckily we're in the position that we can collect more data.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers