Feedback-ID: id4a34324:Fastmail
Date: Mon, 15 May 2023 17:14:47 -0700
From: Andres Freund <andres@anarazel.de>
To: Michael Paquier <michael@paquier.xyz>
Cc: Alexander Lakhin <exclusion@gmail.com>,
	Michael Paquier <michael.paquier@gmail.com>,
	MARK CALLAGHAN <mdcallag@gmail.com>,
	pgsql-hackers@lists.postgresql.org
Subject: Re: benchmark results comparing versions 15.2 and 16
Message-ID: <20230516001447.xrijuuqcn5s7phuz@awork3.anarazel.de>
References: <CAFbpF8N4bVFZ+eyWFj_+fZXZFSayk2kJVtZnouUVKGHudiDrSw@mail.gmail.com>
 <b32bed1b-0746-9b20-1472-4bdc9ca66d52@gmail.com>
 <20230508191117.v6475ip6hxpwukss@awork3.anarazel.de>
 <20230509164824.wtfn5aoggj4rcz67@awork3.anarazel.de>
 <ZFxu+IWkQpJhZkB3@paquier.xyz>
 <ZGHBGE45jKW8FEpe@paquier.xyz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZGHBGE45jKW8FEpe@paquier.xyz>
Archived-At: <https://www.postgresql.org/message-id/20230516001447.xrijuuqcn5s7phuz%40awork3.anarazel.de>
Precedence: bulk

Hi,

On 2023-05-15 14:20:24 +0900, Michael Paquier wrote:
> On Thu, May 11, 2023 at 01:28:40PM +0900, Michael Paquier wrote:
> > On Tue, May 09, 2023 at 09:48:24AM -0700, Andres Freund wrote:
> >> On 2023-05-08 12:11:17 -0700, Andres Freund wrote:
> >>> I can reproduce a significant regression due to f193883fc of a workload just
> >>> running
> >>> SELECT CURRENT_TIMESTAMP;
> >>> 
> >>> A single session running it on my workstation via pgbench -Mprepared gets
> >>> before:
> >>> tps = 89359.128359 (without initial connection time)
> >>> after:
> >>> tps = 83843.585152 (without initial connection time)
> >>> 
> >>> Obviously this is an extreme workload, but that nevertheless seems too large
> >>> to just accept...
> 
> Extreme is adapted for a worst-case scenario.  Looking at my notes
> from a few months back, that's kind of what I did on my laptop, which
> was the only machine I had at hand back then:
> - Compilation of code with -O2.

I assume without assertions as well?


> I have re-run a bit more pgbench (1 client, prepared query with a
> single SELECT on a SQL keyword, etc.).  And, TBH, I am not seeing as
> much difference as you do (nothing with default pgbench setup, FWIW),
> still that's able to show a bit more difference than the other two
> cases.

> HEAD shows me an average output close to 43900 TPS (3 run of
> 60s each, for instance), while relying on SQLValueFunction shows an
> average of 45000TPS.  That counts for ~2.4% output regression here
> on bigbox based on these numbers.  Not a regression as high as
> mentioned above, still that's visible.

45k seems too low for a modern machine, given that I get > 80k in such a
workload, on a workstation with server CPUs (i.e. many cores, but not that
fast individually).  Hence wondering about assertions being enabled...

I get quite variable performance if I don't pin client / server to the same
core, but even the slow performance is faster than 45k.

Greetings,

Andres Freund