Feedback-ID: id4a34324:Fastmail
Date: Fri, 5 May 2023 13:34:31 -0700
From: Andres Freund <andres@anarazel.de>
To: MARK CALLAGHAN <mdcallag@gmail.com>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: benchmark results comparing versions 15.2 and 16
Message-ID: <20230505203431.aq6d2ffcdn3avfgh@awork3.anarazel.de>
References: <CAFbpF8N4bVFZ+eyWFj_+fZXZFSayk2kJVtZnouUVKGHudiDrSw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAFbpF8N4bVFZ+eyWFj_+fZXZFSayk2kJVtZnouUVKGHudiDrSw@mail.gmail.com>
Archived-At: <https://www.postgresql.org/message-id/20230505203431.aq6d2ffcdn3avfgh%40awork3.anarazel.de>
Precedence: bulk

Hi,

On 2023-05-05 10:45:12 -0700, MARK CALLAGHAN wrote:
> This is mostly a hobby project for me - my other hobby is removing
> invasive weeds.

Hah :)


> Summary of the results:
> * from r1 - insert-heavy (l.i0, l.i1) and create indexes (l.x) steps are
> ~2% slower in PG 16
> * from r2 - create index (l.x) step is ~4% slower in PG 16
> * from r3 - regressions are similar to r1
> * from r4, r5 and r6 - regressions are mostly worse than r1, r2, r3. Note
> r4, r5, r6 are the same workload as r1, r2, r3 except the database is
> cached by PG for r1, r2, r3 so the r4, r5 and r6 benchmarks will do much
> more copying from the OS page cache into the Postgres buffer pool.

One thing that's somewhat odd is that there's very marked changes in l.i0's
p99 latency for the four clients cases - but whether 15 or 16 are better
differs between the runs.

r2)
                                p99
20m.pg152_o3_native_lto.cx7     300
20m.pg16prebeta.cx7             23683

r3)
                                p99
20m.pg152_o3_native_lto.cx7     70245
20m.pg16prebeta.cx7             8191

r5)
                                p99
20m.pg152_o3_native_lto.cx7     11188
20m.pg16prebeta.cx7             72720

r6)
                                p99
20m.pg152_o3_native_lto.cx7     1898
20m.pg16prebeta.cx7             31666


I do wonder if there's something getting scheduled in some of these runs
increasing latency?

Or what we're seeing depends on the time between the start of the server and
the start of the benchmark? It is interesting that the per-second throughput
graph shows a lot of up/down at the end:
https://mdcallag.github.io/reports/23_05_04_ibench.beelink.pg16b.4u.1tyes.1g/tput.l.i0.html

Both 15 and 16 have one very high result at 70s, 15 then has one low, but 16
has two low results.

Greetings,

Andres Freund