From: Tom Lane <tgl@sss.pgh.pa.us>
To: =?utf-8?B?6YKx5a6H6Iiq?= <iamqyh@gmail.com>
cc: "Anton A. Melnikov" <a.melnikov@postgrespro.ru>,
        Andres Freund <andres@anarazel.de>,
        "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>,
        David Rowley <dgrowleyml@gmail.com>,
        Michael Paquier <michael@paquier.xyz>
Subject: Re: Some performance degradation in REL_16 vs REL_15
In-reply-to: <7C8BB9F5-FCB9-4422-8E95-BD39781E4463@gmail.com>
References: <79e2f903-8bb0-4fc9-840c-f30d878d2327@postgrespro.ru> <20231013020522.ljwsw7mkpf23dylq@awork3.anarazel.de> <c1004775-977e-4645-aeef-1e37843d2f2b@postgrespro.ru> <7C8BB9F5-FCB9-4422-8E95-BD39781E4463@gmail.com>
Comments: In-reply-to =?utf-8?B?6YKx5a6H6Iiq?= <iamqyh@gmail.com>
	message dated "Wed, 18 Oct 2023 11:45:55 +0800"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <3040239.1697602456.1@sss.pgh.pa.us>
Content-Transfer-Encoding: quoted-printable
Date: Wed, 18 Oct 2023 00:14:16 -0400
Message-ID: <3040240.1697602456@sss.pgh.pa.us>
Archived-At: <https://www.postgresql.org/message-id/3040240.1697602456%40sss.pgh.pa.us>
Precedence: bulk

=3D?utf-8?B?6YKx5a6H6Iiq?=3D <iamqyh@gmail.com> writes:
> I wrote a script and test on branch REL_[10-16]_STABLE, and do see perfo=
rmance drop in REL_13_STABLE, which is about 1~2%.

I'm really skeptical that we should pay much attention to these numbers.
You've made several of the mistakes that we typically tell people not to
make when using pgbench:

* scale <=3D number of sessions means you're measuring a lot of
row-update contention

* once you crank up the scale enough to avoid that problem, running
with the default shared_buffers seems like a pretty poor choice

* 10-second runtime is probably an order of magnitude too small
to get useful, reliable numbers

On top of all that, discrepancies on the order of a percent or two
commonly arise from hard-to-control-for effects like the cache
alignment of hot spots in different parts of the code.  That means
that you can see changes of that size from nothing more than
day-to-day changes in completely unrelated parts of the code.

I'd get excited about say a 10% performance drop, because that's
probably more than noise; but I'm not convinced that any of the
differences you show here are more than noise.

			regards, tom lane