Received: from malur.postgresql.org ([217.196.149.56])
	by arkaria.postgresql.org with esmtps  (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.94.2)
	(envelope-from <pgsql-hackers-owner+archive@lists.postgresql.org>)
	id 1tds9x-00AFYT-9g
	for pgsql-hackers@arkaria.postgresql.org; Fri, 31 Jan 2025 14:38:18 +0000
Received: from localhost ([127.0.0.1] helo=malur.postgresql.org)
	by malur.postgresql.org with esmtp (Exim 4.94.2)
	(envelope-from <pgsql-hackers-owner+archive@lists.postgresql.org>)
	id 1tds9w-000ug4-38
	for pgsql-hackers@arkaria.postgresql.org; Fri, 31 Jan 2025 14:38:16 +0000
Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29])
	by malur.postgresql.org with esmtps  (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.94.2)
	(envelope-from <y.sokolov@postgrespro.ru>)
	id 1tds9v-000ufv-KE
	for pgsql-hackers@lists.postgresql.org; Fri, 31 Jan 2025 14:38:15 +0000
Received: from mail.postgrespro.ru ([93.174.131.139])
	by magus.postgresql.org with esmtps  (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.96)
	(envelope-from <y.sokolov@postgrespro.ru>)
	id 1tds9s-002XwM-0u
	for pgsql-hackers@lists.postgresql.org;
	Fri, 31 Jan 2025 14:38:14 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=postgrespro.ru;
	s=mx2023; t=1738334291;
	bh=j3+ol2oSxOSGggTpJ+K4R4a3e7v0TDoRs943Dducqb0=;
	h=Message-ID:Date:User-Agent:Subject:To:References:From:In-Reply-To:
	 From;
	b=HEKQMe6I0m1Th+yiCK33HTvNowo/YsPtOrBhj41q9x1rposO+mnkZv//l0akrtLO1
	 cvOT3Knq5EWg90iBisl4nNOR2wy6gjfFqMxNjFy9EIDJui2Qth1nrr5FxvDf4xyMSh
	 tWeh6wvdmKLO97O1qrdRZzpydRvbwGAEHvW7y/XKqYHrGRzqX1HqZbZ/TYKiwYM9Tk
	 TC8FmK7hbbH1TFCJTbDXhbYQvyIZ42GigzCnorilvhsvQ1Fa/xyQLk2BsqLp/hFyXU
	 /JhpWCkiDoyVJLmjHCyRVGnIUZdlh8Ckol6ldlmePc92jSiv7KL3xoMcuYmv07/HMS
	 UJCcpy2cZR2bQ==
Received: from [172.30.51.70] (unknown [172.30.51.70])
	(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
	(Client did not present a certificate)
	(Authenticated sender: y.sokolov@postgrespro.ru)
	by mail.postgrespro.ru (Postfix/465) with ESMTPSA id CD70460860
	for <pgsql-hackers@lists.postgresql.org>; Fri, 31 Jan 2025 17:38:10 +0300 (MSK)
Message-ID: <d353f0de-ac5a-4bfa-b1e9-69953bb00522@postgrespro.ru>
Date: Fri, 31 Jan 2025 17:38:10 +0300
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: hash_search_with_hash_value is high in "perf top" on a replica
To: pgsql-hackers@lists.postgresql.org
References: <202501311425.5cp3kqeeribm@alvherre.pgsql>
Content-Language: en-US
From: Yura Sokolov <y.sokolov@postgrespro.ru>
In-Reply-To: <202501311425.5cp3kqeeribm@alvherre.pgsql>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-KSMG-AntiPhishing: NotDetected, bases: 2025/01/31 13:28:00
X-KSMG-AntiSpam-Interceptor-Info: not scanned
X-KSMG-AntiSpam-Status: not scanned, disabled by settings
X-KSMG-AntiVirus: Kaspersky Secure Mail Gateway, version 2.1.0.7854, bases: 2025/01/31 13:26:00 #27170191
X-KSMG-AntiVirus-Status: NotDetected, skipped
X-KSMG-LinksScanning: not scanned, disabled by settings
X-KSMG-Message-Action: skipped
X-KSMG-Rule-ID: 1
List-Id: <pgsql-hackers.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-hackers@lists.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-hackers>
Archived-At: <https://www.postgresql.org/message-id/d353f0de-ac5a-4bfa-b1e9-69953bb00522%40postgrespro.ru>
Precedence: bulk

31.01.2025 17:25, Álvaro Herrera пишет:
> On 2025-Jan-31, Dmitry Koterov wrote:
> 
>> PG "startup recovering" eats up a lot of CPU (like 65 %user and 30 %sys),
>> which is a little surprising (what is it doing with all those CPU cycles?
>> it looked like WAL replay should be more IO bound than CPU bound?).
>>
>> Running "perf top -p <pid>", it shows this:
>>
>> Samples: 1M of event 'cycles:P', 4000 Hz, Event count (approx.):
>> 18178814660 lost: 0/0 drop: 0/0
>> Overhead  Shared Object     Symbol
>>   16.63%  postgres          [.] hash_search_with_hash_value
> 
> Yeah, I noticed that this function was showing high in some profiles a
> couple of days ago as well.  Looking now at hashfn.c (hash_bytes_uint32
> there is the function involved in the buffer mapping hash table), the
> comments state that we're updated to Bob Jenkins code from 2006, but
> there's a version in his website that (if I read correctly) is twice as
> fast as what we're using now:
> http://burtleburtle.net/bob/hash/spooky.html
> 
> Apparently this code in our repo is mostly unchanged since commit
> 1f559b7d3aa4, in 2007.
> 
> He mentions that on Intel chips, Google's CityHash is faster; but they
> in turn claim that the difference is small on Intel chips and that
> Jenkins' hash is better on AMD chips.
> https://github.com/google/cityhash
> 
> Anyway if you wanted to try your luck at improving things, here's your
> chance.
> 

`hash_search_with_hash_value` uses already calculated hash value, so its
performance doesn't depend on performance of hash function
(hash_bytes_uint32 or any other).

I believe, it is memory bound, since dynahash does at least three
indirection jumps before it event reaches first node in linked list

  segp = HTAB->dir[segment_num];
  *bucketptr = segp[segment_ndx];

And then iterates through linked list with each iteration is being (for
large hash table) both cache and TLB (without huge pages) miss.