public inbox for [email protected]  

help / color / mirror / Atom feed

From: Thom Brown <[email protected]>
To: Heikki Linnakangas <[email protected]>
Cc: Dhruv Aron <[email protected]>
Cc: [email protected]
Cc: [email protected]
Subject: Re: Restructured Shared Buffer Hash Table
Date: Tue, 7 Jul 2026 22:08:03 +0100
Message-ID: <CAA-aLv4jjFv_4=M0k0TSxkRzF9_b7CVS1eZq1YoOapWR7xFLPA@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <CAAStW3JX_7LcCD1U+89vyMmLreDs0XsBU-JDdFf5PD_Woy1pag@mail.gmail.com>
	<[email protected]>

On Tue, 7 Jul 2026 at 20:14, Heikki Linnakangas <[email protected]> wrote:
>
> On 07/07/2026 21:41, Dhruv Aron wrote:
> > At Databricks, we’ve found that the existing dynahash table structure is
> > leaving performance gains on the table when it comes to shared buffer
> > lookups: the multi-level structure (directory, segment, bucket chain,
> > freelist) appears excessive for the shared buffers and could be
> > simplified to boost performance and lower memory overhead. As such, we
> > are proposing a specialized hash table just for this purpose and would
> > appreciate feedback on this approach.
>
> Yeah, the dynahash has features that we just don't need in the buffer
> lookup table...
>
> The indirection with the directory is actually unnecessary for all the
> shared memory hash tables, since none of them can be resized. I've
> wondered if we should try to eliminate that from dynahash for all shmem
> hash tables. But the buffer lookup table is very performance-critical,
> so if there are any performance gains to be had, it's indeed probably
> worthwhile to have a separate implementation just for it.
>
> > bufmgr.c also changed slightly to prevent a race condition.
>
> Hmm, we're now holding the buffer header lock much longer than before,
> in InvalidateBuffer(). It's a spinlock, it really should not be held for
> more than a few instructions. BufTableDelete() is very fast in the new
> implementation, but still. Could we perhaps do some of
> BufTableDelete()'s work ahead of time, before we acquire the buffer
> header lock? Or maybe it's not a problem, in which case some kind of a
> worst case scenario benchmark to show that would be nice. Maybe test how
> it behaves when you have a lot of hash collisions, I think that'd make
> BufTableDelete() more expensive.
>
> > My testing
> > (helper script also attached) indicates that all three standard hash
> > table operations (insert, lookup, and delete) generally execute
> > significantly faster than the existing PG18 dynahash counterparts:
>
> Nice!
>
> I wonder how big the impact is with real world workloads. I've certainly
> seen the buffer table lookups consume a fair share of CPU time, so I'd
> assume that it shows up.
>
>
> Another data point:
>
> unpatched master, with shared_buffers='128 MB':
>
> postgres=# select * from pg_shmem_allocations where name like 'Shared
> Buffer%' order by name ;
>              name            |    off    |  size  | allocated_size
> ----------------------------+-----------+--------+----------------
>   Shared Buffer Lookup Table | 141607040 | 926000 |         926108
> (1 row)
>
> With this patch:
>
> postgres=# select * from pg_shmem_allocations where name like 'Shared
> Buffer%' order by name ;
>               name             |    off    |  size  | allocated_size
> ------------------------------+-----------+--------+----------------
>   Shared Buffer Lookup Buckets | 141607040 |  65536 |          65644
>   Shared Buffer Lookup Entries | 141672576 | 393216 |         393216
> (2 rows)
>
> So the new hash table takes much less memory. That's nice because you
> can then fit more in CPU caches.

Where did this thread come from? I can't see any conversation beyond
this single email, and as such, can't see any patch either.

Thom

view thread (5+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected]
  Subject: Re: Restructured Shared Buffer Hash Table
  In-Reply-To: <CAA-aLv4jjFv_4=M0k0TSxkRzF9_b7CVS1eZq1YoOapWR7xFLPA@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox