Subject: Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

From: Ants Aasma <[email-masked]>
Date: Wed, 8 Nov 2023 11:17:23 +0200

On Sat, 4 Nov 2023 at 22:08, Andrey M. Borodin <[email-masked]> wrote:

> On 30 Oct 2023, at 09:20, Dilip Kumar <[email-masked]> wrote:
>
> changed the logic of SlruAdjustNSlots() in 0002, such that now it
> starts with the next power of 2 value of the configured slots and
> keeps doubling the number of banks until we reach the number of banks
> to the max SLRU_MAX_BANKS(128) and bank size is bigger than
> SLRU_MIN_BANK_SIZE (8).  By doing so, we will ensure we don't have too
> many banks
>
> There was nothing wrong with having too many banks. Until bank-wise locks
> and counters were added in later patchsets.
> Having hashtable to find SLRU page in the buffer IMV is too slow. Some
> comments on this approach can be found here [0].
> I'm OK with having HTAB for that if we are sure performance does not
> degrade significantly, but I really doubt this is the case.
> I even think SLRU buffers used HTAB in some ancient times, but I could not
> find commit when it was changed to linear search.
>
> Maybe we could decouple locks and counters from SLRU banks? Banks were
> meant to be small to exploit performance of local linear search. Lock
> partitions have to be bigger for sure.
>

Is there a particular reason why lock partitions need to be bigger? We have
one lock per buffer anyway, bankwise locks will increase the number of
locks < 10%.

I am working on trying out a SIMD based LRU mechanism that uses a 16 entry
bank. The data layout is:

struct CacheBank {
    int page_numbers[16];
    char access_age[16];
}

The first part uses up one cache line, and the second line has 48 bytes of
space left over that could fit a lwlock and page_status, page_dirty arrays.

Lookup + LRU maintenance has 20 instructions/14 cycle latency and the only
branch is for found/not found. Hoping to have a working prototype of SLRU
on top in the next couple of days.

Regards,
Ants Aasma

---

From: "Andrey M. Borodin" <[email-masked]>
Date: Wed, 8 Nov 2023 14:32:48 +0500

> On 8 Nov 2023, at 14:17, Ants Aasma <[email-masked]> wrote:
> 
> Is there a particular reason why lock partitions need to be bigger? We have one lock per buffer anyway, bankwise locks will increase the number of locks < 10%.

The problem was not attracting much attention for some years. So my reasoning was that solution should not have any costs at all. Initial patchset with banks did not add any memory footprint.

> On 8 Nov 2023, at 14:17, Ants Aasma <[email-masked]> wrote:
> 
> I am working on trying out a SIMD based LRU mechanism that uses a 16 entry bank. 

FWIW I tried to pack struct parts together to minimize cache lines touched, see step 3 in [0]. So far I could not prove any performance benefits of this approach. But maybe your implementation will be more efficient.

Thanks!

Best regards, Andrey Borodin.

[0] https://www.postgresql.org/message-id/[email-masked]

---

From: Alvaro Herrera <[email-masked]>
Date: Thu, 9 Nov 2023 12:25:13 +0100

IMO the whole area of SLRU buffering is in horrible shape and many users
are struggling with overall PG performance because of it.  An
improvement doesn't have to be perfect -- it just has to be much better
than the current situation, which should be easy enough.  We can
continue to improve later, using more scalable algorithms or ones that
allow us to raise the limits higher.

The only point on which we do not have full consensus yet is the need to
have one GUC per SLRU, and a lot of effort seems focused on trying to
fix the problem without adding so many GUCs (for example, using shared
buffers instead, or use a single "scaling" GUC).  I think that hinders
progress.  Let's just add multiple GUCs, and users can leave most of
them alone and only adjust the one with which they have a performance
problems; it's not going to be the same one for everybody.

-- 
Álvaro Herrera               48°01'N 7°57'E  —  https://www.EnterpriseDB.com/
"Sallah, I said NO camels! That's FIVE camels; can't you count?"
(Indiana Jones)

---

From: Dilip Kumar <[email-masked]>
Date: Fri, 10 Nov 2023 10:17:49 +0530

> allow us to raise the limits higher.

I agree with this.

> problems; it's not going to be the same one for everybody.

+1

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

---

From: Nathan Bossart <[email-masked]>
Date: Fri, 10 Nov 2023 10:43:32 -0600

+1

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com