public inbox for [email protected]
help / color / mirror / Atom feedFrom: Ashutosh Bapat <[email protected]>
To: Heikki Linnakangas <[email protected]>
Cc: Maxim Orlov <[email protected]>
Cc: Alexander Korotkov <[email protected]>
Cc: Alvaro Herrera <[email protected]>
Cc: wenhui qiu <[email protected]>
Cc: Postgres hackers <[email protected]>
Subject: Re: POC: make mxidoff 64 bits
Date: Thu, 4 Dec 2025 16:06:51 +0530
Message-ID: <CAExHW5s9Bq5F3WQ6rPbLhW7vQV9NWWdz27TF2hqEg+5Fv3zaLg@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <CACG=ezaWg7_nt-8ey4aKv2w9LcuLthHknwCawmBgEeTnJrJTcw@mail.gmail.com>
<CACG=ezbs912S58=uR17b4w8uuWv1=OcCRaTW_OWdFm4+tXZA6w@mail.gmail.com>
<CAGjGUA+BfcWyccNN4=tHsW_E-koRxbg8h8ut6hjvPsHMgmek6w@mail.gmail.com>
<CACG=ezYbYO_KHWdeDedbDcY0tOS0JfaqBxG3=bG5+DdsDK4MpQ@mail.gmail.com>
<CACG=ezYpZRPwoRCz_h3Qerd3XJNdpTHCpwGbZphNdy26tA4_qQ@mail.gmail.com>
<[email protected]>
<[email protected]>
<CACG=ezYUJSvnuxntkURNWo_1vZ+AtmcQfqd_h6WgDzGaudfw+Q@mail.gmail.com>
<[email protected]>
<[email protected]>
<CAExHW5tUEkiQrvm9hgccjKUNkWBnJ5_HDUrAwiHBTxu+Vuj29Q@mail.gmail.com>
<[email protected]>
<[email protected]>
<CAPpHfdsMd9TQ8ZbMnStMfFvBoBMwBCSC+8zcOYY9VDMYQzUYyQ@mail.gmail.com>
<[email protected]>
<CACG=ezY-uzrQvUbm4_txT6hR4r78cgLoXGpT61+OXGodLw=qZA@mail.gmail.com>
<[email protected]>
On Wed, Dec 3, 2025 at 5:34 PM Heikki Linnakangas <[email protected]> wrote:
>
> On 03/12/2025 11:54, Maxim Orlov wrote:
> > The biggest problem with compression, in my opinion, is that losing
> > even one byte causes the loss of the entire compressed block in the
> > worst case scenario. After all, we still don't have checksums for the
> > SLRU's, which is a shame by itself.
> >
> > Again, I'm not against the idea of compression, but the risks need to
> > be considered.
>
> There are plenty of such critical bytes in the system where a single bit
> flip renders the whole block unreadable. Actually, if we had checksums
> on SLRU pages, a single bit flip anywhere in the page would make the
> checksum fail and render the block unreadable.
>
> If things go really bad and you need to open a hex editor and try to fix
> the data manually, it shouldn't be too hard to deduce the correct base
> offset from surrounding data.
>
> > As a software developer, I definitely want to implement compression and
> > save a few gigabytes. However, given my previous experience using
> > Postgres in real-world applications, reliability at the cost of several
> > gigabytes would not have caused me any trouble. Just saying.
>
> +1. If we decide to do some kind of compression here, I want it to be
> very simple. Otherwise it's just not worth the code complexity and risk.
>
> Let's do the math of how much disk space we'd save. Let's assume the
> worst case that every multixid consists of only one transaction ID.
> Currently, every such multixid takes up 4 bytes in the offsets SLRU, and
> 5 bytes in the members SLRU (one flag byte and 4 bytes for the XID). So
> that's 9 bytes. With 64-bit offsets, it becomes 13 bytes. With the
> compression, we're back to 9 bytes again (ignoring the one base offset
> per page). So in an extreme case that you have 1 billion multixids, with
> only one XID per multixid, the difference is between 9 GB and 13 GB.
> That seems acceptable.
>
> And having just one XID per multixid is a rare corner case. Much more
> commonly, you have at at least two XIDs. With two XIDs per multixid, the
> difference is between 14 bytes and 18 bytes.
>
> And having a billion multixids is pretty extreme. Your database is
> likely very large too if you reach that point, and a few gigabytes won't
> matter.
I am in favour of keeping things simpler than using a complex compression.
>
> One could argue that the memory needed for the SLRU cache matters more
> than the disk space. That's perhaps true, but I think this is totally
> acceptable from that point of view, too.
This brings an interesting point. Since the offsets are twice large,
SLRU will contain half the entries than earlier. Have we measured
performance impact of this? Do we need to provide some guidance about
increasing the SLRU size or increase the default SLRU size?
--
Best Wishes,
Ashutosh Bapat
view thread (79+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: POC: make mxidoff 64 bits
In-Reply-To: <CAExHW5s9Bq5F3WQ6rPbLhW7vQV9NWWdz27TF2hqEg+5Fv3zaLg@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox