Message-ID: <94fc0a8b-5d18-4a82-8082-b76a0e725788@wi3ck.info>
Date: Tue, 12 Nov 2024 11:40:39 -0500
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Commit Timestamp and LSN Inversion issue
To: Andres Freund <andres@anarazel.de>
Cc: Amit Kapila <amit.kapila16@gmail.com>, Tomas Vondra <tomas@vondra.me>,
 "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>,
 Aleksander Alekseev <aleksander@timescale.com>,
 pgsql-hackers <pgsql-hackers@postgresql.org>,
 shveta malik <shveta.malik@gmail.com>, Jan Wieck <jan@wi3ck.info>
References: <gkbrjxwyjbqckjbrkep3uy2mvrcq2mrfzprfmegbeiqxidtmbu@3cgs5iocjykk>
 <OS0PR01MB57162A227EC357482FEB4470945D2@OS0PR01MB5716.jpnprd01.prod.outlook.com>
 <xhcapr7lur35itqrbk4qdqn7rzfaqr3g6dvyxzfx6jeyknhvbv@h3wkgixuuuon>
 <CAA4eK1+ZdL=sC4sdUOQz+X9=7Z8Uxhkd0VqwcB9rgzXjvu=6yA@mail.gmail.com>
 <11ee3489-40ba-45d6-ba6a-7c3def725e84@vondra.me>
 <CAA4eK1KQ6H4u5=_5Xq44HhX3RFXW4HW-z0F4T4J1nf2xVCo04w@mail.gmail.com>
 <CAA4eK1L6Vpf9EmL2bpfD3kdvM6_2XQqwJ9GY6uY-hgcot76cqA@mail.gmail.com>
 <00f7cfb2-e3a6-4bd7-b19c-fd75d3838049@wi3ck.info>
 <lyydpy6xly7gefcswkimur733wcbke4bdjwjedvluyo2ebfdr7@xz5k4mbjgivm>
 <296b2c56-c688-41a0-a752-b242268691fd@wi3ck.info>
 <wpitjnzyno7pmhnaudsehw5v6lw644cz52wx4qqd2wvfazlxgq@gsogjmyhn3xw>
Content-Language: en-US
From: Jan Wieck <jan@wi3ck.info>
In-Reply-To: <wpitjnzyno7pmhnaudsehw5v6lw644cz52wx4qqd2wvfazlxgq@gsogjmyhn3xw>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: <https://www.postgresql.org/message-id/94fc0a8b-5d18-4a82-8082-b76a0e725788%40wi3ck.info>
Precedence: bulk

Hello,

On 11/12/24 10:34, Andres Freund wrote:
> I have working code - pretty ugly at this state, but mostly needs a fair bit
> of elbow grease not divine inspiration...  It's not a trivial change, but
> entirely doable.
> 
> The short summary of how it works is that it uses a single 64bit atomic that
> is internally subdivided into a ringbuffer position in N high bits and an
> offset from a base LSN in the remaining bits.  The insertion sequence is
> 
> ...
> 
> This leaves you with a single xadd to contended cacheline as the contention
> point (scales far better than cmpxchg and far far better than
> cmpxchg16b). There's a bit of contention for the ringbuffer[].oldpos being set
> and read, but it's only by two backends, not all of them.

That sounds rather promising.

Would it be reasonable to have both implementations available at least 
at compile time, if not at runtime? Is it possible that we need to do 
that anyway for some time or are those atomic operations available on 
all supported CPU architectures?


> 
> The nice part is this scheme leaves you with a ringbuffer that's ordered by
> the insertion-lsn. Which allows to make WaitXLogInsertionsToFinish() far more
> efficient and to get rid of NUM_XLOGINSERT_LOCKS (by removing WAL insertion
> locks). Right now NUM_XLOGINSERT_LOCKS is a major scalability limit - but at
> the same time increasing it makes the contention on the spinlock *much* worse,
> leading to slowdowns in other workloads.

Yeah, that is a complex wart that I believe was the answer to the NUMA 
overload that Kevin Grittner and myself discovered many years ago, where 
on a 4-socket machine the cacheline stealing would get so bad that 
whoever was holding the lock could not release it.

In any case, thanks for the input. Looks like in the long run we need to 
come up with a different way to solve the inversion problem.


Best Regards, Jan