Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rRqp6-009imy-AX for pgsql-bugs@arkaria.postgresql.org; Mon, 22 Jan 2024 09:42:32 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1rRqp4-002xki-JZ for pgsql-bugs@arkaria.postgresql.org; Mon, 22 Jan 2024 09:42:30 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rRqp4-002xk0-BI for pgsql-bugs@lists.postgresql.org; Mon, 22 Jan 2024 09:42:30 +0000 Received: from lahtoruutu.iki.fi ([185.185.170.37]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rRqp1-002lgi-Gt for pgsql-bugs@lists.postgresql.org; Mon, 22 Jan 2024 09:42:29 +0000 Received: from [192.168.1.115] (dsl-hkibng22-54f8db-125.dhcp.inet.fi [84.248.219.125]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: hlinnaka) by lahtoruutu.iki.fi (Postfix) with ESMTPSA id 4TJQHK6Bkyz49PyM; Mon, 22 Jan 2024 11:42:21 +0200 (EET) Message-ID: <053bc845-3cb5-445e-8ea4-00f6c678ee25@iki.fi> Date: Mon, 22 Jan 2024 11:42:20 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: MarkBufferDirty Assert held LW_EXCLUSIVE lock fail when ginFinishSplit Content-Language: en-US To: feichanghong , pgsql-bugs References: From: Heikki Linnakangas In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 22/01/2024 09:59, feichanghong wrote: > In the function ginFindLeafPage, if we encounter a page marked with > GIN_INCOMPLETE_SPLIT, we call ginFinishSplit to finish the incomplete > split and > remove the GIN_INCOMPLETE_SPLIT flag from that page. ginFinishSplit requires > that "On entry, stack->buffer is exclusively locked," as explained in its > comments. > > The function ginFindLeafPage determines the type of lock held by > ginTraverseLock: > leaf pages hold GIN_EXCLUSIVE, and internal page hold GIN_SHARE. If > ginFindLeafPage > traverses to an internal page with an incomplete split, it will only hold a > GIN_SHARE lock, which does not meet the requirements of ginFinishSplit. If a > crash occurs when an internal page is split, but no downlink is inserted > in the > parent page, this problem may occur. > > I injected the following code into the ginbtree.c file to trigger a crash > during the splitting of an internal page: > ``` > --- a/src/backend/access/gin/ginbtree.c > +++ b/src/backend/access/gin/ginbtree.c > @@ -621,6 +621,12 @@ ginPlaceToPage(GinBtree btree, GinBtreeStack *stack, >                  PageSetLSN(BufferGetPage(lbuffer), recptr); >              if (BufferIsValid(childbuf)) >                  PageSetLSN(childpage, recptr); > + > +            if (stack->parent != NULL && !GinPageIsLeaf(newlpage)) > +            { > +                XLogFlush(recptr); > +                elog(PANIC, "internal page split for block: %u", > stack->blkno); > +            } >          } >          END_CRIT_SECTION(); > ``` > > With the modifications above, the problem can be reproduced with the > following > steps: > ``` > alter system set autovacuum to off; > alter system set gin_pending_list_limit to 64; > select pg_reload_conf(); > create table t (a text); > create extension btree_gin; > create index on t USING gin (a); > -- PANIC crash > insert into t select generate_series(1, 100000); > > -- Assert fail > insert into t select 1; > ``` > > I reviewed all places where ginFinishSplit is called, and only in two > instances > in ginFindLeafPage might it be possible to not hold an exclusive lock on > stack->buffer. > > The patch I provided in the attachment has been verified to fix the issue > mentioned above. > However, it may not be perfect: passing GIN_EXCLUSIVE to ginStepRight might > affect performance. Do we need to add a parameter to ginStepRight, and > within > the function ginStepRight, elevate the lock level for an incomplete > split page? > > Looking forward to suggestions from developers! Thanks, I'll look into this. The fix seems fine at a quick glance, but I'll think about the performance aspect a bit more. I'm thinking of adding tests for this using the new fault-injection facility that Michael just merged in commit d86d20f0ba. For GIN, but also B-tree which has a similar incomplete-split mechanism. Another way to create a scenario with incomplete splits, which doesn't involve any crashes or errors, would be to perform PITR to just between the insert and the finish-split records. But the fault-injection seems easier. -- Heikki Linnakangas Neon (https://neon.tech)