From: Alexander Lakhin <exclusion@gmail.com>
To: Heikki Linnakangas <hlinnaka@iki.fi>
To: Andrew Bille <andrewbille@gmail.com>
To: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #18658: Assert in SerialAdd() due to race condition
Date: Sat, 19 Oct 2024 12:00:00 +0300
Message-ID: <ea48b857-4e07-dd43-375e-564e13f5bfb2@gmail.com> (raw)
In-Reply-To: <ff9d4350-a82d-4a3f-86e5-3948175a9a86@iki.fi>
References: <18658-7dab125ec688c70b@postgresql.org>
<CAJnzarxBO18Ha==H3hX28yz2zd7cVuasXv1wxyJ7ERd9DzJf=Q@mail.gmail.com>
<5634e3e1-6e82-b883-ac87-52d7e926a04d@gmail.com>
<ff9d4350-a82d-4a3f-86e5-3948175a9a86@iki.fi>
Hello Heikki,
18.10.2024 23:15, Heikki Linnakangas wrote:
>
> Thanks for the repro, Andrew & Alexander! I was able to reproduce this too. It reproduces very quickly with the script
> you provided, if you add this sleep to ReleasePredicateLocks():
>
> @@ -3654,6 +3667,8 @@ ReleasePredicateLocks(bool isCommit, bool isReadOnlySafe)
>
> LWLockRelease(SerializableFinishedListLock);
>
> + pg_usleep(1000);
> +
> if (needToClear)
> ClearOldPredicateLocks();
>
> I think the assertion is too strict. It is normal for tailXid to be invalid in this scenario. The condition is that an
> XID was added to the finished list, but the global xmin has already advanced past that XID. It gets cleared from the
> finished list by the ClearOldPredicateLocks() call, but another backend might call SummarizeOldestCommittedSxact()
> before that.
>
> The attached patch fixes it.
>
Thank you for your attention to this!
I also encountered another (more rare) failure with that script (initially
on REL_16_STABLE, but now I've reproduced this on master too), when it
fails due to ENOSPC. (I could reproduce the failure more or less reliably
by running that script with parallel -j4 using 4 different servers.)
With additional logging added (see attached), I see the following:
2024-10-19 07:34:48.254 UTC [3032898:1][client backend][48/278:0] LOG: !!!SerialAdd| xid: 19957,
serialControl->headPage: 4294967295, tailXid: 20491, SERIAL_ENTRIESPERPAGE: 1024, firstZeroPage: 20, targetPage: 19,
isNewPage: 1
2024-10-19 07:34:48.254 UTC [3032898:2][client backend][48/278:0] STATEMENT: INSERT INTO t VALUES(42);
2024-10-19 07:34:48.254 UTC [3032898:3][client backend][48/278:0] LOG: !!!SerialAdd: isNewPage, firstZeroPage: 20,
targetPage: 19
2024-10-19 07:34:48.254 UTC [3032898:4][client backend][48/278:0] STATEMENT: INSERT INTO t VALUES(42);
2024-10-19 07:35:05.105 UTC [3032898:5][client backend][48/278:0] ERROR: could not access status of transaction 0
2024-10-19 07:35:05.105 UTC [3032898:6][client backend][48/278:0] DETAIL: Could not write to file "pg_serial/11FB3" at
offset 8192: No space left on device.
That is, if SerialAdd() gets xid preceding tailXid and belonging to a
preceding page, the page zeroing loop just runs until ENOSPC.
Your proposed fix (adjusted for REL_16_STABLE) eliminates the issue for me.
Thank you!
Best regards,
Alexander
Attachments:
[text/x-patch] SerialAdd-debugging.patch (1.6K, ../ea48b857-4e07-dd43-375e-564e13f5bfb2@gmail.com/2-SerialAdd-debugging.patch)
download | inline diff:diff --git a/src/backend/storage/lmgr/predicate.c b/src/backend/storage/lmgr/predicate.cindex 1af41213b4..4f7c15978a 100644--- a/src/backend/storage/lmgr/predicate.c+++ b/src/backend/storage/lmgr/predicate.c@@ -871,12 +871,16 @@ SerialAdd(TransactionId xid, SerCommitSeqNo minConflictCommitSeqNo)
{
firstZeroPage = SerialPage(tailXid);
isNewPage = true;
+elog(LOG, "!!!SerialAdd| xid: %d, serialControl->headPage: %d, tailXid: %d, SERIAL_ENTRIESPERPAGE: %ld, firstZeroPage: %d, targetPage: %d, isNewPage: %d",+xid, serialControl->headPage, tailXid, SERIAL_ENTRIESPERPAGE, firstZeroPage, targetPage, isNewPage);
}
else
{
firstZeroPage = SerialNextPage(serialControl->headPage);
isNewPage = SerialPagePrecedesLogically(serialControl->headPage,
targetPage);
+elog(LOG, "!!!SerialAdd| xid: %d, serialControl->headPage: %d, firstZeroPage: %d, targetPage: %d, isNewPage: %d",+xid, serialControl->headPage, firstZeroPage, targetPage, isNewPage);
}
if (!TransactionIdIsValid(serialControl->headXid)
@@ -887,6 +891,7 @@ SerialAdd(TransactionId xid, SerCommitSeqNo minConflictCommitSeqNo)
if (isNewPage)
{
+elog(LOG, "!!!SerialAdd: isNewPage, firstZeroPage: %d, targetPage: %d", firstZeroPage, targetPage);
/* Initialize intervening pages. */
while (firstZeroPage != targetPage)
{
@@ -894,6 +899,7 @@ SerialAdd(TransactionId xid, SerCommitSeqNo minConflictCommitSeqNo)
firstZeroPage = SerialNextPage(firstZeroPage);
}
slotno = SimpleLruZeroPage(SerialSlruCtl, targetPage);
+elog(LOG, "!!!SerialAdd: after while, slotno: %d", slotno);
}
else
slotno = SimpleLruReadPage(SerialSlruCtl, targetPage, true, xid);
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: exclusion@gmail.com, hlinnaka@iki.fi, andrewbille@gmail.com, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #18658: Assert in SerialAdd() due to race condition
In-Reply-To: <ea48b857-4e07-dd43-375e-564e13f5bfb2@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox