Subject: Re: Track Oldest Initialized WAL Buffer Page

From: Heikki Linnakangas <[email-masked]>
Date: Mon, 3 Jul 2023 16:27:27 +0300

On 07/02/2023 16:00, Bharath Rupireddy wrote:
> Hi,
> 
> While working on [1], I was looking for a quick way to tell if a WAL
> record is present in the WAL buffers array without scanning but I
> couldn't find one.

/* The end-ptr of the page that contains the record */
expectedEndPtr += XLOG_BLCKSZ - recptr % XLOG_BLCKSZ;

/* get the buffer where the record is, if it's in WAL buffers at all */
idx = XLogRecPtrToBufIdx(recptr);

/* prevent the WAL buffer from being evicted while we look at it */
LWLockAcquire(WALBufMappingLock, LW_SHARED);

/* Check if the page we're interested in is in the buffer */
found = XLogCtl->xlblocks[idx] == expectedEndPtr;

LWLockRelease(WALBufMappingLock, LW_SHARED);

> Hence, I put up a patch that basically tracks the
> oldest initialized WAL buffer page, named OldestInitializedPage, in
> XLogCtl. With OldestInitializedPage, we can easily illustrate WAL
> buffers array properties:
> 
> 1) At any given point of time, pages in the WAL buffers array are
> sorted in an ascending order from OldestInitializedPage till
> InitializedUpTo. Note that we verify this property for assert-only
> builds, see IsXLogBuffersArraySorted() in the patch for more details.
> 
> 2) OldestInitializedPage is monotonically increasing (by virtue of how
> postgres generates WAL records), that is, its value never decreases.
> This property lets someone read its value without a lock. There's no
> problem even if its value is slightly stale i.e. concurrently being
> updated. One can still use it for finding if a given WAL record is
> available in WAL buffers. At worst, one might get false positives
> (i.e. OldestInitializedPage may tell that the WAL record is available
> in WAL buffers, but when one actually looks at it, it isn't really
> available). This is more efficient and performant than acquiring a
> lock for reading. Note that we may not need a lock to read
> OldestInitializedPage but we need to update it holding
> WALBufMappingLock.

You actually hint at the above solution here, so I'm confused. If you're 
OK with slightly stale results, you can skip the WALBufferMappingLock 
above too, and perform an atomic read of xlblocks[idx] instead.

> 3) One can start traversing WAL buffers from OldestInitializedPage
> till InitializedUpTo to list out all valid WAL records and stats, and
> expose them via SQL-callable functions to users, for instance, as
> pg_walinspect functions.
> 
> 4) WAL buffers array is inherently organized as a circular, sorted and
> rotated array with OldestInitializedPage as pivot/first element of the
> array with the property where LSN of previous buffer page (if valid)
> is greater than OldestInitializedPage and LSN of the next buffer page
> (if
> valid) is greater than OldestInitializedPage.

These properties are true, maybe we should document them explicitly in a 
comment. But I don't see the point of tracking OldestInitializedPage. It 
seems cheap enough that we could, if there's a need for it, but I don't 
see the need.

-- 
Heikki Linnakangas
Neon (https://neon.tech)

---

From: Bharath Rupireddy <[email-masked]>
Date: Wed, 18 Oct 2023 00:16:05 +0530

On Mon, Jul 3, 2023 at 6:57 PM Heikki Linnakangas <[email-masked]> wrote:

Thanks a lot for responding. Sorry for being late.

> LWLockRelease(WALBufMappingLock, LW_SHARED);

This is exactly what I'm doing in the 0001 patch here
https://www.postgresql.org/message-id/[email-masked].

My bad! I should have mentioned the requirement properly - I want to
avoid taking WALBufMappingLock to peek into wal_buffers to determine
if the WAL buffer page containing the required WAL record exists.

> above too, and perform an atomic read of xlblocks[idx] instead.

I get that and I see GetXLogBuffer first reading xlblocks without lock
and then to confirm it anyways takes the lock again in
AdvanceXLInsertBuffer.

     * However, we don't hold a lock while we read the value. If someone has
     * just initialized the page, it's possible that we get a "torn read" of
     * the XLogRecPtr if 64-bit fetches are not atomic on this platform. In
     * that case we will see a bogus value. That's ok, we'll grab the mapping
     * lock (in AdvanceXLInsertBuffer) and retry if we see anything else than
     * the page we're looking for. But it means that when we do this unlocked
     * read, we might see a value that appears to be ahead of the page we're
     * looking for. Don't PANIC on that, until we've verified the value while
     * holding the lock.
     */

The the 0001 patch at
https://www.postgresql.org/message-id/[email-masked]
reads the WAL buffer page with WALBufMappingLock. So, the patch can
avoid WALBufMappingLock and do something like [1]:

[1]
{
    idx = XLogRecPtrToBufIdx(ptr);
    expectedEndPtr = ptr;
    expectedEndPtr += XLOG_BLCKSZ - ptr % XLOG_BLCKSZ;

    /*
     * Do a stale read of xlblocks without WALBufMappingLock. All the callers
     * of this function are expected to read WAL that's already flushed to disk
     * from WAL buffers. If this stale read says the requested WAL buffer page
     * doesn't exist, it means that the WAL buffer page either is being or has
     * already been replaced for reuse. If this stale read says the requested
     * WAL buffer page exists, we then take WALBufMappingLock and re-read the
     * xlblocks to ensure the WAL buffer page really exists and nobody is
     * replacing it meanwhile.
     */
    endptr = XLogCtl->xlblocks[idx];

    /* Requested WAL isn't available in WAL buffers. */
    if (expectedEndPtr != endptr)
        break;

    /*
     * Requested WAL is available in WAL buffers, so recheck the existence
     * under the WALBufMappingLock and read if the page still exists, otherwise
     * return.
     */
    LWLockAcquire(WALBufMappingLock, LW_SHARED);

    endptr = XLogCtl->xlblocks[idx];

    /* Requested WAL isn't available in WAL buffers. */
    if (expectedEndPtr != endptr)
        break;

    /*
     * We found the WAL buffer page containing the given XLogRecPtr.
Get starting
     * address of the page and a pointer to the right location given
     * XLogRecPtr in that page.
     */
    page = XLogCtl->pages + idx * (Size) XLOG_BLCKSZ;
    data = page + ptr % XLOG_BLCKSZ;

    return data;
}

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com