Subject: Re: Adding skip scan (including MDAM style range skip scan) to nbtree

From: Tomas Vondra <[email-masked]>
Date: Sat, 7 Sep 2024 17:27:05 +0200

Hi,

I started looking at this patch today. The first thing I usually do for
new patches is a stress test, so I did a simple script that generates
random table and runs a random query with IN() clause with various
configs (parallel query, index-only scans, ...). And it got stuck on a
parallel query pretty quick.

I've seen a bunch of those cases, so it's not a particularly unlikely
issue. The backtraces look pretty much the same in all cases - the
processes are stuck either waiting on the conditional variable in
_bt_parallel_seize, or trying to send data in shm_mq_send_bytes.

Attached is the script I use for stress testing (pretty dumb, just a
bunch of loops generating tables + queries), and backtraces for two
lockups (one is EXPLAIN ANALYZE, but otherwise exactly the same).

I haven't investigated why this is happening, but I wonder if this might
be similar to the parallel hashjoin issues, with trying to send data,
but the receiver being unable to proceed and effectively working on the
sender. But that's just a wild guess.

regards

-- 
Tomas Vondra

---

From: Peter Geoghegan <[email-masked]>
Date: Mon, 9 Sep 2024 16:54:57 -0400

On Sat, Sep 7, 2024 at 11:27 AM Tomas Vondra <[email-masked]> wrote:
> I started looking at this patch today.

Thanks for taking a look!

> The first thing I usually do for

I can reproduce this locally, without too much difficulty.
Unfortunately, this is a bug on master/Postgres 17. Some kind of issue
in my commit 5bf748b8.

The timing of this is slightly unfortunate. There's only a few weeks
until the release of 17, plus I have to travel for work over the next
week. I won't be back until the 16th, and will have limited
availability between then and now. I think that I'll have ample time
to debug and fix the issue ahead of the release of 17, though.

Looks like the problem is a parallel index scan with SAOP array keys
can find itself in a state where every parallel worker waits for the
leader to finish off a scheduled primitive index scan, while the
leader itself waits for the scan's tuple queue to return more tuples.
Obviously, the query will effectively go to sleep indefinitely when
that happens (unless and until the DBA cancels the query). This is
only possible with just the right/wrong combination of array keys and
index cardinality.

I cannot recreate the problem with parallel_leader_participation=off,
which strongly suggests that leader participation is a factor. I'll
find time to study this in detail as soon as I can.

Further background: I was always aware of the leader's tendency to go
away forever shortly after the scan begins. That was supposed to be
safe, since we account for it by serializing the scan's current array
keys in shared memory, at the point a primitive index scan is
scheduled -- any backend should be able to pick up where any other
backend left off, no matter how primitive scans are scheduled. That
now doesn't seem to be completely robust, likely due to restrictions
on when and how other backends can pick up the scheduled work from
within _bt_first, at the point that it calls _bt_parallel_seize.

In short, one or two details of how backends call _bt_parallel_seize
to pick up BTPARALLEL_NEED_PRIMSCAN work likely need to be rethought.

--
Peter Geoghegan