Subject: Regression tests fail on OpenBSD due to low semmns value

From: Alexander Lakhin <[email-masked]>
Date: Mon, 16 Dec 2024 07:00:00 +0200

Hello hackers,

A recent buildfarm timeout failure on sawshark [1] made me wonder, what's
wrong with that animal — beside that failure, this animal (running on
OpenBSD 7.4) produced "too many clients" errors from time to time, e. g.,
[2], [3].

I deployed OpenBSD 7.4 locally and reproduced "too many clients" and that
hang as well. It turned out that OpenBSD has semmns as low as 60 (see [4])
and as a consequence, initdb sets max_connections = 20 for the regression
test database. (This can be helpful sometimes, see e.g., [5].) At the same
time, paralell_schedule contains groups of 20 tests, for instance:
# parallel group (20 tests):  select_into random delete select_having select_distinct_on case prepared_xacts namespace 
select_implicit union arrays portals transactions select_distinct subselect update join aggregates hash_index btree_index

Moreover, prepared_xacts performs "\c", and it adds one more connection
for a short time, according to postmaster.log:
2024-12-16 06:18:20.290 EET [regression][1563560:91][client backend] [pg_regress/prepared_xacts] LOG:  statement: rollback;
...
2024-12-16 06:18:20.290 EET [regression][1563561:2][client backend] [[unknown]] FATAL:  sorry, too many clients already
...
2024-12-16 06:18:20.291 EET [regression][1563560:95][client backend] [pg_regress/prepared_xacts] LOG:  disconnection: 
session time: 0:00:00.018 user=law database=regression host=[local]

sysctl kern.seminfo.semmns=120 makes the issue go away on this OS;
on the hand, "too many clients" failures can be reproduced on other OS,
with "max_connections=20" in TEMP_CONFIG.

As to the hang, it can be reproduced easily with:
TEMP_CONFIG containing
max_connections=2
superuser_reserved_connections=0

and parallel_schedule as simple as:
test: transactions prepared_xacts
test: transactions prepared_xacts

Running `TEMP_CONFIG=.../extra.config make -s check`, I can see:
# +++ regress check in src/test/regress +++
...
# parallel group (2 tests):  prepared_xacts transactions
not ok 1     + transactions                               56 ms
not ok 2     + prepared_xacts                             21 ms
# (test process exited with exit code 2)
# parallel group (2 tests):
### the test is hanging here ###

with one backend waiting inside:
#0  0x000070c41ed2a007 in epoll_wait (epfd=6, events=0x629f1ce529e8, maxevents=1, timeout=-1) at 
../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x0000629f1410d64a in WaitEventSetWaitBlock (set=0x629f1ce52980, cur_timeout=-1, occurred_events=0x7ffd4c4ffed0, 
nevents=1) at latch.c:1564
#2  0x0000629f1410d534 in WaitEventSetWait (set=0x629f1ce52980, timeout=-1, occurred_events=0x7ffd4c4ffed0, nevents=1, 
wait_event_info=134217779) at latch.c:1510
#3  0x0000629f1410c764 in WaitLatch (latch=0x70c41b86bc24, wakeEvents=33, timeout=0, wait_event_info=134217779) at 
latch.c:538
#4  0x0000629f1413d032 in ProcWaitForSignal (wait_event_info=134217779) at proc.c:1893
#5  0x0000629f14132eb9 in GetSafeSnapshot (origSnapshot=0x629f147ad360 <CurrentSnapshotData>) at predicate.c:1579
#6  0x0000629f14133261 in GetSerializableTransactionSnapshot (snapshot=0x629f147ad360 <CurrentSnapshotData>) at 
predicate.c:1695
#7  0x0000629f143afafe in GetTransactionSnapshot () at snapmgr.c:253
#8  0x0000629f1414a7b8 in exec_simple_query (query_string=0x629f1ce580f0 "SELECT * FROM writetest;") at postgres.c:1172
...

So GetSafeSnapshot() waits indefinitely for possibleUnsafeConflicts to
become empty (for other backend to remove itself from the list of possible conflicts
inside ReleasePredicateLocks()), but it doesn't happen.

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sawshark&dt=2024-12-11%2012%3A20%3A05
[2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sawshark&dt=2024-07-22%2001%3A20%3A22
[3] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=sawshark&dt=2024-11-25%2006%3A20%3A22
[4] https://man.openbsd.org/options
[5] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=73c9f91a1

Best regards,
Alexander

---

From: Thomas Munro <[email-masked]>
Date: Tue, 17 Dec 2024 16:11:37 +1300

On Mon, Dec 16, 2024 at 6:00 PM Alexander Lakhin <[email-masked]> wrote:
> It turned out that OpenBSD has semmns as low as 60 (see [4])

Whenever I run into this, or my Mac requires manual ipcrm to clean up
leaked SysV kernel junk, I rebase my patch for sema_kind = 'futex'.
Here it goes.  It could be updated to support NetBSD I believe, but I
didn't try as its futex stuff came out later.

Then I remember why I didn't go anywhere with it.  It triggers a
thought loop about flipping it all around: use futexes to implement
lwlocks directly in place, and get rid of semaphores completely, but
that involves a few rabbit holes and sub-projects.  From memory:
classic r/w lock implementation on futexes is tricky but doable in the
portability constraints, futex fallback implementation even works
surprisingly well but has fun memory map sub-problems, actually lwlock
is not really a classic r/w lock as it has sprouted extra funky APIs
that lead the intrepid rabbit-holer to design an entirely different
new concurrency primitive that is really wanted for those users, a
couple of other places use raw semaphores directly namely procarray.c
and clog.c and if you stare at those for long you will be overwhelmed
with a desire to rewrite them, EOVERFLOW.

---

From: Tom Lane <[email-masked]>
Date: Mon, 16 Dec 2024 23:29:01 -0500

> didn't try as its futex stuff came out later.

FWIW, I looked at a nearby NetBSD 10.0 machine.  It has
/usr/include/sys/futex.h, which includes this enticing comment:

/*
 * Definitions for the __futex(2) synchronization primitive.
 *
 * These definitions are intended to be ABI-compatible with the
 * Linux futex(2) system call.
 */

However, the complete lack of any user-level documentation makes
me misdoubt the extent of their commitment to this :-(

I have the same concern about depending on undocumented macOS
APIs.  Other than that, getting off of SysV semaphores would be
a nice thing to do.

			regards, tom lane