pg.ddx.io  pgbouncer/pgbouncer GitHub issues and pull requests  
help / color / mirror / Atom feed
[pgbouncer/pgbouncer] PR #1667: test: fail tests when PgBouncer exits with an unexpected code
5+ messages / 2 participants
[nested] [flat]

* [pgbouncer/pgbouncer] PR #1667: test: fail tests when PgBouncer exits with an unexpected code
@ 2026-10-23 20:49  "eulerto (@eulerto)" <noreply+eulerto@github.com>
  0 siblings, 0 replies; 5+ messages in thread

From: eulerto (@eulerto) @ 2026-10-23 20:49 UTC (permalink / raw)
  To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com>

Bouncer.wait_for_exit() waited for the process but ignored its exit
code, so a non-zero exit at shutdown went unnoticed. This is the gap
behind the ThreadSanitizer job: TSan reports a detected race only via
its exit code (66) at process exit, so a race that occurs during
shutdown (after the test body has finished) would leave the job green.

Capture the exit code and fail if it is not 0 (excluding Windows). Every
stop path there goes through a signal PgBouncer catches and turns into
exit(0), or an admin SHUTDOWN that also exits 0, so a non-zero code
means PgBouncer crashed or a sanitizer flagged a problem at shutdown.
Windows is excluded because the harness stops PgBouncer with
terminate(), a hard kill that exits non-zero by design. The terminate()
does not provide an argument to pass custom exit code and it hardcodes
the exit code to 1.

This complements PR #1660. The halt_on_error option catches races
during normal operation (PgBouncer dies in the middle of a test), while
this check catches races that only surface at shutdown.

inline patch 1: test/utils.py (+20/-24 lines) download inline-1.patchdiff --git a/test/utils.py b/test/utils.py index 5f69c07f0b97..f78e1e21624a 100644 --- a/test/utils.py +++ b/test/utils.py @@ -1004,7 +1004,6 @@ def __init__( super().__init__("127.0.0.1", self.port_lock.port) self.process: subprocess.Popen | None = None - self.aprocess: asyncio.subprocess.Process | None = None config_dir.mkdir() self.config_dir = config_dir self.ini_path = self.config_dir / "test.ini" @@ -1095,17 +1094,14 @@ def version(self): return pgbouncer_version async def start(self): - # Due to using a SelectorEventLoop for support with psycopg we cannot - # use asyncio subprocesses, since that eventloop does not support them. - # We fall back to regular subprocesses. - if WINDOWS: - self.process = subprocess.Popen( # noqa: ASYNC220 - [*self.base_command(), "--quiet", self.ini_path], close_fds=True - ) - else: - self.aprocess = await asyncio.create_subprocess_exec( - *self.base_command(), "--quiet", str(self.ini_path), close_fds=True - ) + # Launch PgBouncer with a plain subprocess.Popen rather than an asyncio + # subprocess. On Windows the psycopg SelectorEventLoop does not support + # asyncio subprocesses at all. Popen reaps through os.waitpid in the + # subprocess module, independent of any event loop or Python version, + # so wait() always returns PgBouncer's real exit code. + self.process = subprocess.Popen( # noqa: ASYNC220 + [*self.base_command(), "--quiet", str(self.ini_path)], close_fds=True + ) await self.wait_until_running() async def wait_until_running(self): @@ -1139,19 +1135,24 @@ def aadmin(self, query, **kwargs): def running(self): if self.process: return self.process.poll() is None - if self.aprocess: - return self.aprocess.returncode is None return False async def wait_for_exit(self): + returncode = None if self.process is not None: self.process.communicate() - self.process.wait() - if self.aprocess is not None: - await self.aprocess.communicate() - await self.aprocess.wait() + returncode = self.process.wait() self.process = None - self.aprocess = None + + # Every stop path goes through a signal (SIGQUIT/SIGTERM/ SIGINT) that + # PgBouncer catches and turns into exit(0), or through an admin + # SHUTDOWN that also exits 0, so a non-zero exit code means PgBouncer + # crashed or a sanitizer reported a problem at shutdown (except on + # Windows). + if not WINDOWS and returncode not in (None, 0): + raise RuntimeError( + f"PgBouncer exited with unexpected exit code {returncode}" + ) async def stop(self): if not WINDOWS: @@ -1162,9 +1163,6 @@ async def stop(self): if self.process is not None: self.process.terminate() self.process.terminate() - if self.aprocess is not None: - self.aprocess.terminate() - self.aprocess.terminate() await self.wait_for_exit() @@ -1178,8 +1176,6 @@ async def restart(self): await self.start() def send_signal(self, sig): - if self.aprocess: - self.aprocess.send_signal(sig) if self.process: self.process.send_signal(sig)
^ permalink raw [nested|flat] 5+ messages in thread

* Re: [pgbouncer/pgbouncer] PR #1667: test: fail tests when PgBouncer exits with an unexpected code
@ 2026-10-23 20:49  "eulerto (@eulerto)" <noreply+eulerto@github.com>
  3 siblings, 0 replies; 5+ messages in thread

From: eulerto (@eulerto) @ 2026-10-23 20:49 UTC (permalink / raw)
  To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com>

@eulerto applied label "ci-full".

^ permalink  raw  [nested|flat] 5+ messages in thread

* Re: [pgbouncer/pgbouncer] PR #1667: test: fail tests when PgBouncer exits with an unexpected code
@ 2026-10-24 02:18  "eulerto (@eulerto)" <noreply+eulerto@github.com>
  3 siblings, 0 replies; 5+ messages in thread

From: eulerto (@eulerto) @ 2026-10-24 02:18 UTC (permalink / raw)
  To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com>

The CI failure is not a PgBouncer bug and not a real crash. It is a false positive in the test harness introduced by the current commit in this branch.

The `test_shutdown_wait_for_servers` and `test_shutdown_wait_for_clients` are the only tests where PgBouncer exits on its own (`SHUTDOWN WAIT_FOR_SERVERS` and `SHUTDOWN WAIT_FOR_CLIENTS`). In these cases, PgBouncer exits cleanly (`exit(0)`) once connections drain. The CI logs confirm a clean shutdown.

I cannot reproduce using Python 3.11 but if I use the same Python version (3.12) as CI, both tests reproduces the identical failure:

```
ERROR at teardown of test_shutdown_wait_for_servers
E   RuntimeError: PgBouncer exited with unexpected exit code 255
WARNING  asyncio:unix_events.py:960 child process pid 1543700 exit status already read: will report returncode 255
```

AFAICS The `PidfdChildWatcher` default on Python 3.12+ then cannot read the status in `wait()` and fabricates return code 255 ("exit status already read"), which the new check flags as a crash. This is why those two tests fail on CI (Python 3.12) but not on 3.11, whose `ThreadedChildWatcher` reaps in a loop-independent thread.

One possible fix is if it forces `ThreadedChildWatcher` in the test harness. It reaps in a loop-independent thread and reports the true exit code, so clean shutdowns read 0 and a genuine TSan race still reads 66. The problem is `get_child_watcher` and `set_child_watcher` (functions used to accomplish the goal) are deprecated in Python 3.12 and removed in Python 3.14.

I checked what the test harness relies on from the asyncio subprocess. It seems nothing actually uses the asyncio-ness of that process. Every `self.aprocess` reference: start() (creates it), running() (reads return code), wait_for_exit() (communicate() + wait()), stop() (terminate() -- Windows), send_signal(). There is no async stdout and/or stderr streaming. The `create_subprocess_exec` (utils.py) doesn't even pass PIPE (logs) so communicate() is a no-op. The conclusion is that the process is launched with `await` purely because `start()` happens to be async def. Am I missing something?

We are already using a plain `subprocess.Popen` on Windows. It reaps through `os.waitpid` inside subprocess, independent of any loop, watcher or Python version. Its `.wait()` returns the real exit code. Besides that, we gain in simplification since `self.aprocess` is gone.

The next commit implements `subprocess.Popen` on all platforms instead of an asyncio subprocess on Linux.

^ permalink  raw  [nested|flat] 5+ messages in thread

* Re: [pgbouncer/pgbouncer] PR #1667: test: fail tests when PgBouncer exits with an unexpected code
@ 2026-10-30 16:57  "eulerto (@eulerto)" <noreply+eulerto@github.com>
  3 siblings, 0 replies; 5+ messages in thread

From: eulerto (@eulerto) @ 2026-10-30 16:57 UTC (permalink / raw)
  To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com>

@eulerto set milestone to "1.27.0".

^ permalink  raw  [nested|flat] 5+ messages in thread

* Re: [pgbouncer/pgbouncer] PR #1667 (changes requested): test: fail tests when PgBouncer exits with an unexpected code
@ 2026-11-09 10:52  "petere (@petere)" <noreply+petere@github.com>
  3 siblings, 0 replies; 5+ messages in thread

From: petere (@petere) @ 2026-11-09 10:52 UTC (permalink / raw)
  To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com>

According to my research, the issue you address in the second commit is specific to Python 3.12.  Everything works if you use an older (<=3.11) or a newer (>=3.13) version.  This appears to be due to the fix in python/cpython#121126, which was not backported to Python 3.12.  So I don't think we should be doing the major restructuring in your second patch.  We just need to work around the issue for Python 3.12.  One possibility is that we do this in `test/util.py`:

```diff
     def send_signal(self, sig):
-        if self.aprocess:
-            self.aprocess.send_signal(sig)
+        if self.aprocess and self.aprocess.returncode is None:
+            # Don't use aprocess.send_signal(): on Python 3.12 it
+            # calls Popen.poll(), which reaps an exited child and
+            # makes asyncio report returncode 255 instead of the real
+            # exit code.
+            try:
+                os.kill(self.aprocess.pid, sig)
+            except ProcessLookupError:
+                pass
         if self.process:
             self.process.send_signal(sig)
```

I haven't fully studied this, but it appears to work.

(Commit 62bf613 updated the Ubuntu version for most of the CI jobs, so they don't use Python 3.12 anymore.  But if you use ci-full then you will hit the issue on Rocky 10 at least.)

^ permalink  raw  [nested|flat] 5+ messages in thread


end of thread, other threads:[~2026-11-09 10:52 UTC | newest]

Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-10-23 20:49 [pgbouncer/pgbouncer] PR #1667: test: fail tests when PgBouncer exits with an unexpected code "eulerto (@eulerto)" <noreply+eulerto@github.com>
2026-10-23 20:49 ` "eulerto (@eulerto)" <noreply+eulerto@github.com>
2026-10-24 02:18 ` "eulerto (@eulerto)" <noreply+eulerto@github.com>
2026-10-30 16:57 ` "eulerto (@eulerto)" <noreply+eulerto@github.com>
2026-11-09 10:52 ` Re: [pgbouncer/pgbouncer] PR #1667 (changes requested): test: fail tests when PgBouncer exits with an unexpected code "petere (@petere)" <noreply+petere@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox