pg.ddx.io psycopg/psycopg GitHub issues and pull requests
help / color / mirror / Atom feed[psycopg/psycopg] issue #1502: AsyncConnectionPool: a cancel that reaches a worker mid-connect is swallowed, so asyncio.run() never returns if the pool was not closed
2+ messages / 2 participants
[nested] [flat]
* [psycopg/psycopg] issue #1502: AsyncConnectionPool: a cancel that reaches a worker mid-connect is swallowed, so asyncio.run() never returns if the pool was not closed
@ 2026-11-19 02:57 "LeaderOFGnomes (@LeaderOFGnomes)" <noreply+LeaderOFGnomes@github.com>
0 siblings, 0 replies; 2+ messages in thread
From: LeaderOFGnomes (@LeaderOFGnomes) @ 2026-11-19 02:57 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
**Versions:** psycopg 3.3.9, psycopg-pool 3.3.3, Python 3.12.16, Linux. The same code is on master (`psycopg_pool/pool_async.py`).
**What happens**
If an exception escapes the main coroutine of `asyncio.run()` while an `AsyncConnectionPool` is still open and one of its workers is connecting to a server that accepted the TCP connection but has not answered yet, `asyncio.run()` never returns. The process stays alive, and the original exception is never raised or printed.
**Reproduction**
```python
import asyncio
import logging
import socket
import threading
from psycopg_pool import AsyncConnectionPool
logging.basicConfig(level=logging.WARNING)
def silent_server() -> int:
"""A TCP port that accepts connections and never answers (a server that hangs)."""
sock = socket.socket()
sock.bind(("127.0.0.1", 0))
sock.listen()
held = []
def accept_forever():
while True:
held.append(sock.accept()[0])
threading.Thread(target=accept_forever, daemon=True).start()
return sock.getsockname()[1]
async def main():
port = silent_server()
pool = AsyncConnectionPool(f"host=127.0.0.1 port={port} dbname=x user=x", min_size=1, open=False)
await pool.open() # wait=False: a worker starts connecting in the background
await asyncio.sleep(0.1)
raise RuntimeError("startup failed elsewhere, before pool.close()")
asyncio.run(main()) # expected: RuntimeError; actual: never returns
```
Output, then nothing (the process has to be killed):
```text
WARNING:psycopg.pool:error connecting in 'pool-1':
```
**What we think happens**
`asyncio.run()` cancels the pending tasks and waits for them (`asyncio.runners._cancel_all_tasks`). The worker is in `_add_connection()`, awaiting `self._connect()`. In `pool_async.py`, `CLIENT_EXCEPTIONS = (Exception, asyncio.CancelledError)`, so `except CLIENT_EXCEPTIONS as ex:` catches the `CancelledError` and logs it as a connection error. That is why the message after the colon is empty: `str(CancelledError())` is `''`. A retry is scheduled, and the worker goes back to `await q.get()` in `worker()`. The task's only cancellation has been used up, so the `gather()` in `_cancel_all_tasks` waits forever.
A stack dump at that point shows the main thread in `runners._cancel_all_tasks`. The task `pool-1-worker-0` is in state "cancelling" at `pool_async.py:679` (`if isinstance((task := (await q.get())), StopWorker)`).
**Expected**
A worker task that is cancelled ends: it re-raises `CancelledError`, or at least does not go back to its loop. Then `asyncio.run()` can finish and raise the original exception.
**Workaround**
Close the pool on every exit path (`try/finally: await pool.close()`, or `async with`). After `close()` has queued a `StopWorker`, the worker ends once it is back at `q.get()`. `open(wait=True, timeout=...)` also helps, because `wait()` closes the pool on timeout.
We understand why `CancelledError` is caught on the client side, where it keeps the pool consistent when a caller is cancelled. The issue is only that a worker's own cancellation never ends the worker.
^ permalink raw [nested|flat] 2+ messages in thread
* Re: [psycopg/psycopg] issue #1502: AsyncConnectionPool: a cancel that reaches a worker mid-connect is swallowed, so asyncio.run() never returns if the pool was not closed
@ 2026-11-19 03:30 "aron-intframe (@aron-intframe)" <noreply+aron-intframe@github.com>
0 siblings, 0 replies; 2+ messages in thread
From: aron-intframe (@aron-intframe) @ 2026-11-19 03:30 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
Your diagnosis is right, and there is a second place that needs the same change: `worker()` also catches `CLIENT_EXCEPTIONS` (which includes `asyncio.CancelledError`) around `task.run()`. So if `_add_connection()` re-raises, the worker swallows the cancellation one level up and goes back to `q.get()`.
I ran your script on psycopg 3.3.9 / psycopg-pool 3.3.3 (Python 3.12.3) and on master `bb75838`. I added `faulthandler.dump_traceback_later(5, exit=True)` so the hang prints a traceback instead of needing a kill:
```
$ python -I repro.py
WARNING:psycopg.pool:error connecting in 'pool-1':
Timeout (0:00:05)!
File "/usr/lib/python3.12/asyncio/runners.py", line 205 in _cancel_all_tasks
```
When only `_add_connection()` re-raises, it still hangs, and the log line moves to the worker:
```
WARNING:psycopg.pool:task run <AddConnection 'pool-1' at 0x7d66cb9e4140> failed: CancelledError:
Timeout (0:00:05)!
```
With both changes below:
```
$ python -I repro.py
asyncio.run() returned and raised: RuntimeError('startup failed elsewhere, before pool.close()')
```
Patch against master. Both changes sit inside `if True: # ASYNC` blocks, so the generated sync pool does not change (`tools/async_to_sync.py --check` passes):
```diff
inline patch 1: psycopg_pool/psycopg_pool/pool_async.py (+14/-0 lines) download inline-1.patch
--- a/psycopg_pool/psycopg_pool/pool_async.py
+++ b/psycopg_pool/psycopg_pool/pool_async.py
@@ -686,6 +686,11 @@ class AsyncConnectionPool(Generic[ACT], BasePool):
try:
await task.run()
except CLIENT_EXCEPTIONS as ex:
+ if True: # ASYNC
+ if isinstance(ex, asyncio.CancelledError):
+ # The worker itself is being cancelled (e.g. by
+ # asyncio.run() shutting down): terminate.
+ raise
logger.warning(
"task run %s failed: %s: %s", task, ex.__class__.__name__, ex
)
@@ -757,6 +762,15 @@ class AsyncConnectionPool(Generic[ACT], BasePool):
try:
conn = await self._connect()
except CLIENT_EXCEPTIONS as ex:
+ if True: # ASYNC
+ if isinstance(ex, asyncio.CancelledError):
+ # Cancelled while connecting: don't schedule a retry,
+ # let the cancellation reach the worker.
+ async with self._lock:
+ self._nconns -= 1
+ if growing and self._growing:
+ self._growing = False
+ raise
logger.warning("error connecting in %r: %s", self.name, ex)
if attempt.time_to_give_up(now):
logger.warning(
```
The `_nconns` / `_growing` lines do the same bookkeeping as the give-up branch. That keeps the counters right if the pool is still used after a worker has been cancelled.
`pytest tests/pool` against a local PostgreSQL 18.7 with the patch: `351 passed, 8 skipped`.
One path this does not cover: `_reset_connection()` also catches `CancelledError` around `rollback()` and the reset callback. A cancel that lands while a returned connection is being reset would leave the worker running in the same way. I have not built a repro for that path. On Python 3.11+, one check at the end of the worker loop should cover every place that swallows the cancel, because `Task.cancelling()` stays non-zero after the `CancelledError` is caught (I only exercised it on the connect path):
```python
if True: # ASYNC
if (t := asyncio.current_task()) and t.cancelling():
raise asyncio.CancelledError
```
That variant alone (without the other two changes) also makes your script return. It is not available on 3.10, which psycopg_pool still supports, which is why the patch above re-raises explicitly.
Until a fix ships, the workaround you listed is the right one: `async with AsyncConnectionPool(...)` or `try/finally: await pool.close()`.
^ permalink raw [nested|flat] 2+ messages in thread
end of thread, other threads:[~2026-11-19 03:30 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 02:57 [psycopg/psycopg] issue #1502: AsyncConnectionPool: a cancel that reaches a worker mid-connect is swallowed, so asyncio.run() never returns if the pool was not closed "LeaderOFGnomes (@LeaderOFGnomes)" <noreply+LeaderOFGnomes@github.com>
2026-11-19 03:30 Re: [psycopg/psycopg] issue #1502: AsyncConnectionPool: a cancel that reaches a worker mid-connect is swallowed, so asyncio.run() never returns if the pool was not closed "aron-intframe (@aron-intframe)" <noreply+aron-intframe@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox