pg.ddx.io  psycopg/psycopg GitHub issues and pull requests  
help / color / mirror / Atom feed
[psycopg/psycopg] issue #1502: AsyncConnectionPool: a cancel that reaches a worker mid-connect is swallowed, so asyncio.run() never returns if the pool was not closed
2+ messages / 2 participants
[nested] [flat]

* [psycopg/psycopg] issue #1502: AsyncConnectionPool: a cancel that reaches a worker mid-connect is swallowed, so asyncio.run() never returns if the pool was not closed
@ 2026-11-19 02:57  "LeaderOFGnomes (@LeaderOFGnomes)" <noreply+LeaderOFGnomes@github.com>
  0 siblings, 0 replies; 2+ messages in thread

From: LeaderOFGnomes (@LeaderOFGnomes) @ 2026-11-19 02:57 UTC (permalink / raw)
  To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>

**Versions:** psycopg 3.3.9, psycopg-pool 3.3.3, Python 3.12.16, Linux. The same code is on master (`psycopg_pool/pool_async.py`).

**What happens**

If an exception escapes the main coroutine of `asyncio.run()` while an `AsyncConnectionPool` is still open and one of its workers is connecting to a server that accepted the TCP connection but has not answered yet, `asyncio.run()` never returns. The process stays alive, and the original exception is never raised or printed.

**Reproduction**

```python
import asyncio
import logging
import socket
import threading

from psycopg_pool import AsyncConnectionPool

logging.basicConfig(level=logging.WARNING)


def silent_server() -> int:
    """A TCP port that accepts connections and never answers (a server that hangs)."""
    sock = socket.socket()
    sock.bind(("127.0.0.1", 0))
    sock.listen()
    held = []

    def accept_forever():
        while True:
            held.append(sock.accept()[0])

    threading.Thread(target=accept_forever, daemon=True).start()
    return sock.getsockname()[1]


async def main():
    port = silent_server()
    pool = AsyncConnectionPool(f"host=127.0.0.1 port={port} dbname=x user=x", min_size=1, open=False)
    await pool.open()  # wait=False: a worker starts connecting in the background
    await asyncio.sleep(0.1)
    raise RuntimeError("startup failed elsewhere, before pool.close()")


asyncio.run(main())  # expected: RuntimeError; actual: never returns
```

Output, then nothing (the process has to be killed):

```text
WARNING:psycopg.pool:error connecting in 'pool-1': 
```

**What we think happens**

`asyncio.run()` cancels the pending tasks and waits for them (`asyncio.runners._cancel_all_tasks`). The worker is in `_add_connection()`, awaiting `self._connect()`. In `pool_async.py`, `CLIENT_EXCEPTIONS = (Exception, asyncio.CancelledError)`, so `except CLIENT_EXCEPTIONS as ex:` catches the `CancelledError` and logs it as a connection error. That is why the message after the colon is empty: `str(CancelledError())` is `''`. A retry is scheduled, and the worker goes back to `await q.get()` in `worker()`. The task's only cancellation has been used up, so the `gather()` in `_cancel_all_tasks` waits forever.

A stack dump at that point shows the main thread in `runners._cancel_all_tasks`. The task `pool-1-worker-0` is in state "cancelling" at `pool_async.py:679` (`if isinstance((task := (await q.get())), StopWorker)`).

**Expected**

A worker task that is cancelled ends: it re-raises `CancelledError`, or at least does not go back to its loop. Then `asyncio.run()` can finish and raise the original exception.

**Workaround**

Close the pool on every exit path (`try/finally: await pool.close()`, or `async with`). After `close()` has queued a `StopWorker`, the worker ends once it is back at `q.get()`. `open(wait=True, timeout=...)` also helps, because `wait()` closes the pool on timeout.

We understand why `CancelledError` is caught on the client side, where it keeps the pool consistent when a caller is cancelled. The issue is only that a worker's own cancellation never ends the worker.


^ permalink  raw  [nested|flat] 2+ messages in thread

* Re: [psycopg/psycopg] issue #1502: AsyncConnectionPool: a cancel that reaches a worker mid-connect is swallowed, so asyncio.run() never returns if the pool was not closed
@ 2026-11-19 03:30  "aron-intframe (@aron-intframe)" <noreply+aron-intframe@github.com>
  0 siblings, 0 replies; 2+ messages in thread

From: aron-intframe (@aron-intframe) @ 2026-11-19 03:30 UTC (permalink / raw)
  To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>

Your diagnosis is right, and there is a second place that needs the same change: `worker()` also catches `CLIENT_EXCEPTIONS` (which includes `asyncio.CancelledError`) around `task.run()`. So if `_add_connection()` re-raises, the worker swallows the cancellation one level up and goes back to `q.get()`.

I ran your script on psycopg 3.3.9 / psycopg-pool 3.3.3 (Python 3.12.3) and on master `bb75838`. I added `faulthandler.dump_traceback_later(5, exit=True)` so the hang prints a traceback instead of needing a kill:

```
$ python -I repro.py
WARNING:psycopg.pool:error connecting in 'pool-1': 
Timeout (0:00:05)!
  File "/usr/lib/python3.12/asyncio/runners.py", line 205 in _cancel_all_tasks
```

When only `_add_connection()` re-raises, it still hangs, and the log line moves to the worker:

```
WARNING:psycopg.pool:task run <AddConnection 'pool-1' at 0x7d66cb9e4140> failed: CancelledError: 
Timeout (0:00:05)!
```

With both changes below:

```
$ python -I repro.py
asyncio.run() returned and raised: RuntimeError('startup failed elsewhere, before pool.close()')
```

Patch against master. Both changes sit inside `if True:  # ASYNC` blocks, so the generated sync pool does not change (`tools/async_to_sync.py --check` passes):

```diff
inline patch 1: psycopg_pool/psycopg_pool/pool_async.py (+14/-0 lines) download inline-1.patch--- a/psycopg_pool/psycopg_pool/pool_async.py +++ b/psycopg_pool/psycopg_pool/pool_async.py @@ -686,6 +686,11 @@ class AsyncConnectionPool(Generic[ACT], BasePool): try: await task.run() except CLIENT_EXCEPTIONS as ex: + if True: # ASYNC + if isinstance(ex, asyncio.CancelledError): + # The worker itself is being cancelled (e.g. by + # asyncio.run() shutting down): terminate. + raise logger.warning( "task run %s failed: %s: %s", task, ex.__class__.__name__, ex ) @@ -757,6 +762,15 @@ class AsyncConnectionPool(Generic[ACT], BasePool): try: conn = await self._connect() except CLIENT_EXCEPTIONS as ex: + if True: # ASYNC + if isinstance(ex, asyncio.CancelledError): + # Cancelled while connecting: don't schedule a retry, + # let the cancellation reach the worker. + async with self._lock: + self._nconns -= 1 + if growing and self._growing: + self._growing = False + raise logger.warning("error connecting in %r: %s", self.name, ex) if attempt.time_to_give_up(now): logger.warning(
``` The `_nconns` / `_growing` lines do the same bookkeeping as the give-up branch. That keeps the counters right if the pool is still used after a worker has been cancelled. `pytest tests/pool` against a local PostgreSQL 18.7 with the patch: `351 passed, 8 skipped`. One path this does not cover: `_reset_connection()` also catches `CancelledError` around `rollback()` and the reset callback. A cancel that lands while a returned connection is being reset would leave the worker running in the same way. I have not built a repro for that path. On Python 3.11+, one check at the end of the worker loop should cover every place that swallows the cancel, because `Task.cancelling()` stays non-zero after the `CancelledError` is caught (I only exercised it on the connect path): ```python if True: # ASYNC if (t := asyncio.current_task()) and t.cancelling(): raise asyncio.CancelledError ``` That variant alone (without the other two changes) also makes your script return. It is not available on 3.10, which psycopg_pool still supports, which is why the patch above re-raises explicitly. Until a fix ships, the workaround you listed is the right one: `async with AsyncConnectionPool(...)` or `try/finally: await pool.close()`. ^ permalink raw [nested|flat] 2+ messages in thread


end of thread, other threads:[~2026-11-19 03:30 UTC | newest]

Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 02:57 [psycopg/psycopg] issue #1502: AsyncConnectionPool: a cancel that reaches a worker mid-connect is swallowed, so asyncio.run() never returns if the pool was not closed "LeaderOFGnomes (@LeaderOFGnomes)" <noreply+LeaderOFGnomes@github.com>
2026-11-19 03:30 Re: [psycopg/psycopg] issue #1502: AsyncConnectionPool: a cancel that reaches a worker mid-connect is swallowed, so asyncio.run() never returns if the pool was not closed "aron-intframe (@aron-intframe)" <noreply+aron-intframe@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox