pg.ddx.io psycopg/psycopg GitHub issues and pull requests
help / color / mirror / Atom feed[psycopg/psycopg] issue #1427: Hang in futex, linked to garbage collection
10+ messages / 3 participants
[nested] [flat]
* [psycopg/psycopg] issue #1427: Hang in futex, linked to garbage collection
@ 2026-09-26 18:10 "JulienPalard (@JulienPalard)" <noreply+JulienPalard@github.com>
0 siblings, 0 replies; 10+ messages in thread
From: JulienPalard (@JulienPalard) @ 2026-09-26 18:10 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
I'm running a Django under gunicorn (sync worker), psycopg 3.3.6, (with and without pooling, the traces on this issue are with no pooling).
Sometimes workers get stuck and killed by gunicorn:
```
Sep 26 17:48:36 gunicorn[3740958]: [2026-09-26 17:48:36 +0000] [3740958] [CRITICAL] WORKER TIMEOUT (pid:3740964)
Sep 26 17:48:36 gunicorn[3740964]: Exception ignored in: <generator object cursor_iter at 0x7fb60279f890>
Sep 26 17:48:36 gunicorn[3740964]: Traceback (most recent call last):
Sep 26 17:48:36 gunicorn[3740964]: File "/opt/venv/lib/python3.13/site-packages/django/db/models/sql/compiler.py", line 2115, in cursor_iter
Sep 26 17:48:36 gunicorn[3740964]: cursor.close()
Sep 26 17:48:36 gunicorn[3740964]: File "/opt/venv/lib/python3.13/site-packages/psycopg/_server_cursor.py", line 69, in close
Sep 26 17:48:36 gunicorn[3740964]: with self._conn.lock:
Sep 26 17:48:36 gunicorn[3740964]: File "/opt/venv/lib/python3.13/site-packages/gunicorn/workers/base.py", line 199, in handle_abort
Sep 26 17:48:36 gunicorn[3740964]: sys.exit(1)
Sep 26 17:48:36 gunicorn[3740964]: SystemExit: 1
```
My good old friend, `strace`, shows that the process gets stuck in a `futex`:
```
3745521 1790445242.184237 sendto(8, "Q\0\0\0CFETCH FORWARD 2000 FROM \"_django_curs_140419890026368_sync_10\"\0", 68, MSG_NOSIGNAL, NULL, 0) = 68
3745521 1790445242.184311 poll([{fd=8, events=POLLIN|POLLERR}], 1, 100) = 1 ([{fd=8, revents=POLLIN}])
3745521 1790445242.184599 recvfrom(8, "T\0\0\3<\0\34id\0\0\0@[...]\0\rFETCH 20\0Z\0\0\0\5T", 16384, 0, NULL, NULL) = 6055
3745521 1790445242.185241 futex(0x7ffce596d980, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
```
According to strace there's a clean 4 worker processes that have no threads (the first one is the "gunicorn manager", the 4 others are the sync workers):
```
strace: Process 3740958 attached with 2 threads
strace: Process 3740961 attached
strace: Process 3740962 attached
strace: Process 3740963 attached
strace: Process 3745521 attached
```
Used `faulthandler` to get a traceback, got:
```
Current thread 0x00007fb60daa1780 (most recent call first):
Garbage-collecting
File "/opt/[...]/site-packages/psycopg/_server_cursor.py", line 69 in close
File "/opt/[...]/site-packages/django/db/models/sql/compiler.py", line 2115 in cursor_iter
File "/opt/[...]/site-packages/psycopg/_server_cursor_base.py", line 183 in _fetch_gen
File "/opt/[...]/site-packages/psycopg/connection.py", line 495 in wait
File "/opt/[...]/site-packages/psycopg/_server_cursor.py", line 121 in fetchmany
File "/opt/[...]/site-packages/django/db/utils.py", line 98 in inner
File "/opt/[...]/site-packages/django/db/models/sql/compiler.py", line 2112 in <lambda>
File "/opt/[...]/site-packages/django/db/models/sql/compiler.py", line 2112 in cursor_iter
File "/opt/[...]/site-packages/django/db/models/query.py", line 125 in __iter__
File "/opt/[...]/site-packages/django/db/models/query.py", line 520 in _iterator
```
I'm kind of out of ideas to give more useful debugging infos here, as I have no reproducer, don’t hesitate if you have ideas that I can try.
I noticed that it does not happen on the same query every time, but it always seems to happen in a transaction, leaving a:
```
633992+00 | 2026-09-26 17:54:02.184291+00 | 2026-09-26 17:54:02.184516+00 | Client | ClientRead | idle in transaction | 241194 | 241194 | | FETCH FORWARD 2000 FROM "_django_curs_140419890026368_sync_10" | client backend
```
psql side. Or maybe that's just me noticing it only during transactions because it locks a table. But it does not systematically happen on a cursor like this time, it can happen on a SELECT.
Interestingly enough it happens **only** during a `PATCH /api/admin/products/` on [django-oscar](https://github.com/django-oscar/django-oscar/), but maybe it's just because a PATCH on a product on Django Oscar triggers hundreds of SELECT and hundreds of UPDATES, making them a magnet for this bug.
^ permalink raw [nested|flat] 10+ messages in thread
* Re: [psycopg/psycopg] issue #1427: Hang in futex, linked to garbage collection
@ 2026-09-28 11:08 ` "JulienPalard (@JulienPalard)" <noreply+JulienPalard@github.com>
8 siblings, 0 replies; 10+ messages in thread
From: JulienPalard (@JulienPalard) @ 2026-09-28 11:08 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
By commenting:
```
elif _psycopg and sys.platform != "win32" and not _is_select_patched():
wait = wait_c
```
the issue disappear, I bet it means the issue is in psycopyg-binary and not in psycopg.
^ permalink raw [nested|flat] 10+ messages in thread
* Re: [psycopg/psycopg] issue #1427: Hang in futex, linked to garbage collection
@ 2026-09-28 11:39 ` "JulienPalard (@JulienPalard)" <noreply+JulienPalard@github.com>
8 siblings, 0 replies; 10+ messages in thread
From: JulienPalard (@JulienPalard) @ 2026-09-28 11:39 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
As far as I understand now: during a psycopg.cursor.execute, while the lock is held, a GC triggers, leading to a call to _server_cursor.close, which tries to take the lock but can't. The GC pass never ends, deadlocking the thread.
Can easily be shown by using:
```
import gc
gc.set_debug(gc.DEBUG_STATS)`
```
showing this at the end of a locked process:
```
gc: collecting generation 0...
gc: objects in each generation: 2144 8269 343198
gc: objects in permanent generation: 0
```
A normal GC run ends with `gc: done, 35 unreachable, 0 uncollectable, 0.0010s elapsed`.
^ permalink raw [nested|flat] 10+ messages in thread
* Re: [psycopg/psycopg] issue #1427: Hang in futex, linked to garbage collection
@ 2026-09-30 22:39 ` "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
8 siblings, 0 replies; 10+ messages in thread
From: dvarrazzo (@dvarrazzo) @ 2026-09-30 22:39 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
If a GC triggers `ServerCursor.close()` then it means that the cursor is not managed with a context manager, right?
^ permalink raw [nested|flat] 10+ messages in thread
* Re: [psycopg/psycopg] issue #1427: Hang in futex, linked to garbage collection
@ 2026-10-01 11:35 ` "JulienPalard (@JulienPalard)" <noreply+JulienPalard@github.com>
8 siblings, 0 replies; 10+ messages in thread
From: JulienPalard (@JulienPalard) @ 2026-10-01 11:35 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
> If a GC triggers `ServerCursor.close()` then it means that the cursor is not managed with a context manager, right?
I have no proof yet, I bet I should try to use tracemalloc to find out, but what's you're writing looks right to me.
^ permalink raw [nested|flat] 10+ messages in thread
* Re: [psycopg/psycopg] issue #1427: Hang in futex, linked to garbage collection
@ 2026-10-03 08:02 ` "JulienPalard (@JulienPalard)" <noreply+JulienPalard@github.com>
8 siblings, 0 replies; 10+ messages in thread
From: JulienPalard (@JulienPalard) @ 2026-10-03 08:02 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
> If a GC triggers `ServerCursor.close()` then it means that the cursor is not managed with a context manager, right?
Used tracemalloc, you’re right it’s not managed by a context manager, it’s created from django.db.backends.postgresql.base:
```py
@async_unsafe
def create_cursor(self, name=None):
if name:
if is_psycopg3 and (
self.settings_dict["OPTIONS"].get("server_side_binding") is not True
):
# psycopg >= 3 forces the usage of server-side bindings for
# named cursors so a specialized class that implements
# server-side cursors while performing client-side bindings
# must be used if `server_side_binding` is disabled (default).
===>> cursor = ServerSideCursor(
self.connection,
name=name,
scrollable=False,
withhold=self.connection.autocommit,
)
else:
# In autocommit mode, the cursor will be used outside of a
# transaction, hence use a holdable cursor.
cursor = self.connection.cursor(
name, scrollable=False, withhold=self.connection.autocommit
)
else:
cursor = self.connection.cursor()
if is_psycopg3:
# Register the cursor timezone only if the connection disagrees, to
# avoid copying the adapter map.
tzloader = self.connection.adapters.get_loader(TIMESTAMPTZ_OID, Format.TEXT)
if self.timezone != tzloader.timezone:
register_tzloader(self.timezone, cursor)
else:
cursor.tzinfo_factory = self.tzinfo_factory if settings.USE_TZ else None
return cursor
```
The allocation traceback of the cursor is:
```text
.venv/lib/python3.13/site-packages/oscar/apps/catalogue/product_attributes.py:259
.venv/lib/python3.13/site-packages/oscar/apps/catalogue/product_attributes.py:225
.venv/lib/python3.13/site-packages/oscar/apps/catalogue/product_attributes.py:26
.venv/lib/python3.13/site-packages/django/db/models/query.py:520
.venv/lib/python3.13/site-packages/django/db/models/query.py:93
.venv/lib/python3.13/site-packages/django/db/models/sql/compiler.py:1570
.venv/lib/python3.13/site-packages/django/utils/asyncio.py:26
.venv/lib/python3.13/site-packages/django/db/backends/postgresql/base.py:462
.venv/lib/python3.13/site-packages/django/db/backends/base/base.py:297
.venv/lib/python3.13/site-packages/django/utils/asyncio.py:26
.venv/lib/python3.13/site-packages/django/db/backends/postgresql/base.py:416
```
with a Django 5.1.15. I've not dug how Django manage the lifetime of its cursors yet.
^ permalink raw [nested|flat] 10+ messages in thread
* Re: [psycopg/psycopg] issue #1427: Hang in futex, linked to garbage collection
@ 2026-10-03 16:56 ` "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
8 siblings, 0 replies; 10+ messages in thread
From: dvarrazzo (@dvarrazzo) @ 2026-10-03 16:56 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
Hmmm.... actually, I was thinking that `close()` was called by `ServerCursor.__del__()`, but it's not: `__del__` just complains that the cursor was left open, it doesn't take the initiative to close it.
https://github.com/psycopg/psycopg/blob/7b45cbc304e6a8a264f7dbc444a87cfae5b15064/psycopg/psycopg/_se...
So maybe it's the GC of a Django wrapper that calls `close()` on the psycopg server cursor?
^ permalink raw [nested|flat] 10+ messages in thread
* Re: [psycopg/psycopg] issue #1427: Hang in futex, linked to garbage collection
@ 2026-10-04 11:38 ` "JulienPalard (@JulienPalard)" <noreply+JulienPalard@github.com>
8 siblings, 0 replies; 10+ messages in thread
From: JulienPalard (@JulienPalard) @ 2026-10-04 11:38 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
> So maybe it's the GC of a Django wrapper that calls `close()` on the psycopg server cursor?
From the stacktrace given by faulthandler in my initial message I think we can trust the 2 first frames: close is called by:
```python
def cursor_iter(cursor, sentinel, col_count, itersize):
"""
Yield blocks of rows from a cursor and ensure the cursor is closed when
done.
"""
try:
for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
yield rows if col_count is None else [r[:col_count] for r in rows]
finally:
cursor.close()
```
in `django/db/models/sql/compiler.py`.
The next frame though I don’t think we can trust this, I bet the garbage collector mess with frames when it’s started, because it points to:
```python
return self._tx.load_rows(0, res.ntuples, self._make_row)
```
line 183 of `psycopg/_server_cursor_base.py` in `_fetch_gen` and I don’t see how this line can directly call `cursor_iter`, so that should be where the GC triggered, if I understand correctly.
So the GC somehow triggered the `finally:` of this `cursor_iter`, **without** leaving a trace in the call stack (I bet GC does "things" that can make this happen), the only path I can imagine leading to that is: some `__del__` somewhere has a kind of "purge the generator before dying", calling a last `next()` on that `cursor_iter`, triggering the `finally:` clause.
Remembering that it does **not** happen when psycopg-binary is not loaded, and maybe the use of psycopg-binary kinds of mess with the stacktrace too (I don’t know I haven't read psycopg-binary code yet).
^ permalink raw [nested|flat] 10+ messages in thread
* Re: [psycopg/psycopg] issue #1427: Hang in futex, linked to garbage collection
@ 2026-10-04 15:28 ` "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
8 siblings, 0 replies; 10+ messages in thread
From: dvarrazzo (@dvarrazzo) @ 2026-10-04 15:28 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
Yes, maybe the generator produced by this `cursor_iter()` is GC'd before completing, so `Cursor.close()` is called in its `__del__`.
I don't know exactly what to do our side: this code is not under our control but we can't blame it, it's pretty legit code. We should probably make the Django people aware of the issue and figure out if there's a different way to organise this code. I am not familiar with "the second `iter()` form" used here.
A defense that would work here would be adding a timeout on the lock acquisition on `close()`, but the need of a timeout is specific to only the GC invocation, which I don't see how to detect (because this is not our `__del__`), it wouldn't necessarily be appropriate for a `close()` in the normal execution realm, and it is several layers away (the lock is taken by the query executed by the server cursor close). I mean, any Psycopg path might end up being called by someone's destructor and end up in a similar deadlocking.
The effect of the C extension being loaded or not I think is undetermined: the execution of some C code only happens entirely inside the lock, whereas the lock is acquired in Python. `wait_c()` doesn't take a lock itself I seem to remember.
^ permalink raw [nested|flat] 10+ messages in thread
* Re: [psycopg/psycopg] issue #1427: Hang in futex, linked to garbage collection
@ 2026-11-12 11:57 ` "Binary-Refinery (@Binary-Refinery)" <noreply+Binary-Refinery@github.com>
8 siblings, 0 replies; 10+ messages in thread
From: Binary-Refinery (@Binary-Refinery) @ 2026-11-12 11:57 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
I can try to reduce the Django server-cursor cleanup to a subprocess test with a timeout and compare the Python and C implementations. Would those results help decide which project should own the fix?
^ permalink raw [nested|flat] 10+ messages in thread
end of thread, other threads:[~2026-11-12 11:57 UTC | newest]
Thread overview: 10+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-09-26 18:10 [psycopg/psycopg] issue #1427: Hang in futex, linked to garbage collection "JulienPalard (@JulienPalard)" <noreply+JulienPalard@github.com>
2026-09-28 11:08 ` "JulienPalard (@JulienPalard)" <noreply+JulienPalard@github.com>
2026-09-28 11:39 ` "JulienPalard (@JulienPalard)" <noreply+JulienPalard@github.com>
2026-09-30 22:39 ` "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
2026-10-01 11:35 ` "JulienPalard (@JulienPalard)" <noreply+JulienPalard@github.com>
2026-10-03 08:02 ` "JulienPalard (@JulienPalard)" <noreply+JulienPalard@github.com>
2026-10-03 16:56 ` "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
2026-10-04 11:38 ` "JulienPalard (@JulienPalard)" <noreply+JulienPalard@github.com>
2026-10-04 15:28 ` "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
2026-11-12 11:57 ` "Binary-Refinery (@Binary-Refinery)" <noreply+Binary-Refinery@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox