From: hosiet (@hosiet) <noreply+hosiet@github.com>
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
Subject: [psycopg/psycopg] issue #1497: Race condition in test_copy_concurrency causing tests fail randomly
Date: Fri, 13 Nov 2026 14:35:57 +0000
Message-ID: <gh-psycopg-psycopg-1497@github.com> (raw)
Disclaimer: this issue report and patch was drafted by GPT 6.1 Sol and manually revised and approved by a human.
* * *
While working on Debian's packaging related to psycopy, I noticed that `tests/test_copy.py::test_copy_concurrency` intermittently fails because its final assertion requires an ordering that the connection lock does not guarantee.
Debian observed this with psycopg 3.3.5 on riscv64 but it also applies to upstream git HEAD as of 2026-11-13. The copied rows pass their equality check before the execution-log assertion fails.
```text
At index 3 diff: 'worker_completed' != 'exited_copy'
```
[Debian failure log](https://ci.debian.net/data/autopkgtest/testing/riscv64/p/psycopg3/77833266/log.gz)
## Cause
The COPY thread appends `exited_copy` after leaving `with cur.copy(...)`. At that point, `Cursor.copy()` has released `conn.lock`. A waiting worker can acquire it, execute its query and append `worker_completed` before the COPY thread appends `exited_copy`.
```text
COPY thread releases conn.lock
worker thread completes SELECT 1 and appends worker_completed
COPY thread appends exited_copy
```
The test rejects this valid schedule. The same test structure is present in upstream master at bced8a8993064a378d5ca852215d99f73f4c6cae .
## Reproduction
Save the following plugin as `force_copy_race.py` in the checkout root. It pauses the COPY thread immediately after the connection lock is released until a worker completes, forcing the schedule above.
<details>
<summary>force_copy_race.py</summary>
```python
"""Force a legal thread switch after Cursor.copy() releases its lock.
Load with pytest -p force_copy_race and put this directory on PYTHONPATH.
Only the synchronous test_copy_concurrency test is instrumented.
"""
import inspect
import threading
import pytest
from psycopg import Cursor
@pytest.fixture(autouse=True)
def force_copy_race(request):
if request.node.originalname != "test_copy_concurrency" or "async" in request.node.module.__name__:
yield
return
worker_finished = threading.Event()
copy_function = inspect.unwrap(Cursor.copy)
lines, first_line = inspect.getsourcelines(copy_function)
release_line = next(
first_line + offset
for offset, line in enumerate(lines)
if "self._select_current_result(0)" in line
)
forced = []
def trace(frame, event, arg):
if event == "return" and frame.f_code.co_name == "worker_task":
worker_finished.set()
if (
event == "line"
and frame.f_code is copy_function.__code__
and frame.f_lineno == release_line
):
# This runs after the real lock has been released. Give a waiting
# worker time to finish before the COPY thread resumes.
forced.append(worker_finished.wait(5))
return trace
previous_trace = threading.gettrace()
threading.settrace(trace)
try:
yield
finally:
threading.settrace(previous_trace)
assert forced and all(forced), "the intended post-unlock schedule was not exercised"
```
</details>
With a running PostgreSQL server and an existing UTF-8 database accessible through `PSYCOPG_TEST_DSN`, run from the checkout root:
```sh
uv venv --python 3.14 .venv
uv pip install --python .venv/bin/python pytest anyio packaging pytest-timeout
export PSYCOPG_IMPL=python
export PSYCOPG_TEST_DSN='dbname=psycopg_test'
export PYTHONPATH="$PWD/psycopg:$PWD/psycopg_pool:$PWD"
.venv/bin/python -m pytest -p force_copy_race \
tests/test_copy.py::test_copy_concurrency -v --timeout=20
```
The original test reproduced the same assertion in 5/5 instrumented runs. An initial uninstrumented run passed.
## Proposed fix
Record `copy_body_done` at the end of the COPY body, while the connection lock is still held. Compare worker completions against that marker. The patch updates the async source and the generated sync test.
### Proposed patch
```diff
From: Boyuan Yang <byang@debian.org>
Date: Fri, 13 Nov 2026 09:17:59 -0500
Subject: [PATCH] tests: record COPY completion before releasing the connection lock
The synchronous test can record worker_completed before exited_copy even
when Cursor.copy() correctly holds the connection lock for the complete
COPY operation. Once the context manager releases the lock, a waiting
worker may run before the COPY thread appends its final log entry.
Record copy_body_done at the end of the protected body and assert that
workers complete after that marker. Update the async source and its
generated synchronous counterpart.
---
inline patch 1: tests/test_copy_async.py, tests/test_copy.py (+10/-8 lines) download inline-1.patchdiff --git a/tests/test_copy_async.py b/tests/test_copy_async.py--- a/tests/test_copy_async.py+++ b/tests/test_copy_async.py@@ -1008,8 +1008,9 @@
wrote_second.set()
await can_proceed.wait()
- # Copy context exited, lock should now be released- execution_log.append("exited_copy")+ # Record completion while the connection lock is still held.+ # A worker may run as soon as the context manager releases it.+ execution_log.append("copy_body_done")
async def worker_task():
"""
@@ -1055,12 +1056,12 @@
rows = await cur.fetchall()
assert rows == [(1, "first"), (2, "second")]
- # Verify that all workers completed AFTER copy exited+ # Verify that all workers completed after the COPY body.
assert execution_log == [
"entered_copy",
"wrote_row_1",
"wrote_row_2",
- "exited_copy",+ "copy_body_done",
"worker_completed",
"worker_completed",
"worker_completed",
diff --git a/tests/test_copy.py b/tests/test_copy.py--- a/tests/test_copy.py+++ b/tests/test_copy.py@@ -989,8 +989,9 @@
wrote_second.set()
can_proceed.wait()
- # Copy context exited, lock should now be released- execution_log.append("exited_copy")+ # Record completion while the connection lock is still held.+ # A worker may run as soon as the context manager releases it.+ execution_log.append("copy_body_done")
def worker_task():
"""
@@ -1036,12 +1037,12 @@
rows = cur.fetchall()
assert rows == [(1, "first"), (2, "second")]
- # Verify that all workers completed AFTER copy exited+ # Verify that all workers completed after the COPY body.
assert execution_log == [
"entered_copy",
"wrote_row_1",
"wrote_row_2",
- "exited_copy",+ "copy_body_done",
"worker_completed",
"worker_completed",
"worker_completed",
```
Please feel free to take this patch; I waive all my copyright on the drafted texts and diff.