pg.ddx.io  psycopg/psycopg GitHub issues and pull requests  
help / color / mirror / Atom feed
[psycopg/psycopg] issue #1497: Race condition in test_copy_concurrency causing tests fail randomly
3+ messages / 3 participants
[nested] [flat]

* [psycopg/psycopg] issue #1497: Race condition in test_copy_concurrency causing tests fail randomly
@ 2026-11-13 14:35  "hosiet (@hosiet)" <noreply+hosiet@github.com>
  0 siblings, 0 replies; 3+ messages in thread

From: hosiet (@hosiet) @ 2026-11-13 14:35 UTC (permalink / raw)
  To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>

Disclaimer: this issue report and patch was drafted by GPT 6.1 Sol and manually revised and approved by a human.

* * *

While working on Debian's packaging related to psycopy, I noticed that `tests/test_copy.py::test_copy_concurrency` intermittently fails because its final assertion requires an ordering that the connection lock does not guarantee.

Debian observed this with psycopg 3.3.5 on riscv64 but it also applies to upstream git HEAD as of 2026-11-13. The copied rows pass their equality check before the execution-log assertion fails.

```text
At index 3 diff: 'worker_completed' != 'exited_copy'
```

[Debian failure log](https://ci.debian.net/data/autopkgtest/testing/riscv64/p/psycopg3/77833266/log.gz)

## Cause

The COPY thread appends `exited_copy` after leaving `with cur.copy(...)`. At that point, `Cursor.copy()` has released `conn.lock`. A waiting worker can acquire it, execute its query and append `worker_completed` before the COPY thread appends `exited_copy`.

```text
COPY thread       releases conn.lock
worker thread     completes SELECT 1 and appends worker_completed
COPY thread       appends exited_copy
```

The test rejects this valid schedule. The same test structure is present in upstream master at bced8a8993064a378d5ca852215d99f73f4c6cae .

## Reproduction

Save the following plugin as `force_copy_race.py` in the checkout root. It pauses the COPY thread immediately after the connection lock is released until a worker completes, forcing the schedule above.

<details>
<summary>force_copy_race.py</summary>

```python
"""Force a legal thread switch after Cursor.copy() releases its lock.

Load with pytest -p force_copy_race and put this directory on PYTHONPATH.
Only the synchronous test_copy_concurrency test is instrumented.
"""
import inspect
import threading

import pytest
from psycopg import Cursor


@pytest.fixture(autouse=True)
def force_copy_race(request):
    if request.node.originalname != "test_copy_concurrency" or "async" in request.node.module.__name__:
        yield
        return

    worker_finished = threading.Event()
    copy_function = inspect.unwrap(Cursor.copy)
    lines, first_line = inspect.getsourcelines(copy_function)
    release_line = next(
        first_line + offset
        for offset, line in enumerate(lines)
        if "self._select_current_result(0)" in line
    )
    forced = []

    def trace(frame, event, arg):
        if event == "return" and frame.f_code.co_name == "worker_task":
            worker_finished.set()
        if (
            event == "line"
            and frame.f_code is copy_function.__code__
            and frame.f_lineno == release_line
        ):
            # This runs after the real lock has been released. Give a waiting
            # worker time to finish before the COPY thread resumes.
            forced.append(worker_finished.wait(5))
        return trace

    previous_trace = threading.gettrace()
    threading.settrace(trace)
    try:
        yield
    finally:
        threading.settrace(previous_trace)
    assert forced and all(forced), "the intended post-unlock schedule was not exercised"
```

</details>

With a running PostgreSQL server and an existing UTF-8 database accessible through `PSYCOPG_TEST_DSN`, run from the checkout root:

```sh
uv venv --python 3.14 .venv
uv pip install --python .venv/bin/python pytest anyio packaging pytest-timeout
export PSYCOPG_IMPL=python
export PSYCOPG_TEST_DSN='dbname=psycopg_test'
export PYTHONPATH="$PWD/psycopg:$PWD/psycopg_pool:$PWD"
.venv/bin/python -m pytest -p force_copy_race \
    tests/test_copy.py::test_copy_concurrency -v --timeout=20
```

The original test reproduced the same assertion in 5/5 instrumented runs. An initial uninstrumented run passed.

## Proposed fix

Record `copy_body_done` at the end of the COPY body, while the connection lock is still held. Compare worker completions against that marker. The patch updates the async source and the generated sync test.

### Proposed patch

```diff
From: Boyuan Yang <byang@debian.org>
Date: Fri, 13 Nov 2026 09:17:59 -0500
Subject: [PATCH] tests: record COPY completion before releasing the connection lock

The synchronous test can record worker_completed before exited_copy even
when Cursor.copy() correctly holds the connection lock for the complete
COPY operation. Once the context manager releases the lock, a waiting
worker may run before the COPY thread appends its final log entry.

Record copy_body_done at the end of the protected body and assert that
workers complete after that marker. Update the async source and its
generated synchronous counterpart.

---
inline patch 1: tests/test_copy_async.py, tests/test_copy.py (+10/-8 lines) download inline-1.patchdiff --git a/tests/test_copy_async.py b/tests/test_copy_async.py --- a/tests/test_copy_async.py +++ b/tests/test_copy_async.py @@ -1008,8 +1008,9 @@ wrote_second.set() await can_proceed.wait() - # Copy context exited, lock should now be released - execution_log.append("exited_copy") + # Record completion while the connection lock is still held. + # A worker may run as soon as the context manager releases it. + execution_log.append("copy_body_done") async def worker_task(): """ @@ -1055,12 +1056,12 @@ rows = await cur.fetchall() assert rows == [(1, "first"), (2, "second")] - # Verify that all workers completed AFTER copy exited + # Verify that all workers completed after the COPY body. assert execution_log == [ "entered_copy", "wrote_row_1", "wrote_row_2", - "exited_copy", + "copy_body_done", "worker_completed", "worker_completed", "worker_completed", diff --git a/tests/test_copy.py b/tests/test_copy.py --- a/tests/test_copy.py +++ b/tests/test_copy.py @@ -989,8 +989,9 @@ wrote_second.set() can_proceed.wait() - # Copy context exited, lock should now be released - execution_log.append("exited_copy") + # Record completion while the connection lock is still held. + # A worker may run as soon as the context manager releases it. + execution_log.append("copy_body_done") def worker_task(): """ @@ -1036,12 +1037,12 @@ rows = cur.fetchall() assert rows == [(1, "first"), (2, "second")] - # Verify that all workers completed AFTER copy exited + # Verify that all workers completed after the COPY body. assert execution_log == [ "entered_copy", "wrote_row_1", "wrote_row_2", - "exited_copy", + "copy_body_done", "worker_completed", "worker_completed", "worker_completed",
``` Please feel free to take this patch; I waive all my copyright on the drafted texts and diff. ^ permalink raw [nested|flat] 3+ messages in thread

* Re: [psycopg/psycopg] issue #1497: Race condition in test_copy_concurrency causing tests fail randomly
@ 2026-11-17 02:47  "adityashankarv (@adityashankarv)" <noreply+adityashankarv@github.com>
  1 sibling, 0 replies; 3+ messages in thread

From: adityashankarv (@adityashankarv) @ 2026-11-17 02:47 UTC (permalink / raw)
  To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>

Thanks for the write-up and the patch. I agree the assertion is checking an ordering the lock doesn't guarantee, since `exited_copy` is logged after `copy()` has already released `conn.lock`. I'd like to open a PR with your change: record `copy_body_done` inside the `with cur.copy(...)` block in `tests/test_copy_async.py`, then regenerate the sync test with `tools/async_to_sync.py`, so the test only requires workers to run after the COPY body. I'll credit you in the PR.


^ permalink  raw  [nested|flat] 3+ messages in thread

* Re: [psycopg/psycopg] issue #1497: Race condition in test_copy_concurrency causing tests fail randomly
@ 2026-11-17 12:18  "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
  1 sibling, 0 replies; 3+ messages in thread

From: dvarrazzo (@dvarrazzo) @ 2026-11-17 12:18 UTC (permalink / raw)
  To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>

@adityashankarv please keep your AI out of this project thank you.

^ permalink  raw  [nested|flat] 3+ messages in thread


end of thread, other threads:[~2026-11-17 12:18 UTC | newest]

Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-13 14:35 [psycopg/psycopg] issue #1497: Race condition in test_copy_concurrency causing tests fail randomly "hosiet (@hosiet)" <noreply+hosiet@github.com>
2026-11-17 02:47 ` "adityashankarv (@adityashankarv)" <noreply+adityashankarv@github.com>
2026-11-17 12:18 ` "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox