pg.ddx.io  pgbouncer/pgbouncer GitHub issues and pull requests  
help / color / mirror / Atom feed
From: vadv (@vadv) <noreply+vadv@github.com>
To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com>
Subject: [pgbouncer/pgbouncer] issue #1704: Async protocol: PgBouncer does not answer Flush in some cases
Date: Fri, 13 Nov 2026 06:55:36 +0000
Message-ID: <gh-pgbouncer-pgbouncer-1704@github.com> (raw)

Hello. Thank you for PgBouncer.

We want to ask about support for the asynchronous (pipelined) form of the extended query protocol.

## Background

In the pipelined form a client sends several commands in one batch and reads the results together. The client can use the Flush message to say: send me the responses that are ready now, without waiting for the end of the batch (a Sync message). Direct PostgreSQL answers Flush with the pending responses immediately. This is how pipelining works: clients can read partial results mid-batch without breaking the pipeline.

The libpq function `PQsendFlushRequest` (libpq 16+) builds this. The psycopg3 driver uses it to fetch ParseComplete after `cursor.prepare(…)` before continuing the pipeline. Other drivers that implement pipelining work the same way: they rely on Flush to get an early answer.

## What we see

PgBouncer answers Flush in some cases and stays silent in others. The client waits for the response and gets nothing until it sends Sync. The wait lasts as long as the client's configured timeout (2 seconds in our test). Sync releases the buffered response immediately, so the data was ready all along and was held back.

We see this when the Parse message uses a statement name that PgBouncer already tracks. The name is prepared by one client, the connection returns to the pool, and another client parses the same name on the same server connection and sends Flush. A fresh statement name works: the response comes right away.

Version: pgbouncer 1.26.1. We also built the current master (commit `ac3bb6d`, 2026-11-06). The behavior is the same in both.

## Steps (repro scenario)

Setup:
- `pool_mode = transaction`
- `default_pool_size = 1`
- `max_prepared_statements = 10`

Steps:
1. Client 1 connects. Client 1 prepares statement `s1` and executes it. Client 1 disconnects. The server connection goes back to the pool. PgBouncer keeps `s1` in its prepared statement tracker.
2. Client 2 connects and gets the same server connection. Client 2 enters pipeline mode (libpq `PQenterPipelineMode`). Client 2 sends `Parse("s1", "SELECT 1")` and `Flush` (libpq `PQsendPrepare` + `PQsendFlushRequest`). Client 2 does not send Sync.
3. Client 2 polls for data. Nothing arrives. The poll times out after 2 seconds. Then Client 2 sends Sync. ParseComplete arrives immediately.

So do Odyssey and pg_doorman (both support transaction pooling and both track prepared statements).

## Test

We wrote a small test in the style of the `test/` directory in this repository. Put the attached file `test_prepared_flush.py` into `test/` and run:

```bash
pytest test/test_prepared_flush.py -v
```

The test fails on this line:

```python
assert ready, "ParseComplete not delivered after Flush, silence until Sync"
```

The test uses psycopg3 because it has good support for libpq pipelining. It follows the same pattern as other tests in the pgbouncer suite.

## Questions

- Is the response intentionally held until Sync in this case?
- Is there a known issue about incomplete Flush support? We did not find one.
- If it is not intentional, would you accept a change that pushes the pending response to the client on Flush?

We are happy to provide more details or adjust the test if needed.

Thank you.

---

## test_prepared_flush.py

```python
import time

import pytest
from psycopg import pq

from .utils import LIBPQ_SUPPORTS_PIPELINING

# Silence after Parse + Flush, before any Sync. Direct PostgreSQL answers
# ParseComplete in well under a millisecond, so 2s is generous.
FLUSH_TIMEOUT = 2


@pytest.mark.skipif("not LIBPQ_SUPPORTS_PIPELINING")
def test_reprepare_with_flush_gets_parsecomplete_before_sync(bouncer):
    bouncer.admin("set pool_mode=transaction")
    bouncer.admin("set default_pool_size=1")
    bouncer.admin("set max_prepared_statements=10")

    # Client 1: prepare a named statement, run it, disconnect. With
    # default_pool_size=1 the server connection returns to the pool and
    # PgBouncer keeps the name in its prepared statement tracker.
    with bouncer.conn() as conn:
        pg = conn.pgconn
        pg.enter_pipeline_mode()
        pg.send_prepare(b"s1", b"SELECT 1")
        pg.send_query_prepared(b"s1", [])
        pg.pipeline_sync()
        assert pg.get_result().status == pq.ExecStatus.COMMAND_OK
        assert pg.get_result() is None
        assert pg.get_result().status == pq.ExecStatus.TUPLES_OK
        assert pg.get_result() is None
        assert pg.get_result().status == pq.ExecStatus.PIPELINE_SYNC
        pg.exit_pipeline_mode()

    # Client 2 (reconnect, lands on the same server connection): re-prepare
    # the tracked name mid-pipeline and ask for the answer with a Flush,
    # before any Sync. libpq builds exactly this via PQsendPrepare +
    # PQsendFlushRequest (libpq >= 16); Flush means "deliver pending
    # results now", see the PostgreSQL docs on pipeline mode.
    with bouncer.conn() as conn:
        pg = conn.pgconn
        pg.enter_pipeline_mode()
        pg.send_prepare(b"s1", b"SELECT 1")
        pg.send_flush_request()
        pg.flush()
        try:
            deadline = time.monotonic() + FLUSH_TIMEOUT
            ready = False
            while time.monotonic() < deadline:
                pg.consume_input()
                if not pg.is_busy():
                    ready = True
                    break
                time.sleep(0.05)
            assert ready, (
                "ParseComplete not delivered after Flush, silence until Sync"
            )
            assert pg.get_result().status == pq.ExecStatus.COMMAND_OK
            assert pg.get_result() is None
        finally:
            # End the pipeline so the connection is returned cleanly.
            pg.pipeline_sync()
            while (res := pg.get_result()) is not None:
                if res.status == pq.ExecStatus.PIPELINE_SYNC:
                    break
            pg.exit_pipeline_mode()
```


view thread (3+ messages)  latest in thread

Message-ID: <gh-pgbouncer-pgbouncer-1704@github.com>
Permalink:  ../gh-pgbouncer-pgbouncer-1704@github.com/

 · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox