pg.ddx.io  pgbouncer/pgbouncer GitHub issues and pull requests  
help / color / mirror / Atom feed
[pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode
4+ messages / 3 participants
[nested] [flat]

* [pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode
@ 2020-06-22 16:53  "ahesselgrave (@ahesselgrave)" <noreply+ahesselgrave@github.com>
  0 siblings, 0 replies; 4+ messages in thread

From: ahesselgrave (@ahesselgrave) @ 2020-06-22 16:53 UTC (permalink / raw)
  To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com>

* gevent version: 1.5.0
* Python version: CPython 3.7.4
* Operating System: AWS ECS Docker image based off python:3.7.4-slim
* Django version: 2.2.3
* Celery version: 4.3.0
* pgbouncer version: [edoburu/pgbouncer:1.12.0](https://github.com/edoburu/docker-pgbouncer/blob/v1.12.x/Dockerfile) Docker image

### Description:
#### Setup
My goal was to create a load testing environment to see how we could get Celery + gevent + pgbouncer to play nice together. Without pgbouncer, Celery + gevent would create far too many connections to our PostgreSQL database and cause downtime.

I set up 8 Celery tasks on ECS with the following command per task:

`celery -A cgp worker -l info -P gevent -c 1000`

and connected it a single pgbouncer image with the following configuration:

```
DEFAULT_POOL_SIZE=50
LISTEN_PORT=6543
MAX_CLIENT_CONN=10000
MAX_DB_CONNECTIONS=100
MIN_POOL_SIZE=20
POOL_MODE=transaction
VERBOSE=2
```

I then queued up 4000 identical tasks to test the task completion throughput and confirm that our database was not being aggressively tasked.

#### What I Expected
I expected the tasks to fully complete with our database connections limited by the pool settings

#### What Happened
Most times it would succeed, but on occasion all the database connections in the pool would get stuck in the `idle in transaction` state and would have no way of committing. pgbouncer would time out the connections based off the `query_wait_timeout` parameter and the next wave of connections and tasks would attempt to execute. More often than not, these tasks would fall in this same cycle of getting stuck `idle in transaction`.

I redeployed my configuration but instead switched celery to the prefork pool mode:

`celery -A cgp worker -l info -P prefork -c 8`

and tried running batches of tasks all the way to 80,000. Not once did it hang, so I believe that there was something related to our gevent implementation that caused that deadlock.

I am looking for guidance on how to better debug or understand the hanging issue. I've also posted an issue to gevent [here](https://github.com/gevent/gevent/issues/1652)

Happy to provide more context where helpful!



^ permalink  raw  [nested|flat] 4+ messages in thread

* Re: [pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode
@ 2022-10-22 09:57  "rmaleki (@rmaleki)" <noreply+rmaleki@github.com>
  2 siblings, 0 replies; 4+ messages in thread

From: rmaleki (@rmaleki) @ 2022-10-22 09:57 UTC (permalink / raw)
  To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com>

Hi @ahesselgrave 
did you find the answer?

^ permalink  raw  [nested|flat] 4+ messages in thread

* Re: [pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode
@ 2026-11-07 22:36  "eulerto (@eulerto)" <noreply+eulerto@github.com>
  2 siblings, 0 replies; 4+ messages in thread

From: eulerto (@eulerto) @ 2026-11-07 22:36 UTC (permalink / raw)
  To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com>

Per your [link](https://github.com/gevent/gevent/issues/1652), it seems PgBouncer was not the culprit.

^ permalink  raw  [nested|flat] 4+ messages in thread

* Re: [pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode
@ 2026-11-07 22:36  "eulerto (@eulerto)" <noreply+eulerto@github.com>
  2 siblings, 0 replies; 4+ messages in thread

From: eulerto (@eulerto) @ 2026-11-07 22:36 UTC (permalink / raw)
  To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com>

@eulerto closed this.

^ permalink  raw  [nested|flat] 4+ messages in thread


end of thread, other threads:[~2026-11-07 22:36 UTC | newest]

Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-06-22 16:53 [pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode "ahesselgrave (@ahesselgrave)" <noreply+ahesselgrave@github.com>
2022-10-22 09:57 ` "rmaleki (@rmaleki)" <noreply+rmaleki@github.com>
2026-11-07 22:36 ` "eulerto (@eulerto)" <noreply+eulerto@github.com>
2026-11-07 22:36 ` "eulerto (@eulerto)" <noreply+eulerto@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox