pg.ddx.io pgbouncer/pgbouncer GitHub issues and pull requestshelp / color / mirror / Atom feed
[pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode 4+ messages / 3 participants [nested] [flat]
* [pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode @ 2020-06-22 16:53 "ahesselgrave (@ahesselgrave)" <noreply+ahesselgrave@github.com> 0 siblings, 0 replies; 4+ messages in thread From: ahesselgrave (@ahesselgrave) @ 2020-06-22 16:53 UTC (permalink / raw) To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com> * gevent version: 1.5.0 * Python version: CPython 3.7.4 * Operating System: AWS ECS Docker image based off python:3.7.4-slim * Django version: 2.2.3 * Celery version: 4.3.0 * pgbouncer version: [edoburu/pgbouncer:1.12.0](https://github.com/edoburu/docker-pgbouncer/blob/v1.12.x/Dockerfile) Docker image ### Description: #### Setup My goal was to create a load testing environment to see how we could get Celery + gevent + pgbouncer to play nice together. Without pgbouncer, Celery + gevent would create far too many connections to our PostgreSQL database and cause downtime. I set up 8 Celery tasks on ECS with the following command per task: `celery -A cgp worker -l info -P gevent -c 1000` and connected it a single pgbouncer image with the following configuration: ``` DEFAULT_POOL_SIZE=50 LISTEN_PORT=6543 MAX_CLIENT_CONN=10000 MAX_DB_CONNECTIONS=100 MIN_POOL_SIZE=20 POOL_MODE=transaction VERBOSE=2 ``` I then queued up 4000 identical tasks to test the task completion throughput and confirm that our database was not being aggressively tasked. #### What I Expected I expected the tasks to fully complete with our database connections limited by the pool settings #### What Happened Most times it would succeed, but on occasion all the database connections in the pool would get stuck in the `idle in transaction` state and would have no way of committing. pgbouncer would time out the connections based off the `query_wait_timeout` parameter and the next wave of connections and tasks would attempt to execute. More often than not, these tasks would fall in this same cycle of getting stuck `idle in transaction`. I redeployed my configuration but instead switched celery to the prefork pool mode: `celery -A cgp worker -l info -P prefork -c 8` and tried running batches of tasks all the way to 80,000. Not once did it hang, so I believe that there was something related to our gevent implementation that caused that deadlock. I am looking for guidance on how to better debug or understand the hanging issue. I've also posted an issue to gevent [here](https://github.com/gevent/gevent/issues/1652) Happy to provide more context where helpful! ^ permalink raw [nested|flat] 4+ messages in thread
* Re: [pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode @ 2022-10-22 09:57 "rmaleki (@rmaleki)" <noreply+rmaleki@github.com> 2 siblings, 0 replies; 4+ messages in thread From: rmaleki (@rmaleki) @ 2022-10-22 09:57 UTC (permalink / raw) To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com> Hi @ahesselgrave did you find the answer? ^ permalink raw [nested|flat] 4+ messages in thread
* Re: [pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode @ 2026-11-07 22:36 "eulerto (@eulerto)" <noreply+eulerto@github.com> 2 siblings, 0 replies; 4+ messages in thread From: eulerto (@eulerto) @ 2026-11-07 22:36 UTC (permalink / raw) To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com> Per your [link](https://github.com/gevent/gevent/issues/1652), it seems PgBouncer was not the culprit. ^ permalink raw [nested|flat] 4+ messages in thread
* Re: [pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode @ 2026-11-07 22:36 "eulerto (@eulerto)" <noreply+eulerto@github.com> 2 siblings, 0 replies; 4+ messages in thread From: eulerto (@eulerto) @ 2026-11-07 22:36 UTC (permalink / raw) To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com> @eulerto closed this. ^ permalink raw [nested|flat] 4+ messages in thread
end of thread, other threads:[~2026-11-07 22:36 UTC | newest] Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-06-22 16:53 [pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode "ahesselgrave (@ahesselgrave)" <noreply+ahesselgrave@github.com> 2022-10-22 09:57 ` "rmaleki (@rmaleki)" <noreply+rmaleki@github.com> 2026-11-07 22:36 ` "eulerto (@eulerto)" <noreply+eulerto@github.com> 2026-11-07 22:36 ` "eulerto (@eulerto)" <noreply+eulerto@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox