From: ahesselgrave (@ahesselgrave) <noreply+ahesselgrave@github.com>
To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com>
Subject: [pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode
Date: Mon, 22 Jun 2020 16:53:18 +0000
Message-ID: <gh-pgbouncer-pgbouncer-494@github.com> (raw)
* gevent version: 1.5.0
* Python version: CPython 3.7.4
* Operating System: AWS ECS Docker image based off python:3.7.4-slim
* Django version: 2.2.3
* Celery version: 4.3.0
* pgbouncer version: [edoburu/pgbouncer:1.12.0](https://github.com/edoburu/docker-pgbouncer/blob/v1.12.x/Dockerfile) Docker image
### Description:
#### Setup
My goal was to create a load testing environment to see how we could get Celery + gevent + pgbouncer to play nice together. Without pgbouncer, Celery + gevent would create far too many connections to our PostgreSQL database and cause downtime.
I set up 8 Celery tasks on ECS with the following command per task:
`celery -A cgp worker -l info -P gevent -c 1000`
and connected it a single pgbouncer image with the following configuration:
```
DEFAULT_POOL_SIZE=50
LISTEN_PORT=6543
MAX_CLIENT_CONN=10000
MAX_DB_CONNECTIONS=100
MIN_POOL_SIZE=20
POOL_MODE=transaction
VERBOSE=2
```
I then queued up 4000 identical tasks to test the task completion throughput and confirm that our database was not being aggressively tasked.
#### What I Expected
I expected the tasks to fully complete with our database connections limited by the pool settings
#### What Happened
Most times it would succeed, but on occasion all the database connections in the pool would get stuck in the `idle in transaction` state and would have no way of committing. pgbouncer would time out the connections based off the `query_wait_timeout` parameter and the next wave of connections and tasks would attempt to execute. More often than not, these tasks would fall in this same cycle of getting stuck `idle in transaction`.
I redeployed my configuration but instead switched celery to the prefork pool mode:
`celery -A cgp worker -l info -P prefork -c 8`
and tried running batches of tasks all the way to 80,000. Not once did it hang, so I believe that there was something related to our gevent implementation that caused that deadlock.
I am looking for guidance on how to better debug or understand the hanging issue. I've also posted an issue to gevent [here](https://github.com/gevent/gevent/issues/1652)
Happy to provide more context where helpful!