pg.ddx.io  pgbouncer/pgbouncer GitHub issues and pull requests  
help / color / mirror / Atom feed
From: ahesselgrave (@ahesselgrave) <noreply+ahesselgrave@github.com>
To: pgbouncer/pgbouncer <noreply+pgbouncer-pgbouncer@github.com>
Subject: [pgbouncer/pgbouncer] issue #494: Django + Celery + Gevent sometimes causes serverside hanging in transaction mode
Date: Mon, 22 Jun 2020 16:53:18 +0000
Message-ID: <gh-pgbouncer-pgbouncer-494@github.com> (raw)

* gevent version: 1.5.0
* Python version: CPython 3.7.4
* Operating System: AWS ECS Docker image based off python:3.7.4-slim
* Django version: 2.2.3
* Celery version: 4.3.0
* pgbouncer version: [edoburu/pgbouncer:1.12.0](https://github.com/edoburu/docker-pgbouncer/blob/v1.12.x/Dockerfile) Docker image

### Description:
#### Setup
My goal was to create a load testing environment to see how we could get Celery + gevent + pgbouncer to play nice together. Without pgbouncer, Celery + gevent would create far too many connections to our PostgreSQL database and cause downtime.

I set up 8 Celery tasks on ECS with the following command per task:

`celery -A cgp worker -l info -P gevent -c 1000`

and connected it a single pgbouncer image with the following configuration:

```
DEFAULT_POOL_SIZE=50
LISTEN_PORT=6543
MAX_CLIENT_CONN=10000
MAX_DB_CONNECTIONS=100
MIN_POOL_SIZE=20
POOL_MODE=transaction
VERBOSE=2
```

I then queued up 4000 identical tasks to test the task completion throughput and confirm that our database was not being aggressively tasked.

#### What I Expected
I expected the tasks to fully complete with our database connections limited by the pool settings

#### What Happened
Most times it would succeed, but on occasion all the database connections in the pool would get stuck in the `idle in transaction` state and would have no way of committing. pgbouncer would time out the connections based off the `query_wait_timeout` parameter and the next wave of connections and tasks would attempt to execute. More often than not, these tasks would fall in this same cycle of getting stuck `idle in transaction`.

I redeployed my configuration but instead switched celery to the prefork pool mode:

`celery -A cgp worker -l info -P prefork -c 8`

and tried running batches of tasks all the way to 80,000. Not once did it hang, so I believe that there was something related to our gevent implementation that caused that deadlock.

I am looking for guidance on how to better debug or understand the hanging issue. I've also posted an issue to gevent [here](https://github.com/gevent/gevent/issues/1652)

Happy to provide more context where helpful!



view thread (4+ messages)  latest in thread

Message-ID: <gh-pgbouncer-pgbouncer-494@github.com>
Permalink:  ../gh-pgbouncer-pgbouncer-494@github.com/

 · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox